Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I'm practicing the code from 'Web Scraping with Python', and I keep having this certificate problem:

from urllib.request import urlopen 
from bs4 import BeautifulSoup 
import re
pages = set()
def getLinks(pageUrl):
    global pages
    html = urlopen("http://en.wikipedia.org"+pageUrl)
    bsObj = BeautifulSoup(html)
    for link in bsObj.findAll("a", href=re.compile("^(/wiki/)")):
        if 'href' in link.attrs:
            if link.attrs['href'] not in pages:
                #We have encountered a new page
                newPage = link.attrs['href'] 
                print(newPage) 
                pages.add(newPage) 
                getLinks(newPage)
getLinks("")
The error is:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1049)>
Btw,I was also practicing scrapy, but kept getting the problem: command not found: scrapy (I tried all sorts of solutions online but none works... really frustrating)
                urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1049)>
– Catherine4j
                May 8, 2018 at 14:32
                There are 529 existing questions on SSL: CERTIFICATE_VERIFY_FAILED, please figure out which is your solution then close this as duplicate.
– smci
                May 9, 2018 at 0:02
                If you install Python using Homebrew that file does not exist. The solution is here: stackoverflow.com/a/44649450/412896
– Sampo
                Mar 6, 2019 at 11:09
                See answer from @Rambod which is fast and quick and solved for me. import ssl ssl._create_default_https_context = ssl._create_unverified_context that's it and it worked like a charm. Yes there may be security isue, but for one off scripts and known source.. who gives a *
– ihightower
                Oct 21, 2020 at 3:36
                this is good because why do I have to make system changes if all I need to add is two lines of code.
– Nagri
                Jul 29, 2020 at 8:20
                this answer should probably mention that this code could introduce huge security concerns depending on what the user does with the data after it's scraped..
– Michael Altfield
                Aug 9, 2020 at 12:27
                for sure when you use unverified certificate you are at risk. and this line of code only good if you know what data source is and its not good for production mode to set it up as default. question is about how to scarp some unverified page this code wont make big security concern for this situation at least.
– Rambod
                Aug 9, 2020 at 19:45
                @MiguelSilva open is specific to MacOS. You have Google colab running with MacOS? Wouldn't know about colab.
– Hillsie
                Mar 18, 2021 at 4:22
                @MiguelSilva The above command is at the operating system level and independent of the browser. Its related to Python. Additionally,  there is one constant in this industry and it's Change. So what worked in the past, might not work in the present.
– Hillsie
                Mar 21, 2021 at 23:31
To solve this: 
All you need to do is to install Python certificates! A common issue on macOS.  
Open these files: 
Install Certificates.command
Update Shell Profile.command
Simply Run these two scripts and you wont have this issue any more.  
Hope this helps!
                I don't see a Python directory in my Application Dir.. I'm using Anaconda and created a new env for python 3.6 since I previously had python 2.7 installed (which apparently did not create a folder in my Applications Dir either!)... Can't find the Certificates.command on my mac
– Kai
                Apr 17, 2019 at 13:21
                Why did this answer even get upvoted at all? It doesn't specify where these commands are supposed to be run, and it lacks information about those "two scripts"
– oeter
                Jan 21, 2022 at 13:59
                I'm running into his problem with a Python lambda in AWS, so this doesn't help because I don't have access to this.
– sovemp
                Dec 22, 2022 at 17:13
For anyone who is using anaconda, you would install the certifi package, see more at: 
https://anaconda.org/anaconda/certifi
To install, type this line in your terminal:
conda install -c anaconda certifi
                I have this problem too. Can't create an environment. I just installed miniconda3 (macOS) and I see certifi installed ('conda list')..
– Kai
                Dec 12, 2019 at 20:41
                For me this simply yields ERROR: Could not install packages due to an OSError: Cannot move the non-empty directory '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certifi-2021.10.8.dist-info/': Lacking write permission to '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certifi-2021.10.8.dist-info/'.
– eddyoc
                Jul 29, 2022 at 4:09
Take a look at this post, it seems like for later versions of Python, certificates are not pre installed which seems to cause this error. You should be able to run the following command to install the certifi package: /Applications/Python\ 3.6/Install\ Certificates.command
Post 1: urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error
Post 2: Airbrake error: urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
I had the same error and solved the problem by running the program code below:
# install_certifi.py
# sample script to install or update a set of default Root Certificates
# for the ssl module.  Uses the certificates provided by the certifi package:
#       https://pypi.python.org/pypi/certifi
import os
import os.path
import ssl
import stat
import subprocess
import sys
STAT_0o775 = ( stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
             | stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP
             | stat.S_IROTH |                stat.S_IXOTH )
def main():
    openssl_dir, openssl_cafile = os.path.split(
        ssl.get_default_verify_paths().openssl_cafile)
    print(" -- pip install --upgrade certifi")
    subprocess.check_call([sys.executable,
        "-E", "-s", "-m", "pip", "install", "--upgrade", "certifi"])
    import certifi
    # change working directory to the default SSL directory
    os.chdir(openssl_dir)
    relpath_to_certifi_cafile = os.path.relpath(certifi.where())
    print(" -- removing any existing file or link")
        os.remove(openssl_cafile)
    except FileNotFoundError:
    print(" -- creating symlink to certifi certificate bundle")
    os.symlink(relpath_to_certifi_cafile, openssl_cafile)
    print(" -- setting permissions")
    os.chmod(openssl_cafile, STAT_0o775)
    print(" -- update complete")
if __name__ == '__main__':
    main()
i didn't solve the problem, sadly.
but managed to make to codes work (almost all of my codes have this probelm btw)
the local issuer certificate problem happens under python3.7
so i changed back to python2.7 QAQ
and all that needed to change including "from urllib2 import urlopen" instead of "from urllib.request import urlopen"
so sad...
def getLinks(pageUrl):
    global pages
    html = requests.get("http://en.wikipedia.org"+pageUrl, verify=False).text
    bsObj = BeautifulSoup(html)
    for link in bsObj.findAll("a", href=re.compile("^(/wiki/)")):
        if 'href' in link.attrs:
            if link.attrs['href'] not in pages:
                #We have encountered a new page
                newPage = link.attrs['href']
                print(newPage)
                pages.add(newPage)
                getLinks(newPage)
getLinks("")
Check if this works for you
I'm a relative novice compared to all the experts on Stack Overflow.
I have 2 versions of jupyter notebook running (one through a fresh Anaconda Navigator installation and one through ????). I think this is because Anaconda was installed as a local installation on my Mac (per Anaconda instructions). 
I already had python 3.7 installed. After that, I used my terminal to open jupyter notebook and I think that it put another version globally onto my Mac. 
However, I'm not sure because I'm just learning through trial and error!
I did the terminal command: 
conda install -c anaconda certifi 
(as directed above, but it didn't work.) 
My python 3.7 is installed on OS Catalina10.15.3 in:
/Library/Python/3.7/site-packages AND
~/Library/Python/3.7/lib/python/site-packages
The certificate is at:
~/Library/Python/3.7/lib/python/site-packages/certifi-2019.11.28.dist-info
I tried to find the Install Certificate.command ... but couldn't find it through looking through the file structures...not in Applications...not in links above.
I finally installed it by finding it through Spotlight (as someone suggested above). And it double clicked automatically and installed ANOTHER certificate in the same folder as:
~/Library/Python/3.7/lib/python/site-packages/
NONE of the above solved anything for me...I still got the same error.  
So, I solved the problem by:
closing my jupyter notebook.
opening Anaconda Navigator.
opening jupyter notebook through the Navigator GUI (instead of
through Terminal). 
opening my notebook and running the code.
I can't tell you why this worked.  But it solved the problem for me.
I just want to save someone the hassle next time. If someone can tell my why it worked, that would be terrific.
I didn't try the other terminal commands because of the 2 versions of jupyter notebook that I knew were a problem. I just don't know how to fix that.
For me the problem was that I was setting REQUESTS_CA_BUNDLE in my .bash_profile
/Users/westonagreene/.bash_profile:
export REQUESTS_CA_BUNDLE=/usr/local/etc/openssl/cert.pem
Once I set REQUESTS_CA_BUNDLE to blank (i.e. removed from .bash_profile), requests worked again.
export REQUESTS_CA_BUNDLE=""
The problem only exhibited when executing python requests via a CLI (Command Line Interface). If I ran requests.get(URL, CERT) it resolved just fine.
Mac OS Catalina (10.15.6).
Pyenv of 3.6.11.
Error message I was getting: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
My answer elsewhere: https://stackoverflow.com/a/64151964/4420657
I am using Debian 10 buster and try download a file with youtube-dl and get this error:
sudo youtube-dl -k https://youtu.be/uscis0CnDjk
[youtube] uscis0CnDjk: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)> (caused by URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))
Certificates with python2 and python3.8 are installed correctly, but i persistent receive the same error.
finally (which is not the best solution, but works for me was to eliminate the certificate check as it is given as an option in youtube-dl) whith this command
sudo youtube-dl -k --no-check-certificate https://youtu.be/uscis0CnDjk 
I am seeing this issue on a Ubuntu 20.04 system and none of the "real fixes" (like this one) helped.
While Firefox was willing to open the site just fine neither GNOME Web (i.e. Epiphany) nor Python3 or wget were accepting the certificate. After some searching, I came across this answer on ServerFault which lists two common reasons:
The certificate is really signed by an unknown CA (for instance an internal CA).
The certificate is signed with an intermediate CA certificate from one of the well known CA's and the remote server is misconfigured in the regard that it doesn't include that intermediate CA certificate as a CA chain it's response.
You can use the Qualys SSL Labs website to check the site's certificates and if there are issues, contact the site's administrator to have it fixed.
If you really need to work around the issue right now, I'd recommend a temporary solution like Rambod's confined to the site(s) you're trying to access.
                Could you add some context to your answer, so readers know why the websockets version is important? Thanks! :)
– Aaron Meese
                Aug 10, 2022 at 0:14
I had the problem that python somehow was trying to use a cert.pem file that didn't exist. This can be seen by running:
import ssl
paths = ssl.get_default_verify_paths()
The openssl_cafile pointed to /etc/ssl/cert.pem which did not exist under that path.
Setting SSL_CERT_FILE to a path that does exist solved the problem:
export SSL_CERT_FILE=/etc/pki/tls/cert.pem
import pymongo
from pymongo.mongo_client import MongoClient
CONNECTION_STRING = "mongodb+srv://username:password@clustername.g3gasa2.mongodb.net/?retryWrites=true&w=majority**&ssl_cert_reqs=CERT_NONE**"
client = pymongo.MongoClient(CONNECTION_STRING )
In windows I tried to connect mongodb with jupyter notebook, finally by adding &ssl_cert_reqs=CERT_NONE in my CONNECTION_STRING helped me.
This basically disables SSL certificate verification (not recommended for production)
            thirukarthika nadar is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.        
This will work. Set the environment variable PYTHONHTTPSVERIFY to 0.
By typing linux command:
export PYTHONHTTPSVERIFY = 0
Using in python code:
import os
os.environ["PYTHONHTTPSVERIFY"] = "0"
BTW guys if you are getting the same error using aiohttp just put verify_ssl=False argument into your TCPConnector:
import aiohttp
async with aiohttp.ClientSession(
    connector=aiohttp.TCPConnector(verify_ssl=False)
) as session:
    async with session.get(url) as response:
        body = await response.text()
I am using anaconda on windows. Was getting the same error until I tried the following;
import urllib.request
link = 'http://docs.python.org'
with urllib.request.urlopen(link) as response:
    htmlSource = response.read()
which I got from the stackoverflow thread on using urlopen:
Python urllib urlopen not working
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.