Getting "OSError: [Errno 0] Error" when using either requests or urllib from python

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I would like to use the GTEx API . I did use requests recently (a few weeks ago), and it worked well, but for an unknown reason to me, it's not working anymore.
For example, I want to retrieve information using 48 different tissues, as available in the API.
Here follows a minimum reproducible code to get the error:
# Import libraries
import json
import urllib
import requests
# Set the paremeters  
gene = 'NOS1AP'
snp = 'rs10918593'
datasetId = 'gtex_v7'
tissues = ['Adipose_Subcutaneous', 'Adipose_Visceral_Omentum', 'Adrenal_Gland', 'Artery_Aorta', 'Artery_Coronary', 'Artery_Tibial', 'Brain_Amygdala', 'Brain_Anterior_cingulate_cortex_BA24', 'Brain_Caudate_basal_ganglia', 'Brain_Cerebellar_Hemisphere', 'Brain_Cerebellum', 'Brain_Cortex', 'Brain_Frontal_Cortex_BA9', 'Brain_Hippocampus', 'Brain_Hypothalamus', 'Brain_Nucleus_accumbens_basal_ganglia', 'Brain_Putamen_basal_ganglia', 'Brain_Spinal_cord_cervical_c-1', 'Brain_Substantia_nigra', 'Breast_Mammary_Tissue', 'Cells_EBV-transformed_lymphocytes', 'Cells_Transformed_fibroblasts', 'Colon_Sigmoid', 'Colon_Transverse', 'Esophagus_Gastroesophageal_Junction', 'Esophagus_Mucosa', 'Esophagus_Muscularis', 'Heart_Atrial_Appendage', 'Heart_Left_Ventricle', 'Liver', 'Lung', 'Minor_Salivary_Gland', 'Muscle_Skeletal', 'Nerve_Tibial', 'Ovary', 'Pancreas', 'Pituitary', 'Prostate', 'Skin_Not_Sun_Exposed_Suprapubic', 'Skin_Sun_Exposed_Lower_leg', 'Small_Intestine_Terminal_Ileum', 'Spleen', 'Stomach', 'Testis', 'Thyroid', 'Uterus', 'Vagina', 'Whole_Blood']
When I run the following code, I get an error for some initial tissues:
# using requests
for tissue in tissues:
    print(f'Getting eQTL for tissue {tissue}...')
    server = 'https://gtexportal.org/rest/v1/'
    ext = f'association/dyneqtl?gencodeId={gene}&variantId={snp}&tissueSiteDetailId={tissue}&datasetId={datasetId}'
    r = requests.get(server+ext, headers={"Accept" : "application/json"})
    if not r.ok:
        print(f'--- Request for SNP "{snp}" and "{gene}" returned an error! ---')
        continue
    decoded = r.json()
    r.close()
Error from the code above:
Getting eQTL for tissue Adipose_Subcutaneous...
Getting eQTL for tissue Adipose_Visceral_Omentum...
Getting eQTL for tissue Adrenal_Gland...
Getting eQTL for tissue Artery_Aorta...
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
~/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    669             # Make the request on the httplib connection object.
--> 670             httplib_response = self._make_request(
    671                 conn,
~/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    380         try:
--> 381             self._validate_conn(conn)
    382         except (SocketTimeout, BaseSSLError) as e:
~/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
    977         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
--> 978             conn.connect()
~/miniconda3/lib/python3.8/site-packages/urllib3/connection.py in connect(self)
--> 362         self.sock = ssl_wrap_socket(
    363             sock=conn,
~/miniconda3/lib/python3.8/site-packages/urllib3/util/ssl_.py in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data)
    385         if HAS_SNI and server_hostname is not None:
--> 386             return context.wrap_socket(sock, server_hostname=server_hostname)
~/miniconda3/lib/python3.8/ssl.py in wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
    499         # ctx._wrap_socket()
--> 500         return self.sslsocket_class._create(
    501             sock=sock,
~/miniconda3/lib/python3.8/ssl.py in _create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
   1039                         raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1040                     self.do_handshake()
   1041             except (OSError, ValueError):
~/miniconda3/lib/python3.8/ssl.py in do_handshake(self, block)
   1308                 self.settimeout(None)
-> 1309             self._sslobj.do_handshake()
   1310         finally:
OSError: [Errno 0] Error
During handling of the above exception, another exception occurred:
ProtocolError                             Traceback (most recent call last)
~/miniconda3/lib/python3.8/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    438             if not chunked:
--> 439                 resp = conn.urlopen(
    440                     method=request.method,
~/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
--> 726             retries = retries.increment(
    727                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
~/miniconda3/lib/python3.8/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    409             if read is False or not self._is_method_retryable(method):
--> 410                 raise six.reraise(type(error), error, _stacktrace)
    411             elif read is not None:
~/miniconda3/lib/python3.8/site-packages/urllib3/packages/six.py in reraise(tp, value, tb)
    733             if value.__traceback__ is not tb:
--> 734                 raise value.with_traceback(tb)
    735             raise value
~/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    669             # Make the request on the httplib connection object.
--> 670             httplib_response = self._make_request(
    671                 conn,
~/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    380         try:
--> 381             self._validate_conn(conn)
    382         except (SocketTimeout, BaseSSLError) as e:
~/miniconda3/lib/python3.8/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
    977         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
--> 978             conn.connect()
~/miniconda3/lib/python3.8/site-packages/urllib3/connection.py in connect(self)
--> 362         self.sock = ssl_wrap_socket(
    363             sock=conn,
~/miniconda3/lib/python3.8/site-packages/urllib3/util/ssl_.py in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data)
    385         if HAS_SNI and server_hostname is not None:
--> 386             return context.wrap_socket(sock, server_hostname=server_hostname)
~/miniconda3/lib/python3.8/ssl.py in wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
    499         # ctx._wrap_socket()
--> 500         return self.sslsocket_class._create(
    501             sock=sock,
~/miniconda3/lib/python3.8/ssl.py in _create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
   1039                         raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1040                     self.do_handshake()
   1041             except (OSError, ValueError):
~/miniconda3/lib/python3.8/ssl.py in do_handshake(self, block)
   1308                 self.settimeout(None)
-> 1309             self._sslobj.do_handshake()
   1310         finally:
ProtocolError: ('Connection aborted.', OSError(0, 'Error'))
During handling of the above exception, another exception occurred:
ConnectionError                           Traceback (most recent call last)
<ipython-input-6-7dc3e6722683> in <module>
      4     server = 'https://gtexportal.org/rest/v1/'
      5     ext = f'association/dyneqtl?gencodeId={gene}&variantId={snp}&tissueSiteDetailId={tissue}&datasetId={datasetId}'
----> 6     r = requests.get(server+ext, headers={"Accept" : "application/json"})
      7     if not r.ok:
      8         print(f'--- Request for SNP "{snp}" and "{gene}" returned an error! ---')
~/miniconda3/lib/python3.8/site-packages/requests/api.py in get(url, params, **kwargs)
     75     kwargs.setdefault('allow_redirects', True)
---> 76     return request('get', url, params=params, **kwargs)
~/miniconda3/lib/python3.8/site-packages/requests/api.py in request(method, url, **kwargs)
     59     # cases, and look like a memory leak in others.
     60     with sessions.Session() as session:
---> 61         return session.request(method=method, url=url, **kwargs)
~/miniconda3/lib/python3.8/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    528         }
    529         send_kwargs.update(settings)
--> 530         resp = self.send(prep, **send_kwargs)
    532         return resp
~/miniconda3/lib/python3.8/site-packages/requests/sessions.py in send(self, request, **kwargs)
    642         # Send the request
--> 643         r = adapter.send(request, **kwargs)
    645         # Total elapsed time of the request (approximately)
~/miniconda3/lib/python3.8/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    497         except (ProtocolError, socket.error) as err:
--> 498             raise ConnectionError(err, request=request)
    500         except MaxRetryError as e:
ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
Interestingly, it's getting an error in different points when I re-run the script. For example, the error above occurred using the tissue "Artery_Aorta", but when I  ran again, it got stuck in the tissue "Brain_Cerebellum". Therefore, it seems it's not URL-specific.
I also tried using urllib:
# using urllib
for tissue in tissues:
    print(f'Getting eQTL for tissue {tissue}...')
    server = 'https://gtexportal.org/rest/v1/'
    ext = f'association/dyneqtl?gencodeId={gene}&variantId={snp}&tissueSiteDetailId={tissue}&datasetId={datasetId}'
    r = urllib.request.urlopen(server+ext)
    if r.code != 200:
        print(f'--- Request for SNP "{snp}" and "{gene}" returned an error! ---')
        continue
    decoded = json.load(r)
    r.close()
Then I got:
Getting eQTL for tissue Adipose_Subcutaneous...
Getting eQTL for tissue Adipose_Visceral_Omentum...
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
~/miniconda3/lib/python3.8/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1318             try:
-> 1319                 h.request(req.get_method(), req.selector, req.data, headers,
   1320                           encode_chunked=req.has_header('Transfer-encoding'))
~/miniconda3/lib/python3.8/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1229         """Send a complete request to the server."""
-> 1230         self._send_request(method, url, body, headers, encode_chunked)
~/miniconda3/lib/python3.8/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1275             body = _encode(body, 'body')
-> 1276         self.endheaders(body, encode_chunked=encode_chunked)
~/miniconda3/lib/python3.8/http/client.py in endheaders(self, message_body, encode_chunked)
   1224             raise CannotSendHeader()
-> 1225         self._send_output(message_body, encode_chunked=encode_chunked)
~/miniconda3/lib/python3.8/http/client.py in _send_output(self, message_body, encode_chunked)
   1003         del self._buffer[:]
-> 1004         self.send(msg)
~/miniconda3/lib/python3.8/http/client.py in send(self, data)
    943             if self.auto_open:
--> 944                 self.connect()
    945             else:
~/miniconda3/lib/python3.8/http/client.py in connect(self)
-> 1399             self.sock = self._context.wrap_socket(self.sock,
   1400                                                   server_hostname=server_hostname)
~/miniconda3/lib/python3.8/ssl.py in wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
    499         # ctx._wrap_socket()
--> 500         return self.sslsocket_class._create(
    501             sock=sock,
~/miniconda3/lib/python3.8/ssl.py in _create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
   1039                         raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1040                     self.do_handshake()
   1041             except (OSError, ValueError):
~/miniconda3/lib/python3.8/ssl.py in do_handshake(self, block)
   1308                 self.settimeout(None)
-> 1309             self._sslobj.do_handshake()
   1310         finally:
OSError: [Errno 0] Error
During handling of the above exception, another exception occurred:
URLError                                  Traceback (most recent call last)
<ipython-input-8-d7e4dc39c80e> in <module>
      4     server = 'https://gtexportal.org/rest/v1/'
      5     ext = f'association/dyneqtl?gencodeId={gene}&variantId={snp}&tissueSiteDetailId={tissue}&datasetId={datasetId}'
----> 6     r = urllib.request.urlopen(server+ext)
      7     if r.code != 200:
      8         print(f'--- Request for SNP "{snp}" and "{gene}" returned an error! ---')
~/miniconda3/lib/python3.8/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    220     else:
    221         opener = _opener
--> 222     return opener.open(url, data, timeout)
    224 def install_opener(opener):
~/miniconda3/lib/python3.8/urllib/request.py in open(self, fullurl, data, timeout)
    524         sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
--> 525         response = self._open(req, data)
    527         # post-process response
~/miniconda3/lib/python3.8/urllib/request.py in _open(self, req, data)
    541         protocol = req.type
--> 542         result = self._call_chain(self.handle_open, protocol, protocol +
    543                                   '_open', req)
    544         if result:
~/miniconda3/lib/python3.8/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    500         for handler in handlers:
    501             func = getattr(handler, meth_name)
--> 502             result = func(*args)
    503             if result is not None:
    504                 return result
~/miniconda3/lib/python3.8/urllib/request.py in https_open(self, req)
   1361         def https_open(self, req):
-> 1362             return self.do_open(http.client.HTTPSConnection, req,
   1363                 context=self._context, check_hostname=self._check_hostname)
~/miniconda3/lib/python3.8/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1320                           encode_chunked=req.has_header('Transfer-encoding'))
   1321             except OSError as err: # timeout error
-> 1322                 raise URLError(err)
   1323             r = h.getresponse()
   1324         except:
URLError: <urlopen error [Errno 0] Error>
The same non-URL specificity applies here...
Interestingly, when using the same machine and the same internet, I could successfully ran this (actually, my full code with more genes and SNPs) using R:
library(glue)
library(httr)
library(jsonlite)
gene = "NOS1AP" 
snp = "rs10918593"
datasetId='gtex_v7'
tissues <- c('Adipose_Subcutaneous', 'Adipose_Visceral_Omentum', 'Adrenal_Gland', 'Artery_Aorta', 'Artery_Coronary', 'Artery_Tibial', 'Brain_Amygdala', 'Brain_Anterior_cingulate_cortex_BA24', 'Brain_Caudate_basal_ganglia', 'Brain_Cerebellar_Hemisphere', 'Brain_Cerebellum', 'Brain_Cortex', 'Brain_Frontal_Cortex_BA9', 'Brain_Hippocampus', 'Brain_Hypothalamus', 'Brain_Nucleus_accumbens_basal_ganglia', 'Brain_Putamen_basal_ganglia', 'Brain_Spinal_cord_cervical_c-1', 'Brain_Substantia_nigra', 'Breast_Mammary_Tissue', 'Cells_EBV-transformed_lymphocytes', 'Cells_Transformed_fibroblasts', 'Colon_Sigmoid', 'Colon_Transverse', 'Esophagus_Gastroesophageal_Junction', 'Esophagus_Mucosa', 'Esophagus_Muscularis', 'Heart_Atrial_Appendage', 'Heart_Left_Ventricle', 'Liver', 'Lung', 'Minor_Salivary_Gland', 'Muscle_Skeletal', 'Nerve_Tibial', 'Ovary', 'Pancreas', 'Pituitary', 'Prostate', 'Skin_Not_Sun_Exposed_Suprapubic', 'Skin_Sun_Exposed_Lower_leg', 'Small_Intestine_Terminal_Ileum', 'Spleen', 'Stomach', 'Testis', 'Thyroid', 'Uterus', 'Vagina', 'Whole_Blood')
for(tissue in tissues) {
  print(glue('Getting eQTL for tissue {tissue}...'))
  server = 'https://gtexportal.org/rest/v1/'
  ext = glue('association/dyneqtl?gencodeId={gene}&variantId={snp}&tissueSiteDetailId={tissue}&datasetId={datasetId}')
  r <- GET(paste0(server, ext), content_type("application/json"))
  if(r$status_code!=200){
    print(glue('--- Request for SNP "{snp}" and "{gene}" returned an error! ---'))
  decoded <- fromJSON(toJSON(content(r)))
Although it works in R, I've already organized a script in Python, and I really would like to use requests and urllib normally...
Also, the versions I used:
python==3.8.2
requests==2.24.0
Any help, please?
                I've tried the above code today... and it worked as expected! I still don't know why I was getting such error...
– Cainã Max Couto-Silva
                Oct 31, 2020 at 20:31
I bumped into this problem recently, there are few things we need to get out of way before we head to some debugging that's not wasting our time.
Firstly, requests 2.24 requires urllib3 < 1.26 and in urllib3 1.25, the wrap_socket call was not from a SSLContext; this seemed to cause problem when making SSL handshake; in urllib3 1.26 the team has changed it to context.wrap_socket, which seemed to solve the issue.
Secondly, remember after you upgrade urllib3, update your requests to the latest as well.
Then upgrade or install the latest httplib2, it was silently causing problem for me until I found someone on SO shouted it out.
After all this, one of the main issue that's causing Errno 0 error, is the flaky TCP connection; this is something, most of time, out of our control..
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.