Forums

Requests error

import requests

URL = 'https://ya.ru'
HEADERS = {'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'}

r = requests.get(URL, headers=HEADERS)
print(r)

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 594, in urlopen
    self._prepare_proxy(conn)
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
    conn.connect()
  File "/usr/lib/python3.8/site-packages/urllib3/connection.py", line 308, in connect
    self._tunnel()
  File "/usr/lib/python3.8/http/client.py", line 898, in _tunnel

 raise OSError("Tunnel connection failed: %d %s" % (code,
OSError: Tunnel connection failed: 403 Forbidden
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 637, in urlopen
    retries = retries.increment(method, url, error=e, _pool=self,
  File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ya.ru', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidd
en')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/MihailSegeev/Parser.py", line 12, in <module>
    r = requests.get(URL, headers=HEADERS)
  File "/usr/lib/python3.8/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 510, in send
    raise ProxyError(e, request=request)

requests.exceptions.ProxyError: HTTPSConnectionPool(host='ya.ru', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden
')))

Free accounts are limited in their internet access. See https://www.pythonanywhere.com/whitelist/

I'm also trying to add headers to my .get() call and the function never seems to return - I have to interrupt it. This code runs on my local Windows machine fine.

This is the code that never returns:

html = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})

This does return:

html = requests.get(url)

Any idea what is going on here? I just upgraded to a paid account, too.

what is the url? If you just upgraded, you would need to start new consoles.

The URL I'm trying to use is http://www.bestbuy.com. I tried restarting the console and I'm still getting the same thing. When I ctrl-c to stop it, this is the traceback I get:

Python 3.8.0 (default, Nov 14 2019, 22:29:45) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> url = "http://www.bestbuy.com"
>>> html = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
^CTraceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 377, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 597, in urlopen
    httplib_response = self._make_request(conn, method, url,
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 264, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt

url is "http://www.bestbuy.com"

And I tried restarting the console too

This is the traceback if I let it run:

Traceback (most recent call last):

File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 597, in urlopen httplib_response = self._make_request(conn, method, url, File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 384, in _make_request six.raise_from(e, None) File "<string>", line 2, in raise_from File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 380, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.8/http/client.py", line 1322, in getresponse response.begin() File "/usr/lib/python3.8/http/client.py", line 303, in begin version, status, reason = self._read_status() File "/usr/lib/python3.8/http/client.py", line 264, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib/python3.8/socket.py", line 669, in readinto return self._sock.recv_into(b) File "/usr/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 302, in recv_into raise SocketError(str(e)) OSError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 637, in urlopen retries = retries.increment(method, url, error=e, _pool=self, File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 368, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/lib/python3.8/site-packages/urllib3/packages/six.py", line 685, in reraise raise value.with_traceback(tb) File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 597, in urlopen httplib_response = self._make_request(conn, method, url, File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 384, in _make_request six.raise_from(e, None) File "<string>", line 2, in raise_from File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 380, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.8/http/client.py", line 1322, in getresponse response.begin() File "/usr/lib/python3.8/http/client.py", line 303, in begin version, status, reason = self._read_status() File "/usr/lib/python3.8/http/client.py", line 264, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib/python3.8/socket.py", line 669, in readinto return self._sock.recv_into(b) File "/usr/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 302, in recv_into raise SocketError(str(e)) urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError("(104, 'ECONNRESET')"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.8/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, kwargs) File "/usr/lib/python3.8/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(104, 'ECONNRESET')"))

It looks like http://www.bestbuy.com is blocking requests from cloud computing environments -- some sites do that in order to prevent web-scraping.

Hi is there any solution to prevent a server blocking requests from cloud environments?

@temi95 no, it's all on their side.