Max retries exceeded with url on paid account : Forums : PythonAnywhere

Max retries exceeded with url on paid account

Hi all

I have a paid account and I am seeing the following errors - New Connection Error, Max Retry Error, Connection Refused Error:

ConnectionRefusedError: [Errno 111] Connection refused

NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9792a553c0>: Failed to establish a new connection: [Errno 111] Connection
 refused

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=58979): Max retries exceeded with url: /session/53172be96dc2e8d7198cd6da616d308e/elem ent (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9792a553c0>: Failed to establish a new connection: [Errno 111] Connection ref used'))

I am using selenium and assume that I am getting the above issues as the website: https://hudoc.echr.coe.int/eng is blocking access to Python Anywhere specifically. I am running the same code on my local machine with absolutely no issues.

The following are the Chrome Options in my code:

options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("--disable-extensions")
options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-gpu")
options.add_argument("--user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36'")
options.add_experimental_option('useAutomationExtension', False)
options.add_experimental_option("excludeSwitches", ["enable-automation"])

Thanks for any and all help :)

deleted-user-13584695 | 3 posts | Sept. 22, 2022, 8:18 a.m. | permalink

That message is usually because chrome is slow or crashed. See our help page for suggested arguments: https://help.pythonanywhere.com/pages/selenium/

glenn | 9718 posts | PythonAnywhere staff | Sept. 22, 2022, 3:52 p.m. | permalink

I've followed all the steps in that guide and it does not work. My selenium version is 4.1.5. And all the options are up, including changing the user agent.

The script previously worked, then it stopped working, I think the focus of the issue is:

Max retries exceeded with url: /session/53172be96dc2e8d7198cd6da616d308e/elem

deleted-user-13584695 | 3 posts | Sept. 22, 2022, 4:20 p.m. | permalink

Yes, that is the focus of the problem. Your selenium code is unable to connect to chromedriver to drive it. Have you disabled dev shm usage? Also, make sure that you are not in the tarpit.

glenn | 9718 posts | PythonAnywhere staff | Sept. 22, 2022, 4:57 p.m. | permalink

Definitely not in the tarpit and have disabled dev shm usage, my options are:

options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("--disable-extensions")
options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-gpu")
$options.add_argument("user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36'")
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-dev-shm-usage')
options.add_experimental_option("excludeSwitches", ["enable-automation"])

Could it be something with the website detecting the bot/scraper? If so, any suggestions on how I could skip that detection?

deleted-user-13584695 | 3 posts | Sept. 23, 2022, 9:53 a.m. | permalink

It could be, but another possibility is that Chrome is using >3GiB of RAM trying to load that page. You would normally get an email telling you about processed running out of RAM if you're subscribed to resource warning emails (which it looks like you are) -- perhaps those are going out but winding up in spam?

Another thing I noticed while looking at your account is that you're out of disk space; perhaps that is causing a Chrome crash?

giles | 12074 posts | PythonAnywhere staff | Sept. 24, 2022, 3:03 p.m. | permalink