Forums

So I have a website where I need to be able to web scrape using urllib and bs4, but I am getting a 403 error, even though the website I am trying to access is served through https. How do I solve this issue?

Hi there,

Free users do not get unlimited external access and can only access a whitelist. We are happy to add the domain to the whitelist if it is a official api endpoint made public to everyone. Otherwise you will have to upgrade.

Ok, so the website's URL is 'https://www.ndtv.com'. Can it be added?

Please see this page for details on what our criteria are for adding new sites to the whitelist, and what information you need to provide: https://help.pythonanywhere.com/pages/RequestingWhitelistAdditions/

Hello! I'm posting here because of the title "Unable to use urllib and beautiful soup 4" although my problem is not the same. My problem is: urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate ( _ssl.c:1045)> .

I have followed instructions after getting my API Token, I set up "letsencrypt" in my home directory. I followed all the instructions here successfully. https://help.pythonanywhere.com/pages/LetsEncrypt/ . I run my beautiful soup script (which is inside my project directory). and it is only a little script to see the soup and I get the message that my ssl certificate verify failed.

I installed pip install --upgrade certifi as recommended by someone in the internet... but no luck. CAN YOU PLEASE HELP ME?

I don't understand what beautifulsoup has to do with a letsencrypt certificate. What are you trying to do with your script?