Forums

Unable to use urllib and beautiful soup 4

So I have a website where I need to be able to web scrape using urllib and bs4, but I am getting a 403 error, even though the website I am trying to access is served through https. How do I solve this issue?

Hi there,

Free users do not get unlimited external access and can only access a whitelist. We are happy to add the domain to the whitelist if it is a official api endpoint made public to everyone. Otherwise you will have to upgrade.

Ok, so the website's URL is 'https://www.ndtv.com'. Can it be added?

Please see this page for details on what our criteria are for adding new sites to the whitelist, and what information you need to provide: https://help.pythonanywhere.com/pages/RequestingWhitelistAdditions/