Forums

Webscraping Facebook Not Working on Paid Account

Hi, I have a web scraper that I programmed and tested on my machine, which scrapes Facebook data using Requests, Regex, and BeautifulSoup. When I run it on my local machine, the scraper can extract the data I need. However when it is run on PythonAnywhere, it extracts from some completely different page (I suspect the login page), and therefore the scraper is not able to extract the data I need.

Any way I can get around this? I have already been using random proxies and imitating an HTTP request from a regular browser. Kind of bummed because I bought a custom plan just for this.

I can provide code if needed, but it works perfectly on my machine so I don't think that would be the issue.

Appreciate it!

You probably need to be logged into facebook to reach the page you want to scrap.

I don't think you do. I tried the same url on an incognito tab and it works fine. Also the scraper works on my device so shouldn't it work on PythonAnywhere the same way?

Would really appreciate some guidance.

Thank you

Have you tried saving the page that it gets stuck on and viewing it in your local browser? It might be a login / redirect page or something

Ok, so I believe it is a login page, which sucks. Is there any reason this happens on PythonAnywhere but not on my device? And do you know any way to circumvent this?

Thanks

Is the selenium on your local machine using a browser which has a logged in session for facebook, such as your regular chrome browser?

No I am using python requests.

It could be Facebook blocking programmatic access from known headless servers, and as PythonAnywhere is running in AWS data center it could be the case.