I am in the process of writing a django application that scrapes auctions websites such as ebay, gumtree, preloved etc. This is written and working on my own 'home' development environment using the built in django runsever. It utilises beautiful soup and urllib in order to request and subsequently scrape these websites.
This program was successfully uploaded to PythonAnywhere and running beautifully. (thanks guys, so much neater, cleaner and intuitive that Amazon Web Services ElasticBeanstalk offering)
The issue I am experiencing is with ebay. The urllib request call does not return any results when utilised from PythonAnywhere. This was originally due to the expected blocks on certain non-white listed sites, but I coughed up and paid my $6...
I cannot work out why this is not working, the request is successfully executed; but the results are not there.
One such example of a urlcall would be:
https://www.ebay.co.uk/sch/i.html/search?_from=R40&_sacat=0&_nkw=red+bike&rt=nc&_sadis=15&_fspt=1&LH_PrefLoc=99&_pgn=1&_stpos=s662ry**
This is the correct URL for ebay to search for red bikes within 15 miles of postcode s662ry.
As stated, this works no issues when calling urlib from my home environment. It returns zero results when exactly the same code is run from within PythonAnywhere.
I am pretty sure the setup is the same; django v2, python3.6, bs4 and urllib.
I have tried this with a simple requests.get(), and the same error occurs on PythonAnywhere; but it works in development environment.
Any help or pointers, please?
As an aside, the PythonAnywhere verions runs a lot, lot slower. Can anyone indicate how the speed can be improved. (currently using urllib and bs4 as html.parser)
Thanks in advance Matthew
harry
|
2710
posts
|
PythonAnywhere staff
|
