I'm scraping data from a website in a scheduled task and when I try to download it I get:
HTTPError: 403 while retrieving
I can't figure out why this is happening since I have a paid account, and when I run the same script straight from the console and the same virtualenv it works fine and downloads the data. If it helps to fetch the data I'm using urlopen but I don't see how the code can be the problem since it works fine from the console.
The only thing I could possibly think of is that the request is somehow blocked from the tasks server on pythonanywhere, is this possible? It seems unlikely since I'm pulling public data that is intended to be used for things like this...
I'm calling the task like this:
source virtualenvwrapper.sh && workon myvirtualenv && /home/myname/myfolder/scrape
Can anyone help me fix this? Thanks!
EDIT: Just realized this could be important to note, but I am running a bash script, not a python script. If I call a bash script from a scheduled python script to run the command could that possibly help? Or should I try importing subprocess and try calling the bash command from a python script?