Forums

hey there,

I just started using PythonAnywhere to do a basic web crawler to sync user information when they give us their uid from another site. To do this, I have bought the 100 Squid proxies server. However, I was wondering if we really need to buy the proxies at all if we were to do this over python anywhere? Our need is like at most 3-5 url requests per user that decided on sync. As we are still in bootstrapping phrase and would like to save as much cost as possible. (I am also paid for PythonAnywhere account as well). So i hope you guys could give me some insight on this

Best

I'm not quite sure I understand what you wanted to use a squid proxy for, but it sounds to me like you could happily do your web scraping from pythonanywhere, if you have a paid account... Why not try it?

You'll probably want to use a scheduled task if you need the syncing to happen on a regular basis...

Ah, before I use python anywhere, I was running it on my own machine so I brought the proxy to prevent getting block by the server in which I'm requesting from. Anyway, thank you so much. That helps alot

Ah. OK, well, we'd rather you didn't do anything that would get our server IPs blocked by any third party websites?

sure thing :) Back then I was generated the initial data set. So the request rate is a lot higher. Now it is basically generated on-demand. So the number of request per minute/hour should be very low. I don't think we will get block from the requests (as I have described earlier). But I will take it into consideration and re-implementing the proxies code if the need for scaling ever arise. Best