Hi, conrad! So, I finally took the time to investigate it further. Apparently, almost 1 minute is spent in this command:
So, it appears to be a problem with the loading of the page. Sometimes it is relatively fast (12s to load) and everything works fine, but sometimes it can be 4 times that, and that's where my app stops working properly, since the browser cuts the connection if the API doesn't respond in 30s.
Depending on how many people you expect to be using your site, it may not make sense to keep the scraper running directly off of the website- this is because if say you have 4 workers, then if you get 4 requests that do scraping, everyone else would be blocked for 25s-3min until the 4 requests are done. And say you get 8 requests that do scraping, then everyone else would be blocked for 50s-6min etc. Instead you probably want to offload that scraping stuff to say an always on task, and just have the web request return with a "processing" type of message until it is done.
This actually is not a problem, because this operation is used very sparsely. Since it won't be activated frequently, I don't see it being a burden.