Forums

In my web app, I have integrated code to scrape a website. It works fine the first time. It's supposed to scrape real-time data from another website and since it always changes, I want it to change every time someone clicks on my web app. Does that make sense? It's supposed to scrap and provide current data but even after reload it won't scrap the new data. I tried the /reload but it's still not working. It only works again if I click the reload button in the web app dashboard. I'm sorry I'm very new to this, can someone help me

One solution would be to schedule a cron job. It runs the script after every x amount of time and you could store the result in a database or a file and display it to the user. But, if your data is ever-changing, a better option would be to fetch the website and scrape the data only when the user requests your website. If you do not need scraped data to be stored, you can parse it and directly display it to the user at the same instance itself. Hope this helps :)

Bruhadroop is heading you the right direction. If you do scraping in a scheduled task and store the results in the database you will not need to reload your web app to keep it up to date.

I already have a scheduler for scraping daily data. However, this particular data that I want to scrap changes every minutes. Is it possible to scrap the website by placing the scraping code in the wsgi.py file ? Like before calling the flask app. Would that work ? How would I pass the data from the wsgi.py to my actual flask_app.py

you could potentially just put the scraping code in the flask app endpoint response code itself.

putting it in the wsgi file before importing the flask app wouldn't work- it would only run on startup.

The scraping code IS in the flask app. It scraps only once, after the the web app is reloaded. I'm sorry if I'm being annoying but what is the endpoint response ?

seems like it's run once from the flask app, not every time an endpoint is hit. put it inside of the function that responds to that @route url endpoint so it's run every single time.

how long does the scraping take btw?

The scraping only takes around 1 - 2 seconds. I'll try putting it in @route url next and see if that works.

Yes, that should work. Remember that we don't run all of your code on each hit. Your Flask script is run just once, when it is started up. When requests come in later on, only the view function related to that URL is called -- not code that appears outside functions.