Forums

Hi, I developed a small scraper, that scrapes gas prices off of the internet. I recently had it running on repl.it, but i need something more permanent.

My general structure is that I have a main file, which does the scraping every like 10 minutes and a subfile, that has a Flask script in it. In the Flask script are some dynamic variables, that get set from time to time by the main script. Here is my first issue:

I have an example Flask script ("webapp.py") running as a Webapp (file and working directory is "username/flasktest", WSGI is set correctly) :

import flask
testvar = "old"
app = flask.Flask(__name__)
@app.route("/")
def home_func():
    return f"testvar: {testvar}"

But when I open a Bash console in this folder and do (to simulate a future main script):

18:58 ~/flasktest $ python3
Python 3.8.0 (default, Nov 14...                                                                                                                                                            
>>> import webapp
>>> webapp.testvar
'old'                                                                                                                                     
>>> webapp.testvar = "new"
>>> webapp.testvar                                                                                                                       
'new'                                                                                                                                     
>>>

However, when I reload my webpage, the changed variable remains at value "old". What am I doing wrong here? It is mandatory to set these variables, because it's mandatory that the Flask app can only open and write to files in specific time spaces that get set by the main script.

Greetz

You're using a global variable testvar which is set when the web app is loaded by WSGI -- each time you reload the web app, it will be reset to this value, see also this help page.

Also -- you will most probably will not be able to scrape using a free account and if you need a script that is running constantly, you should consider using the Always-on task feature, since our consoles are ephemeral by design.

If you've got a web app & script model, you may also consider changing the architecture a bit, like in the example mentioned here.