Forums

Hello,

I would like to run a script continuously. The initial script is like this:

1. connectionDB = sqlite3.connect("database.db")
2. cursorDB = connectionDB.cursor()
3. while True:
4. insert an item in the sqlite database
5. sleep(5minutes)
6. connectionDB.commit() and go back to step 3

So a scheduled task seems appropriate. I used to be free user and run this script daily. However, the task was killed after approximately 3 hours. Now I upraded to a payer account and I tried to use the exact same script but running the task hourly. However, in the task log, I get this approximately every hour:

sqlite3.OperationalError: database is locked

For me it's either due to a combination of the following:

• The task is not killed exactly after 1 hour so that the 2nd task cannot complete.
• I don't close the sqlite connection (because the initial script is an infinite loop)

So how should I do to run this script (that pulls data from the web and insert it into a database every 5minutes) ? Shall I modify the script itself by replacing the "while" by a "for" with a number of iterations equals to int(60/5), close the database connection at the end of the script, and run it hourly ?

Thank you

have you seen this page?

https://help.pythonanywhere.com/pages/LongRunningTasks

it has a suggestion for how to create a lock to ensure that two copies of the same task can't be running at the same time. using that you could make it so that it's just one script that runs forever, except if it gets killed in which case it is restarted on the next hourly run...

• Ok so this is the script in the link that should be run as a task right (and this task will call my initial script)?
• When you say

except if it gets killed in which case it is restarted on the next hourly run...

Do you mean I should set this unique task mentioned above hourly ? Also, the fact that I don't have any gap in my database is important. So if I understand you well, if for some reason the task is killed at +1second, then I will have to wait the next hour and miss all the data from +1" to +59" right ?

Tx

Hi there, you've understood correctly yes. Some people schedule several hourly tasks (eg 6 of them at 10-minute intervals) to try and avoid any downtime.

We are working on a better solution for always-on tasks.

Ok so i will do the following: 12 hourly tasks (at 5 minutes interval) running the script in the link that will call my initial script. With this solution, the amount of data that I could miss in case if 1 task is killed is reduced.

Right, that should work OK. Would you like us to notify you when we have the always-on functionality working? We have a beta test running, but there are some bugs we need to sort out before inviting more people to the beta.

Yes sure. Btw, although I get the error "database is locked", after inspecting my database, it is correctly filled (I thought this error would stop the script until the next hour). Tx

That's odd -- are you using the code from the page at https://help.pythonanywhere.com/pages/LongRunningTasks to make sure you don't have multiple instances running? Or something else?

Hello, trying to post a question into my dilemma so that I can adjust accordingly. My intent is to have my twitter streamer run continuously. Currently, I just started with PythonAnywhere and won't my initial first python script and run it successfully as a task. It seems to end after 3 Hours (almost). It had an expiry date for 1 month. My expectations was to just "refresh" the expiry date as it closer to expiration.

Before just doing the paid option (and unsure if that will just fix me), I like to understand what I am missing. 1) Do schedule tasks just complete after three hours? 2) What is the purpose of the expiry date? 3) Is there anything I need to consider aside from the task naturally ending to allow from my free account to have a task run for 30 days?

1) Hard to say. What was the return code?

2) Tasks on free accounts have expiry date to prevent abandoned code to be run.

3) You need to consider CPU usage and disk usage. You also need to consider checking logs from time to time or setting some notification to see if your task works as intended. etc.

1) return code is 137 "2019-10-24 00:08:04 -- Completed task, took 8563.00 seconds, return code was 137" -- Seems like the Task ran to a completion. 2) Expiry date is 1 month, not three hours as shown by the # seconds. 3) Task is working as intended, building a .json file of tweepy implemented streamer tweets.

Trying to determine how it runs its course... internal timeout? when no tweets pushed after a certain time?

Again, I thought that I wrote a Task that runs indefinitely to collected Stream tweets (which it does when up), but it ends after an approx. 3 hours (or seconds). What am I missing?

Hi, just to clarify, free users have scheduled tasks, which do get stopped due to maintenance. Schedule tasks are meant to be for tasks that say run every day and take 5min to run and then stop. That long running task page contained an old recommendation, but now if you want to run an always on task (eg: a twitter streamer that runs continuously), then you should upgrade and use an always on task. Otherwise be prepared for it to not work well from free accounts.

Will give the paid account a trial to see if it stays up. I misunderstood thinking the expiry date was the date that it would be killed (naturally) if it was left unattended.

Ah sorry for the confusion, the scheduled tasks is a separate concept vs the always on tasks.

The scheduled tasks are more of a trigger it every day/every hour type thing, vs the always on tasks are tasks that we will automatically spin up for you again if it ever gets killed.