Forums

scheduled task

Hello,

I would like to run a script continuously. The initial script is like this:

  1. connectionDB = sqlite3.connect("database.db")
  2. cursorDB = connectionDB.cursor()
  3. while True:
  4. insert an item in the sqlite database
  5. sleep(5minutes)
  6. connectionDB.commit() and go back to step 3

So a scheduled task seems appropriate. I used to be free user and run this script daily. However, the task was killed after approximately 3 hours. Now I upraded to a payer account and I tried to use the exact same script but running the task hourly. However, in the task log, I get this approximately every hour:

sqlite3.OperationalError: database is locked

For me it's either due to a combination of the following:

  • The task is not killed exactly after 1 hour so that the 2nd task cannot complete.
  • I don't close the sqlite connection (because the initial script is an infinite loop)

So how should I do to run this script (that pulls data from the web and insert it into a database every 5minutes) ? Shall I modify the script itself by replacing the "while" by a "for" with a number of iterations equals to int(60/5), close the database connection at the end of the script, and run it hourly ?

Thank you

have you seen this page?

https://help.pythonanywhere.com/pages/LongRunningTasks

it has a suggestion for how to create a lock to ensure that two copies of the same task can't be running at the same time. using that you could make it so that it's just one script that runs forever, except if it gets killed in which case it is restarted on the next hourly run...

  • Ok so this is the script in the link that should be run as a task right (and this task will call my initial script)?
  • When you say

except if it gets killed in which case it is restarted on the next hourly run...

Do you mean I should set this unique task mentioned above hourly ? Also, the fact that I don't have any gap in my database is important. So if I understand you well, if for some reason the task is killed at +1second, then I will have to wait the next hour and miss all the data from +1" to +59" right ?

Tx

Hi there, you've understood correctly yes. Some people schedule several hourly tasks (eg 6 of them at 10-minute intervals) to try and avoid any downtime.

We are working on a better solution for always-on tasks.

Ok so i will do the following: 12 hourly tasks (at 5 minutes interval) running the script in the link that will call my initial script. With this solution, the amount of data that I could miss in case if 1 task is killed is reduced.

Right, that should work OK. Would you like us to notify you when we have the always-on functionality working? We have a beta test running, but there are some bugs we need to sort out before inviting more people to the beta.

Yes sure. Btw, although I get the error "database is locked", after inspecting my database, it is correctly filled (I thought this error would stop the script until the next hour). Tx

That's odd -- are you using the code from the page at https://help.pythonanywhere.com/pages/LongRunningTasks to make sure you don't have multiple instances running? Or something else?