Forums

Hello,

I am trying to create an app that processes an uploaded CSV, appends data from various APIs, then returns the spreadsheet with the new data. These spreadsheets sometimes have hundreds of rows, so the processing time is typically around ~5-10 minutes when I run these scripts locally.

I understand the 504 load-balancer error occurs when processes take longer than a certain amount of time. I also saw there's the option of creating an always-on task, which I've attempted, but I'm not sure if I'm doing it incorrectly, or if this type of task is not meant for what I'm trying to achieve.

Any suggestions for what to do here? Thanks!

An always-on task is definitely the right solution for this kind of problem. What kind of issues did you have when setting one up?

After adding the always-on task, it gets stuck at "Starting". Then, when I look at the error log, nothing comes up, so I'm not sure where it's freezing.

If a task is stuck in starting, it's likely that it's exiting as soon as it starts (which means that there's a tiny window of time when it's running, but mostly it's waiting for the system to restart it after it -- as far as the system is concerned -- has crashed).

Roughly what code do you have in the script you're setting up as an always-on task?

Hmm, OK I see. The website works alright when the CSV's I upload are shorter (~10-15 rows), so I wonder why it would be crashing when run as a task.

The structure of the code basically follows the final step under this tutorial: https://blog.pythonanywhere.com/169/

The main difference is just how the files are processed.

How long does it take to process the shorter CSVs?

Usually under a minute, so they work fine with just running the regular site (not adding the script as an always-on task)

That would work fine if you don't have high traffic.

The problem is the shorter csv's are just a test-run. The vast majority of use cases for this app will be for CSV's with hundreds of rows, which is when the app gets killed because the request takes too long, hence the need for an always-on task

The problem is the shorter csv's are just a test-run. The vast majority of use cases for this app will be for CSV's with hundreds of rows, which is when the app gets killed because the request takes too long, hence the need for an always-on task

Could you give some more details about the structure of the script you're using for the always-on task? Or alternatively, can we take a look at your files? We can see them from our admin interface, but we always ask for permission first.

Yep, sure, please go ahead and look at the files.

Ah, OK -- I see the problem -- you've just scheduled your Flask code as an always-on task. You need to do a bit more than that. I'd got the impression that you'd got the idea of using a task from a specific help page, which has more details, but I guess it was from a forum post or something like that. Here's the help page with information on how to set things up properly.

Got it, OK, thanks very much for the helpful info and the link! I'll give this a try.