Performance Issue...:( : Forums : PythonAnywhere

Performance Issue...:(

I'm sorry to be the one to bring up a performance issue, but I happen to have stumbled across one.

I have a silly script that is scheduled to run daily at 23:59. The purpose of this script running is so I can keep an eye on how the Schedule tab is working. As you can see by this code, the script doesn't do much:

#!/usr/local/bin/python3.2

import sys, os

print(sys.executable, os.sys.executable)
print(sys.argv)

The command line configured on the Schedule tab is:

cookbook/args.py n o t h i n g

So, the issue is that because it doesn't do much, one would pretty much expect the script to finish in near 0 seconds. Take look at the following log entries for the script. It has taken as long as 1420 seconds to complete.

 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-13 23:59:21 -- Completed task, took 3.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-14 23:59:18 -- Completed task, took 3.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-15 23:59:25 -- Completed task, took 4.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-16 23:59:22 -- Completed task, took 3.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-17 23:59:47 -- Completed task, took 2.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-18 23:59:28 -- Completed task, took 6.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-19 23:59:24 -- Completed task, took 3.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-20 23:59:29 -- Completed task, took 4.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-21 23:59:41 -- Completed task, took 7.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-22 23:59:34 -- Completed task, took 4.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-23 23:59:26 -- Completed task, took 4.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-24 23:59:22 -- Completed task, took 4.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-25 23:59:30 -- Completed task, took 7.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-26 23:59:42 -- Completed task, took 10.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-27 23:59:28 -- Completed task, took 7.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-28 23:59:27 -- Completed task, took 6.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-29 23:59:29 -- Completed task, took 8.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-30 23:59:53 -- Completed task, took 11.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-07-31 23:59:39 -- Completed task, took 10.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-02 00:00:22 -- Completed task, took 20.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-02 23:59:29 -- Completed task, took 8.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-03 23:59:31 -- Completed task, took 8.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-04 23:59:25 -- Completed task, took 7.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-05 23:59:32 -- Completed task, took 10.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-06 23:59:37 -- Completed task, took 11.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-07 23:59:47 -- Completed task, took 12.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-08 23:59:37 -- Completed task, took 11.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-09 23:59:43 -- Completed task, took 12.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-10 23:59:49 -- Completed task, took 13.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-11 23:59:41 -- Completed task, took 11.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-12 23:59:44 -- Completed task, took 12.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-13 23:59:46 -- Completed task, took 15.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-15 00:00:48 -- Completed task, took 31.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-15 23:59:29 -- Completed task, took 7.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-16 23:59:30 -- Completed task, took 8.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-17 23:59:37 -- Completed task, took 9.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-18 23:59:39 -- Completed task, took 9.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-19 23:59:40 -- Completed task, took 11.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-20 23:59:40 -- Completed task, took 12.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-22 00:01:39 -- Completed task, took 51.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-22 23:59:36 -- Completed task, took 10.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-23 23:59:35 -- Completed task, took 10.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-25 00:23:27 -- Completed task, took 1420.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-25 23:59:34 -- Completed task, took 9.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-26 23:59:49 -- Completed task, took 13.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-27 23:59:41 -- Completed task, took 11.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-28 23:59:42 -- Completed task, took 11.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-30 00:01:29 -- Completed task, took 112.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-08-31 00:01:54 -- Completed task, took 132.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-01 00:00:22 -- Completed task, took 49.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-01 23:59:56 -- Completed task, took 16.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-03 00:01:23 -- Completed task, took 106.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-04 00:00:18 -- Completed task, took 28.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-05 00:00:27 -- Completed task, took 41.00 seconds, return code was 0.

If I can provide any other useful information, please let me know.

a2j | 684 posts | Sept. 5, 2012, 7:37 a.m. | permalink

Wow! That's terrible. I'll have a look at that today and see what's going on. Thanks for pointing that out, a2j.

glenn | 9718 posts | PythonAnywhere staff | Sept. 5, 2012, 10:01 a.m. | permalink

I'm glad to be of service. I just wish it was better news...☺

a2j | 684 posts | Sept. 5, 2012, 10:42 a.m. | permalink

a2j,

I have this sneaking suspicion that 23:59 and 00:00 are by far the most popular time for scheduled tasks to run. It may simply be that there are 1,000 scheduled tasks all being fired up at the same time...

rcs1000 | 311 posts | Sept. 5, 2012, 1:50 p.m. | permalink

@rcs1000: Good point. I've randomly:

print(random.randint(0, 23), ':', random.randint(0, 59), sep='')

chosen a new time and added it to my Schedule. I now have a daily task also running at 14:21, so we'll see the results of that as well in a few days.

a2j | 684 posts | Sept. 5, 2012, 5:19 p.m. | permalink

Actually I also have another task that runs hourly at :29. The relevant parts of the log for that task are:

2012-09-03 15:29:11 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-03 16:29:10 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-03 17:29:14 -- Completed task, took 4.00 seconds, return code was 0.
2012-09-03 18:29:11 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-03 19:29:11 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-03 20:29:16 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-03 21:29:12 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-03 22:29:13 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-03 23:29:18 -- Completed task, took 6.00 seconds, return code was 0.
2012-09-04 00:29:14 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-04 01:29:15 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-04 02:29:18 -- Completed task, took 4.00 seconds, return code was 0.
2012-09-04 03:29:13 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-04 04:29:11 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 05:29:13 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-04 06:29:11 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 07:29:25 -- Completed task, took 9.00 seconds, return code was 0.
2012-09-04 08:29:12 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 09:29:13 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-04 10:29:17 -- Completed task, took 4.00 seconds, return code was 0.
2012-09-04 11:29:11 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-04 12:29:12 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-04 13:29:11 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-04 14:29:21 -- Completed task, took 4.00 seconds, return code was 0.
2012-09-04 15:29:14 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 16:29:11 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 17:29:12 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 18:29:12 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-04 19:29:13 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 20:29:17 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 21:29:18 -- Completed task, took 4.00 seconds, return code was 0.
2012-09-04 22:29:12 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-04 23:29:14 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-05 00:29:23 -- Completed task, took 6.00 seconds, return code was 0.
2012-09-05 01:29:17 -- Completed task, took 4.00 seconds, return code was 0.
2012-09-05 02:29:15 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-05 03:29:14 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-05 04:29:22 -- Completed task, took 5.00 seconds, return code was 0.
2012-09-05 05:29:15 -- Completed task, took 4.00 seconds, return code was 0.
2012-09-05 06:29:13 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-05 07:29:13 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-05 08:29:20 -- Completed task, took 6.00 seconds, return code was 0.
2012-09-05 09:29:13 -- Completed task, took 2.00 seconds, return code was 0.
2012-09-05 10:29:16 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-05 11:29:16 -- Completed task, took 4.00 seconds, return code was 0.
2012-09-05 12:29:15 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-05 13:29:13 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-05 14:29:28 -- Completed task, took 10.00 seconds, return code was 0.
2012-09-05 15:29:10 -- Completed task, took 3.00 seconds, return code was 0.
2012-09-05 16:29:11 -- Completed task, took 3.00 seconds, return code was 0.

The log above is for the same underlying script as before, with a slightly different command line:

cookbook/args.py Woot "woot" 14 7 83 42 "of COURSE"

As you can see it does run more regularly, but still takes up to 10 seconds to complete.

a2j | 684 posts | Sept. 5, 2012, 5:30 p.m. | permalink

Interesting. I think there are 2 issues, here

Users tend to run things at minute/hour boundaries
When we start a lot of scheduled tasks, they all get held-up clamouring for resources

We can probably alleviate this a bit by adding a bit of jitter in the start times of scheduled tasks to reduce the collisions, but the main thing we need to do is identify the resource that's causing the bottleneck and providing more of it.

I'll keep you posted about the progress of my investigations.

glenn | 9718 posts | PythonAnywhere staff | Sept. 5, 2012, 5:55 p.m. | permalink

Perhaps in addition to allowing us to choose our hourly & daily start times, PA should add a generic daily and hourly schedule option. The idea being where the user doesn't require a particular schedule, but just the routine of the task doing its work. Then behind the scenes PA could run the tasks linearly from a queue to manage resource spikes.

The user just knows their task will run daily/hourly...not at what minute. The queue could actually be paused during peak demand.

I prefer this to adding jitter for the tasks that are time sensitive.

If you like this idea then perhaps you would also have 12 hourly (2x/day) and 6 hourly (4x/day) options as well.

It may not be popular to suggest, but maybe these queue options will be the only choices for free accounts.

a2j | 684 posts | Sept. 5, 2012, 6:58 p.m. | permalink

a2j; I've just noticed that you actually post more than most of the developers.nice work!

rcs1000 | 311 posts | Sept. 5, 2012, 8:41 p.m. | permalink

That's because all I can do it post. They actually solve problems...☺

a2j | 684 posts | Sept. 6, 2012, 7:18 a.m. | permalink

@a2j: I hear where you're coming from about time-sensitive scripts, but I think if your script is so time critical that a few minutes either side is a problem then a virtualised environment is always going to present potential problems because you have no control over the resource loading.

@glenn: Instead of random jitter I was going to suggest limiting the number of concurrent scripts and then starting new ones with a "one out one in" policy. I guess the downside is this lets users effectively DoS other users tasks by creating one which just does time.sleep(86400). Identifying the bottlenecks is, of course, great, but I can imagine that could prove tricky?

Some other possible approaches also occurred to me (none of which are ideal, but might be worth consideration):

Treat scheduled tasks less like cron and more like a very low priority process where the only guarantee is that it will be invoked at least once per hour (or whatever), or that it will receive X amount of CPU every hour. Not sure how easy the environment makes it to enforce these things.
Monitor the running time of scheduled processes and invoke shorter-running ones first. This allows developers who treat the system with respect (i.e. write resource-light tasks) to get better scheduling. Really badly-behaved tasks could be penalised by "missing" time slots (though of course they'd need to then be prioritised higher to avoid starvation). This approach sounds quite cool to me, but the flip side is you've got almost as much complexity as writing a general process scheduler, and that's not easy to do right.
Use a combination of the above two approaches to allow scheduled tasks X amount of CPU time per day and allow authors to choose whether they want to use that as some regularly-timed short bursts, or just as a longer-running background task. Sounds like the sort of thing which would be really fun to implement on a home project, but could be a hideous time-sink for comparatively little gain on a commercial venture like PA!

@rcs1000: I'm not sure about a2j, but since I moved into a job doing embedded development again, PA is great outlet to keep my Python skills fresh. I hope I'm not too boring in the process...

deleted-user-39880 | 669 posts | Sept. 6, 2012, 10:15 a.m. | permalink

Just an update on the new task mentioned above.

Log for cookbook/args.py n o t h i n g (Scheduled daily at 14:21)

/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-06 14:21:07 -- Completed task, took 1.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-07 14:21:09 -- Completed task, took 1.00 seconds, return code was 0.

Only two days of data here, but it certainly does perform as expected at this time of day.

For reference here is the last few entries from the job that runs daily at 23:59:

/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-03 00:01:23 -- Completed task, took 106.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-04 00:00:18 -- Completed task, took 28.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-05 00:00:27 -- Completed task, took 41.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-05 23:59:55 -- Completed task, took 18.00 seconds, return code was 0.
/usr/local/bin/python3.2 /usr/local/bin/python3.2
['cookbook/args.py', 'n', 'o', 't', 'h', 'i', 'n', 'g']

2012-09-06 23:59:58 -- Completed task, took 18.00 seconds, return code was 0.

a2j | 684 posts | Sept. 7, 2012, 7:16 p.m. | permalink

If this was a desktop system I'd certainly say the sudden performance drops were a result of thrashing. The fact that this is a virtualised system makes it more complex, but I'm sure there's some equivalent condition where the overhead of switching between a lot of tasks overcomes the real work done.

deleted-user-39880 | 669 posts | Sept. 9, 2012, 6:15 a.m. | permalink

EDIT: thashing == thrashing

a2j | 684 posts | Sept. 10, 2012, 3:20 a.m. | permalink

@a2j: Well spotted, thanks. (Original post now fixed, for anybody confused by a2j's post).

deleted-user-39880 | 669 posts | Sept. 10, 2012, 8:53 a.m. | permalink

What about offering a 'forecast' when scheduling a job? If it's not important what time it runs, I'd like to know when I'd expect the best run times (plus, I'd probably help out on your monthly hosting bills by avoiding such huge spikes at 11:59pm).

deleted-user-45046 | 1 post | Sept. 14, 2012, 12:43 p.m. | permalink

@droogans: Welcome to PA. It's good to see another face. I hope you find our community to be truly fantastic!!

@all:

In case you need further proof that the human race is doomed because of stupidity and/or litigiousness, here are some actual label instructions on consumer goods.

On a Sears hairdryer: "Do not use while sleeping." (But that's the only time I have to work on my hair!)

On a bag of Fritos: "You could be a winner! No purchase necessary. Details inside." (The shoplifter special?)

On a bar of Dial soap: "Directions: Use like regular soap." (And that would be how???....)

On some Swanson frozen dinners: "Serving suggestion: Defrost." (But it's just a suggestion.)

On Tesco's Tiramisu dessert (printed on bottom): "Do not turn upside down." (Well...duh, a bit late, huh!)

On Marks & Spencer Bread Pudding: "Product will be hot after heating." (And you thought???...)

On packaging for a Rowenta iron: "Do not iron clothes on body." (But that would save so much time!)

On Nytol Sleep Aid: "Warning: May cause drowsiness." (And I'm taking this because???....)

On a Japanese food processor: "Not to be used for the other use." (Now, somebody out there, help me on this. I'm a bit curious.)

On Sunsbury's peanuts: "Warning: contains nuts." (Talk about a news flash!)

On an American Airlines packet of nuts: "Instructions: Open packet, eat nuts." (Step 3: maybe, uh...fly Delta?)

On a child's superman costume: "Wearing of this garment does not enable you to fly."

a2j | 684 posts | Sept. 14, 2012, 4:02 p.m. | permalink

Updated logs from the simple daily tasks previously mentioned:

23:59

2012-09-10 00:00:03 -- Completed task, took 23.00 seconds, return code was 0.
2012-09-11 00:00:08 -- Completed task, took 28.00 seconds, return code was 0.
2012-09-12 00:00:25 -- Completed task, took 39.00 seconds, return code was 0.
2012-09-13 00:01:14 -- Completed task, took 93.00 seconds, return code was 0.
2012-09-14 00:01:07 -- Completed task, took 71.00 seconds, return code was 0.

14:21

2012-09-10 14:21:11 -- Completed task, took 1.00 seconds, return code was 0.
2012-09-11 14:21:08 -- Completed task, took 0.00 seconds, return code was 0.
2012-09-12 14:21:09 -- Completed task, took 0.00 seconds, return code was 0.
2012-09-13 14:21:05 -- Completed task, took 0.00 seconds, return code was 0.
2012-09-14 14:21:09 -- Completed task, took 1.00 seconds, return code was 0.

a2j | 684 posts | Sept. 14, 2012, 4:08 p.m. | permalink

Wow, thanks a2j. Okay, I think I have a couple of ideas about what we need to do now... Looks like a job for Mr Random Default Time.

hansel | 459 posts | PythonAnywhere staff | Sept. 14, 2012, 5:16 p.m. | permalink

I'd recommend that you do a mostly random default time. I.E. remove certain results from the random calculation such as :00, :10, :15, :20, :30, :40, :45, :50. Of course there may be others, I just figured these are times likely to have higher concentrations chosen by users. I imagine you will query the system to determine where the trends are to avoid them altogether. None of my examples above may even be an issue.

In fact it may be fun to have the system choose a time that is most unused by currently scheduled jobs rather than a random choice.

a2j | 684 posts | Sept. 14, 2012, 6:03 p.m. | permalink

Random default time now implemented - great detective work a2j, thanks!

harry | 2710 posts | PythonAnywhere staff | Sept. 20, 2012, 1:40 p.m. | permalink

I'm glad to be of service...☺

a2j | 684 posts | Sept. 20, 2012, 2:26 p.m. | permalink