Forums

Cloning my PythonAnywhere instance

I think because I'm using apscheduler in a non-PythonAnywhere friendly way, I can't run more than one or two instances of my script without all instances of the script crashing (error below in case anyone has advice on how to solve this, I've posted about it before but got a "read the docs" type answer).

My question: Is it possible to clone my whole PythonAnywhere server somehow? I.e. like in DigitalOcean where you can simply clone an existing Droplet...?

Error submitting job "Test.main_run (trigger: interval[0:01:00], next run at: 2018-04-02 22:26:14 UTC)" to executor "default"
Traceback (most recent call last):
  File "/home/tjw0000/.local/lib/python3.6/site-packages/apscheduler/schedulers/base.py", line 958, in _process_jobs
    executor.submit_job(job, run_times)
  File "/home/tjw0000/.local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 71, in submit_job
    self._do_submit_job(job, run_times)
  File "/home/tjw0000/.local/lib/python3.6/site-packages/apscheduler/executors/pool.py", line 22, in _do_submit_job
    f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 115, in submit
    self._adjust_thread_count()
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 134, in _adjust_thread_count
    t.start()
  File "/usr/lib/python3.6/threading.py", line 846, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

[edit by admin: formatting]

Do you know how many new threads you are creating? You are doing this from a scheduled task right? Not a webapp? For webapps you cannot use threads. For scheduled tasks, you are only allowed to spawn a limited number of threads (128) so you may be hitting up against the limit.

Do you know how many new threads you are creating? You are doing this from a scheduled task right? Not a webapp? For webapps you cannot use threads. For scheduled tasks, you are only allowed to spawn a limited number of threads (128) so you may be hitting up against the limit.

Thanks for your reply, you guys are often so helpful around here!

Yes, it's a scheduled task. I don't know how many threads it is creating, or how to limit the number of threads it uses. Is there a way to visualize this with a print command or something?

OK, so in a scheduled task you should be OK so long as you don't exceed the 128-thread limit. (Actually, to be precise, it's a 128 process limit -- but from the Linux kernel's perspective, a thread is a special kind of process.)

Is the script you're running something you wrote yourself? Or is it something you've cloned/downloaded from somewhere?

OK, so in a scheduled task you should be OK so long as you don't exceed the 128-thread limit. (Actually, to be precise, it's a 128 process limit -- but from the Linux kernel's perspective, a thread is a special kind of process.)

Understood.

Is the script you're running something you wrote yourself? Or is it something you've cloned/downloaded from somewhere?

Yes, I wrote it myself, and I'm quite sure it's not the most 'performant' code

Yes, I wrote it myself, and I'm quite sure it's not the most 'performant' code

OK, that should make things easier -- it's always simpler to help someone with their own code than with a third party's :-)

So it looks like you're using one of the classes from the apscheduler package. Which one is it? And how have you configured it?

thanks for your response. So I have a couple instances... Here's a simplified version of the two uses

```

from apscheduler.schedulers.background import BackgroundScheduler
def __init__(self):
        scheduler = BackgroundScheduler()
        scheduler.add_job(self.get_margin_available , 'interval', minutes=1)
        scheduler.start()

```

In another file ...

```

from apscheduler.schedulers.background import BackgroundScheduler

scheduler = BackgroundScheduler()
        if self.ETF[0] == 'M':
            scheduler.add_job(self.update_dfs, trigger='cron',
                              minute='*/{}'.format(self.ETF[1]))
        elif self.ETF[0] == 'H':
            scheduler.add_job(self.update_dfs, trigger='cron',
                              hour='*/{}'.format(self.ETF[1]))
        scheduler.start()

```

Hmm, according to the apscheduler docs, the default thread pool size for a BackgroundScheduler is ten threads. Is it possible that you're creating multiple instances [edit: that is, more than ten or so] of the class? Perhaps you could just use one instance for your script as a whole?