Forums

I am using celery in my application (developed on my local machine) to schedule tasks to run at a given time, as well as to perform longer-running tasks (like emailing) in the background so as not to degrade user-facing performance. Can PythonAnywhere support running celery? I would probably use CloudAMQP to host the RabbitMQ task queue. If PythonAnywhere cannot support celery, is there any way that I may be able to somehow implement something similar using PythonAnywhere?

Excellent question :-) We don't have any experience with Celery in-house, but here are some thoughts:

• You can install any Python package on PythonAnywhere using pip with the --user flag (or without the flag, if you're working in a virtualenv). So you should definitely be able to install the celery package with pip install --user Celery.
• We can't run a RabbitMQ server, but it sounds like you have that sorted through using an external provider. You have a paid account so you should be able to connect to that without problems.

So that handles the code to put stuff on the Celery queue -- I've not tried it, but I believe everything's in place so that you should be able to do it. As regards writing worker servers to process the tasks, it's a little trickier but should be doable:

• PythonAnywhere's support for long-running server process (apart from WSGI web apps) is not ideal right now (that's one of our priorities for the new year), but you can get it to work.
• Processes that you run in PythonAnywhere consoles don't have a guarantee -- that is, they can stop at pretty much any time if the system is upgraded, or if the console server glitches. (Probably once a week or so for any given server.)
• The trick is to use the PythonAnywhere "Schedule" tab. You can write a scheduled task that runs (say) once an hour, and checks if it's already running. If it is, then it exits -- otherwise it starts up, connects to the queue, and starts processing tasks from it.
• Here's the recommended code for doing that "am I running" check.

Hope that helps! Let us know if you have any further questions.

Hi VirtuFitness, Were you able to get Celery working on PythonAnywhere? I'm researching the same thing. Any snags or workarounds needed?

Thanks!

Helpful PA docs link for people who end up at this dead-end via a web search: AsyncInWebApps

Thanks! We should have thought of linking to that page from here.

I've managed to get Django Celery and RabbitMQ playing nicely together by setting up a free account on https://cloudamqp.com. My script takes about 60 minutes to run, and the web app remains functional throughout. The documentation on CloudAMQP is pretty good and I didn't have much trouble getting it to work, even though I'm new to both Celery and RabbitMQ.

Once you set up the free (Little Lemur) account and update your app with the relevant settings in Django, you'll need to run the following bash command but update myApp for your app. It's also probably a good idea to set up an 'always-on task' to keep the worker active.

celery -A myApp worker --without-heartbeat --without-gossip --without-mingle

Glad to hear that you made it work!