Avil Page (Posts about celery)

Load Testing Celery With Different Brokers

Anand Reddy Pandikunta — Fri, 09 Feb 2018 13:49:18 GMT

Celery is mainly used to offload work from request/response cycle in web applications and to build pipelines in data processing applications. Lets run a load test on celery to see how well it queues the tasks with various brokers.

Let us take a simple add task and measure queueing time.

import timeit

from celery import Celery

broker = 'memory://'


app = Celery(broker=broker)


@app.task
def add(x, y):
    return x + y


tasks = 1000
start_time = timeit.default_timer()
results = [add.delay(1, 2) for i in range(tasks)]
duration = timeit.default_timer() - start_time
rate = tasks//duration
print("{} tasks/sec".format(str(rate))

On development machine, with AMD A4-5000 CPU, queueing time is as follows

memory ---> 400 tasks/sec
rabbitmq ---> 300 tasks/sec
redis ---> 250 tasks/sec
postgres ---> 30 tasks/sec

On production machine, with Intel(R) Xeon(R) CPU E5-2676, queueing time is as follows

memory ---> 2000 tasks/sec
rabbitmq ---> 1400 tasks/sec
redis ---> 1200 tasks/sec
postgres ---> 200 tasks/sec

For low/medium traffic webistes and applications, 1000 tasks/second should be fine. For high traffic webistes, there will be multiple servers queueing up the tasks.

Incase if we need to queue the tasks at a higher rate and if we have task arguments before hand, we can chunk the tasks.

tasks = add.chunks(zip(range(1000), range(1000)), 10)

This will divide 1000 tasks into 10 groups of 100 tasks each. As there is no messaging overhead, it can queue any number of tasks in less than a second.

How To Auto Reload Celery Workers In Development?

Anand Reddy Pandikunta — Sun, 07 May 2017 04:22:47 GMT

We can pass --autoreload option when starting celery worker. This will restart worker when codebase changes.

celery worker -l info -A foo --autoreload

Unfortunately, it doesn't work as expected and it is deprecated.

During development, we will keep on changing the code base. Manually restarting celery worker everytime is a tedious process. It would be handy if workers can be auto reloaded whenever there is a change in the codebase.

Watchdog provides Python API and shell utilities to monitor file system events. We can install it with

pip install watchdog

Watchdog provides watchmedo a shell utilitiy to perform actions based on file events. It has auto-restart subcommand to start a long-running subprocess and restart it. So, celery workers can be auto restarted using this.

watchmedo auto-restart -- celery worker -l info -A foo

By default it will watch for all files in current directory. These can be changed by passing corresponding parameters.

watchmedo auto-restart -d . -p '*.py' -- celery worker -l info -A foo

If you are using django and don't want to depend on watchdog, there is a simple trick to achieve this. Django has autoreload utility which is used by runserver to restart WSGI server when code changes.

The same functionality can be used to reload celery workers. Create a seperate management command called celery. Write a function to kill existing worker and start new worker. Now hook this function to autoreload as follows.

import shlex
import subprocess

from django.core.management.base import BaseCommand
from django.utils import autoreload


def restart_celery():
    cmd = 'pkill -9 celery'
    subprocess.call(shlex.split(cmd))
    cmd = 'celery worker -l info -A foo'
    subprocess.call(shlex.split(cmd))


class Command(BaseCommand):

    def handle(self, *args, **options):
        print('Starting celery worker with autoreload...')
        autoreload.main(restart_celery)

Now you can run celery worker with python manage.py celery which will start a celery worker and autoreload it when codebase changes.

Scaling Celery - Sending Tasks To Remote Machines

Anand Reddy Pandikunta — Sun, 09 Nov 2014 08:35:00 GMT

Celery:

Celery is a Python package which implements a task queue mechanism with a foucs on real-time processing, while also supporting task scheduling.

It has 3 main components.

Celery Application(or Client): It is responsible for adding tasks to the queue.

Celery Worker(or Server): It is responsible for executing the tasks given to it.

Broker: It is responsible for transporting messages between client and server.

What You Should Know:

You should know basics of Celery and you should be familiar with

creating celery tasks


from celery import Celery

app = Celery('tasks', backend='amqp',
                      broker='amqp://<user>:<password>@<ip>/<vhost>')

@app.task
def add(x, y):
    return x + y

adding tasks to a queue,


add.apply_async(args=[1,2])

and consuming tasks with a worker


celery worker -A my_app -l info

If your tasks doesn't need much system resources, you can setup all of them in the same machine. But, if you have a lot of jobs which consume resources, then you need to spread them out in several machines.

In this tutorial lets move our celery workers into a remote machine keeping client and broker in same machine.

Sending Tasks To Another Machine:

On Machine A:

Install Celery & RabbitMQ.
Configure RabbitMQ so that Machine B can connect to it.


# add new user
sudo rabbitmqctl add_user <user> <password>

# add new virtual host
sudo rabbitmqctl add_vhost <vhost_name>

# set permissions for user on vhost
sudo rabbitmqctl set_permissions -p <vhost_name> <user> ".*" ".*" ".*"

# restart rabbit
sudo rabbitmqctl restart

Create a new file remote.py with a simple task. Here we have broker installed in machine A. So give ip address of machine 1 in broker url option.


from celery import Celery

app = Celery('tasks', backend='amqp',
                      broker='amqp://<user>:<password>@<ip>/<vhost>')

def add(x, y):
     return x + y

Now we have everything setup on machine A. We can now put some tasks in queue.


In [1]: from remote import add

In [2]: add.delay(1, 2)
Out[2]: <AsyncResult: 3eb96a11-aa61-46d3-9b9d-e0e1703438d0>

In [3]: b.delay(2, 3)
Out[3]: <AsyncResult: ec40db1a-a43c-4486-9530-0a3153fe1380>

In [4]: b.delay(3, 4)
Out[4]: <AsyncResult: ca53a4c7-061b-408e-82ee-86c2d43d21a0>

Everything is setup on machine A. Now lets get into machine B.

On Machine B:

Install Celery.
Copy remote.py file from machine A to this machine.
Run a worker to consume the tasks


celery worker -l info -A remote

As soon as you launch the worker, you will receive the tasks you queued up and gets executed immediately.


[2014-11-09 00:01:19,168: INFO/MainProcess] Received task: remote.add[c2d2bb27-ff5f-47da-b2b9-6fb11669ee1a]
[2014-11-09 00:01:19,170: INFO/MainProcess] Received task: remote.add[8daa1a5c-17d0-46dc-9c93-faf7fbeccdd9]
[2014-11-09 00:01:19,172: INFO/MainProcess] Received task: remote.add[79603d15-24f1-43f8-b8b7-525b7cd4b9a2]
[2014-11-09 00:01:19,401: INFO/MainProcess] Task remote.add[8daa1a5c-17d0-46dc-9c93-faf7fbeccdd9] succeeded in 0.226168102003s: 3
[2014-11-09 00:01:19,462: INFO/MainProcess] Task remote.add[c2d2bb27-ff5f-47da-b2b9-6fb11669ee1a] succeeded in 0.286503815001s: 5
[2014-11-09 00:01:19,464: INFO/MainProcess] Task remote.add[79603d15-24f1-43f8-b8b7-525b7cd4b9a2] succeeded in 0.288741396998s: 7

This is just a simple guide on how to send tasks to remote machines.

Depending on your need, you might have to set up a cluster of servers and route tasks accordingly to scale.