Michael Waterfall | 23 May 13:12
Picon
Gravatar

Django Celery Routing Problem

I'm using Django and Celery and I'm trying to setup routing to multiple queues. When I specify a task's `routing_key` and `exchange` (either in the task decorator or using `apply_async()`), the task isn't added to the broker (which is Kombu connecting to my MySQL database).

If I specify the queue name in the task decorator (which will mean the routing key is ignored), the task works fine. It appears to be a problem with the routing/exchange setup.

Any idea what the problem could be?

Here's the setup:

Settings.py

    INSTALLED_APPS = (
        ...
        'kombu.transport.django',
        'djcelery',
    )

    BROKER_BACKEND = 'django'
    CELERY_IMPORTS = (
        'all_tasks.tasks',
    )
    CELERYD_CONCURRENCY = 1
    CELERY_DISABLE_RATE_LIMITS = True
    CELERY_DEFAULT_QUEUE = 'default'
    CELERY_DEFAULT_EXCHANGE = "tasks"
    CELERY_DEFAULT_EXCHANGE_TYPE = "topic"
    CELERY_DEFAULT_ROUTING_KEY = "task.default"
    CELERY_QUEUES = {
        'default': {
            'binding_key':'task.#',
        },
        'i_tasks': {
            'binding_key':'important_task.#',
        },
    }

Task.py

    from celery.task import task
    
    <at> task(name='all_tasks.my_important_task', 
          routing_key='important_task.update')
    def my_important_task():
        try:
            ...
        except Exception as exc:
            my_important_task.retry(exc=exc)

Initiate task:

    from all_tasks import tasks
    tasks.my_important_task.delay()


I've asked this on StackOverflow too, however I don't have any answers yet.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/NX0YBXd53NkJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Jacek Bzdak | 20 May 12:37
Picon
Gravatar

How to attach callback to TaskSet

I use celery and djcelery using rabbitmq and database result backend. 
I have a TaskSet that contains couple thousand tasks, and I would like to attach a callback that would gather some statistics for this task set and store them somewhere. Is this possible? I didn't any revelant example in the docs (every callback example included handcoding submission of callback task). 

If this is not possible do you have any insights that could be usefull in my use case. 

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/OQfxCji6PWAJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Sarang | 16 May 10:40
Picon

How to create centralized celerybeat ?

Hi,
I'm running celerybeat with -B option in celeryd. 
I'm having to instaces of celeryd that means 2 celerybeat.
How can create centralized celerybeat system?

dilwin | 16 May 00:18
Picon

Best practices for using django-celery in a production envrironment

We decided to use django-celery for adding asynchronous task
processing capability to our product. We have tested launching celeryd
using the canned daemon scripts provided on the celery site and we
have not faced any problems in our test environment. We are using the
djcelery scripts, which launch celeryd thru manage.py.
http://docs.celeryproject.org/en/latest/cookbook/daemonizing.html#example-django-configuration

We have concerns about using manage.py in a production environment
given the general caveats (security/performance) that come with django-
admin.py. I've tried searching forums but I'm not able to find
anything which recommends against using manage.py for launching
celeryd in production. My hunch is, manage.py is just a shell that
will cede control to celeryd and there should be no difference (in
terms of production behavior) between running celeryd with or without
manage.py.

I'm hoping to get a definitive recommendation from the gurus in the
forum if it is safe to do so. In general, if there are any proven best
practices for using django-celery in production, please point me to
the same.

Thanks

Mahdi Yusuf | 15 May 16:13
Picon
Gravatar

WorkerLostError

Hi, 


I have only seen on my OSX machine during local development. I have the same development stack on my Ubuntu machine at home with no issues; For some reason the workers are exiting prematurely. I first thought this was a memory issue, after freeing up some memory the issue is till persisting. 

That being said, here are all the details of the issue; Another person had a similar experience outlined here.

Anyone experience this figure out what is going on?

--
Mahdi Yusuf

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/wVWnWeeS3nYJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Picon

How to set up Celery for multiple Django projects on the same server?

Hello!

I am using virtual environments for different Django projects on the
same server. How could I set Celery to work with each installation of
Django in those virtual environments?

For example, /home/myproject/ is the virtual environment for myproject
user. /home/myproject/project/mydjangoproject/ is the Django project
containing manage.py. /home/myproject/lib/python2.6/site-packages/
contains Django, celery, django-kombu, and django-celery.

I am looking forward for your answer!

Aidas Bendoraitis

Stefano Masini | 15 May 00:16
Picon
Gravatar

Multiple queues and consuming order

Hi,


say that I have 3 workers (w1, w2, w3), and 3 types of tasks (with routing keys A, B and C).

Is there a way to implement the following policy for consumers?

- w1 consumes preferably tasks A, if they exist. If no pending task with key A exists, then consume B and C, in whatever order.
- w2 consumes preferably tasks B, if they exist. If no pending task with key B exists, then consume A and C, in whatever order.
- w3 consumes preferably tasks C, if they exist. If no pending task with key C exists, then consume A and B, in whatever order.

If I start a worker like this:

celeryd -Q qA,qB,qC

does it mean that the worker eats up on queues qA, qB and qC, *in that order*?
In that case the solution to my problem would be to start 3 workers in the following way:

celeryd -Q qA,qB,qC
celeryd -Q qB,qA,qC
celeryd -Q qC,qA,qB

I went through all the documentation, but couldn't find a definitive answer.

I'm still a Celery novice. Can anyone point me to the right direction?

Thanks!
Stefano

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/eRuK-Aj3rXYJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Chris Angove | 10 May 19:47
Picon

In consumer.py why is timeout hardcoded to 1 sec

In celery/worker/consumer.py there are places where the socket timeout
is set to 1sec.  While this is fine in many cases it does seem a
little low for checing the queue under load conditions.  Is there a
reason this is set to 1 sec?

Chris Angove | 10 May 19:16
Picon

Expected timeline/reelase schedule for 2.6

We are currently using 2.4.5 and looking to upgrade. Would like to
time it to use 2.6, is there an official schedule for releasing 2.6
into the wildnerness of PyPi?

thong | 9 May 08:37
Picon

celeryd dies after 'No handlers could be found for logger "multiprocessing" '

I am running the newest stable version of celery on python 2.7 on
windows.

While trying to run:

manage.py celeryd -E --loglevel=INFO

... I get the following errors printed 4 times...

No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"

... and then celeryd dies... returning me to the command line in my
terminal

Any ideas? I have searched every available stackoverflow and git issue
on this, and have tried all the known fixes, but to no avail.

Benjamin Gross | 9 May 21:39
Picon

Celery not accepting tasks

I've just installed RabbitMQ v2.8.2 and Celery 2.5.3.  When I try to
execute the example "add"  task, I see RabbitMQ accepting the AMQP
connection:

=INFO REPORT==== 9-May-2012::15:19:34 ===
accepting AMQP connection <0.937.0> (127.0.0.1:57561 ->
127.0.0.1:5672)

but I never see the task being accepted/recognized in the Celery.

Within a Python console,

add.delay(4,4)

returns immediately,

but calling result.get() never returns:

result = add.delay(4,4)
result.get()

celeryd seems to be starting up & connecting to RabbitMQ properly.
Here is a snippet from the celeryd console after start up:

[2012-05-09 15:19:09,511: DEBUG/MainProcess] Start from server,
version: 8.0, properties: {u'information': u'Licensed under the MPL.
See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright':
u'Copyright (C) 2007-2012 VMware, Inc.', u'capabilities': {},
u'platform': u'Erlang/OTP', u'version': u'2.8.2'}, mechanisms:
[u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2012-05-09 15:19:09,512: DEBUG/MainProcess] Open OK! known_hosts []
[2012-05-09 15:19:09,512: DEBUG/MainProcess] Consumer: Connection
established.
[2012-05-09 15:19:09,513: DEBUG/MainProcess] using channel_id: 1
[2012-05-09 15:19:09,514: DEBUG/MainProcess] Channel open
[2012-05-09 15:19:09,516: DEBUG/MainProcess] basic.qos: prefetch_count-
>96
[2012-05-09 15:19:09,517: DEBUG/MainProcess] using channel_id: 2
[2012-05-09 15:19:09,518: DEBUG/MainProcess] Channel open
[2012-05-09 15:19:09,523: DEBUG/MainProcess] Consumer: Starting
message consumer...
[2012-05-09 15:19:09,523: DEBUG/MainProcess] Consumer: Ready to accept
tasks!

Any suggestions?

Thanks,
Benjamin


Gmane