Re: Using instance variables in sub-classed tasks

Re: Using instance variables in sub-classed tasks

Django Admin task status and periodic tasks

Hi,

I am django celery. I have set of periodic tasks that i have added to
django admin. I have set
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

I have celerybeat running on one server and 6 celery workers.

Now I have two problems:

1> If I add a new periodic task to admin then celerybeat will not pick
it up immediately. It picks it up eventually later on or if i run the
task once manually (havent really been able to figure out)

2> The periodic task and sometimes other tasks show status as Started
in admin even though they have actually finished successfully. Why
wont the status update to Success.

Thanks,
Rohan

Re: Way to restart workers after certain tasks?

Mher,

Thanks.. I did see that option, but it doesn't solve the problem I
have very elegantly. The problem I have is that I've got a couple of
tasks that eat up loads of memory. Basically, any worker that runs one
of these tasks ought to be shut down afterward... These tasks run
rarely, so it shouldn't be a huge deal. I don't want to set my max-
tasks to 1 though, which is what I would need to do to accomplish
that. Make sense?

Phill

On Jul 29, 11:04 pm, Mher Movsisyan <mher.movsis...@...> wrote:
> On Jul 30, 2011, at 12:49 AM, Phill wrote:
>
> > Wacky question.. I've got a couple really big memory/cpu tasks that appear to
> > be  leaking (I'm rebuilding some haystack indexes nightly).. I'm not sure
> > why the memory isn't being recovered yet (since it seems to work nicely in my
> > local environment).... but until I get it figured out, I'm wanting to make
> > sure the process that ran the task doesn't stick around.
>
> > Of course I can hook up a cron job that runs the task in a management command
> > independently, but I figured I'd ask if there was a way to do this in through
> > celery/celerybeat instead... just in case there was a neat way to do this.
>
> CELERYD_MAX_TASKS_PER_CHILD option sets the number of tasks a worker process executes before it’s
replaced by a new process.
>
> http://ask.github.com/celery/userguide/workers.html#max-tasks-per-chi...
>
> Mher

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Re: Re: Way to restart workers after certain tasks?

Phill,

In that case you can create a special worker with CELERYD_MAX_TASKS_PER_CHILD and route your "special"
tasks to that worker. Remaining tasks will go to other workers with no task-per-child limitations.

Mher 

On Aug 2, 2011, at 8:22 AM, phill wrote:

> Mher,
> 
> Thanks.. I did see that option, but it doesn't solve the problem I
> have very elegantly. The problem I have is that I've got a couple of
> tasks that eat up loads of memory. Basically, any worker that runs one
> of these tasks ought to be shut down afterward... These tasks run
> rarely, so it shouldn't be a huge deal. I don't want to set my max-
> tasks to 1 though, which is what I would need to do to accomplish
> that. Make sense?
> 
> Phill
> 
> On Jul 29, 11:04 pm, Mher Movsisyan <mher.movsis...@...> wrote:
>> On Jul 30, 2011, at 12:49 AM, Phill wrote:
>> 
>>> Wacky question.. I've got a couple really big memory/cpu tasks that appear to
>>> be  leaking (I'm rebuilding some haystack indexes nightly).. I'm not sure
>>> why the memory isn't being recovered yet (since it seems to work nicely in my
>>> local environment).... but until I get it figured out, I'm wanting to make
>>> sure the process that ran the task doesn't stick around.
>> 
>>> Of course I can hook up a cron job that runs the task in a management command
>>> independently, but I figured I'd ask if there was a way to do this in through
>>> celery/celerybeat instead... just in case there was a neat way to do this.
>> 
>> CELERYD_MAX_TASKS_PER_CHILD option sets the number of tasks a worker process executes before it’s
replaced by a new process.
>> 
>> http://ask.github.com/celery/userguide/workers.html#max-tasks-per-chi...
>> 
>> Mher
> 
> -- 
> You received this message because you are subscribed to the Google Groups "celery-users" group.
> To post to this group, send email to celery-users@...
> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
> 

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Re: Re: Way to restart workers after certain tasks?

Hi Phill,

You can also use signals to raise an exception that kills the worker
just after executing the task. If an exception raises in a signal
handler it will not affect the status of the task.

from celery.signals import task_postrun
from celery.task import task

 <at> task
def do_something():
    return "whatever"

def shutdown_worker(**kwargs):
    raise SystemExit()

task_postrun.connect(shutdown_worker, sender=do_something)

bye

On Tue, Aug 2, 2011 at 12:39 PM, Mher Movsisyan
<mher.movsisyan@...> wrote:
> Phill,
>
> In that case you can create a special worker with CELERYD_MAX_TASKS_PER_CHILD and route your "special"
tasks to that worker. Remaining tasks will go to other workers with no task-per-child limitations.
>
> Mher
>
>
> On Aug 2, 2011, at 8:22 AM, phill wrote:
>
>> Mher,
>>
>> Thanks.. I did see that option, but it doesn't solve the problem I
>> have very elegantly. The problem I have is that I've got a couple of
>> tasks that eat up loads of memory. Basically, any worker that runs one
>> of these tasks ought to be shut down afterward... These tasks run
>> rarely, so it shouldn't be a huge deal. I don't want to set my max-
>> tasks to 1 though, which is what I would need to do to accomplish
>> that. Make sense?
>>
>> Phill
>>
>> On Jul 29, 11:04 pm, Mher Movsisyan <mher.movsis...@...> wrote:
>>> On Jul 30, 2011, at 12:49 AM, Phill wrote:
>>>
>>>> Wacky question.. I've got a couple really big memory/cpu tasks that appear to
>>>> be  leaking (I'm rebuilding some haystack indexes nightly).. I'm not sure
>>>> why the memory isn't being recovered yet (since it seems to work nicely in my
>>>> local environment).... but until I get it figured out, I'm wanting to make
>>>> sure the process that ran the task doesn't stick around.
>>>
>>>> Of course I can hook up a cron job that runs the task in a management command
>>>> independently, but I figured I'd ask if there was a way to do this in through
>>>> celery/celerybeat instead... just in case there was a neat way to do this.
>>>
>>> CELERYD_MAX_TASKS_PER_CHILD option sets the number of tasks a worker process executes before it’s
replaced by a new process.
>>>
>>> http://ask.github.com/celery/userguide/workers.html#max-tasks-per-chi...
>>>
>>> Mher
>>
>> --
>> You received this message because you are subscribed to the Google Groups "celery-users" group.
>> To post to this group, send email to celery-users@...
>> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups "celery-users" group.
> To post to this group, send email to celery-users@...
> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
>
>

-- 
Obstáculos es lo que ves cuando apartas la vista del objetivo.

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Re: Re: Way to restart workers after certain tasks?

Clever!

I'll try this. Mher's solution of routing to the tasks to a special worker is nice as well, but requires that I
have a mostly-idle worker hanging about doing nothing all day while it's waiting to run this task, so I'm a
bigger fan of this shutdown-after approach. Thanks,.

Phill

On Aug 2, 2011, at 5:32 AM, Miguel Hernández wrote:

> Hi Phill,
> 
> You can also use signals to raise an exception that kills the worker
> just after executing the task. If an exception raises in a signal
> handler it will not affect the status of the task.
> 
> from celery.signals import task_postrun
> from celery.task import task
> 
>  <at> task
> def do_something():
>    return "whatever"
> 
> def shutdown_worker(**kwargs):
>    raise SystemExit()
> 
> task_postrun.connect(shutdown_worker, sender=do_something)
> 
> bye
> 
> On Tue, Aug 2, 2011 at 12:39 PM, Mher Movsisyan
> <mher.movsisyan@...> wrote:
>> Phill,
>> 
>> In that case you can create a special worker with CELERYD_MAX_TASKS_PER_CHILD and route your "special"
tasks to that worker. Remaining tasks will go to other workers with no task-per-child limitations.
>> 
>> Mher
>> 
>> 
>> On Aug 2, 2011, at 8:22 AM, phill wrote:
>> 
>>> Mher,
>>> 
>>> Thanks.. I did see that option, but it doesn't solve the problem I
>>> have very elegantly. The problem I have is that I've got a couple of
>>> tasks that eat up loads of memory. Basically, any worker that runs one
>>> of these tasks ought to be shut down afterward... These tasks run
>>> rarely, so it shouldn't be a huge deal. I don't want to set my max-
>>> tasks to 1 though, which is what I would need to do to accomplish
>>> that. Make sense?
>>> 
>>> Phill
>>> 
>>> On Jul 29, 11:04 pm, Mher Movsisyan <mher.movsis...@...> wrote:
>>>> On Jul 30, 2011, at 12:49 AM, Phill wrote:
>>>> 
>>>>> Wacky question.. I've got a couple really big memory/cpu tasks that appear to
>>>>> be  leaking (I'm rebuilding some haystack indexes nightly).. I'm not sure
>>>>> why the memory isn't being recovered yet (since it seems to work nicely in my
>>>>> local environment).... but until I get it figured out, I'm wanting to make
>>>>> sure the process that ran the task doesn't stick around.
>>>> 
>>>>> Of course I can hook up a cron job that runs the task in a management command
>>>>> independently, but I figured I'd ask if there was a way to do this in through
>>>>> celery/celerybeat instead... just in case there was a neat way to do this.
>>>> 
>>>> CELERYD_MAX_TASKS_PER_CHILD option sets the number of tasks a worker process executes before
it’s replaced by a new process.
>>>> 
>>>> http://ask.github.com/celery/userguide/workers.html#max-tasks-per-chi...
>>>> 
>>>> Mher
>>> 
>>> --
>>> You received this message because you are subscribed to the Google Groups "celery-users" group.
>>> To post to this group, send email to celery-users@...
>>> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
>>> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
>>> 
>> 
>> --
>> You received this message because you are subscribed to the Google Groups "celery-users" group.
>> To post to this group, send email to celery-users@...
>> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
>> 
>> 
> 
> 
> 
> -- 
> Obstáculos es lo que ves cuando apartas la vista del objetivo.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "celery-users" group.
> To post to this group, send email to celery-users@...
> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
> 

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Re: Re: Way to restart workers after certain tasks?

So… I coded this up today and it seems to work nicely. I wanted to make sure this is acceptable though..
there isn't cleanup that ought to get done by Celery for these workers in cases like this? When I see these
lines on a graceful Celery shutdown it gives me pause about throwing SystemExit from the signal…

[2011-08-02 18:08:52,771: DEBUG/MainProcess] joining pool workers
[2011-08-02 18:08:52,771: DEBUG/MainProcess] pool workers joined
[2011-08-02 18:08:52,772: DEBUG/MainProcess] running the remaining "atexit" finalizers

Phill

On Aug 2, 2011, at 5:32 AM, Miguel Hernández wrote:

> Hi Phill,
> 
> You can also use signals to raise an exception that kills the worker
> just after executing the task. If an exception raises in a signal
> handler it will not affect the status of the task.
> 
> from celery.signals import task_postrun
> from celery.task import task
> 
>  <at> task
> def do_something():
>    return "whatever"
> 
> def shutdown_worker(**kwargs):
>    raise SystemExit()
> 
> task_postrun.connect(shutdown_worker, sender=do_something)
> 
> bye
> 
> On Tue, Aug 2, 2011 at 12:39 PM, Mher Movsisyan
> <mher.movsisyan@...> wrote:
>> Phill,
>> 
>> In that case you can create a special worker with CELERYD_MAX_TASKS_PER_CHILD and route your "special"
tasks to that worker. Remaining tasks will go to other workers with no task-per-child limitations.
>> 
>> Mher
>> 
>> 
>> On Aug 2, 2011, at 8:22 AM, phill wrote:
>> 
>>> Mher,
>>> 
>>> Thanks.. I did see that option, but it doesn't solve the problem I
>>> have very elegantly. The problem I have is that I've got a couple of
>>> tasks that eat up loads of memory. Basically, any worker that runs one
>>> of these tasks ought to be shut down afterward... These tasks run
>>> rarely, so it shouldn't be a huge deal. I don't want to set my max-
>>> tasks to 1 though, which is what I would need to do to accomplish
>>> that. Make sense?
>>> 
>>> Phill
>>> 
>>> On Jul 29, 11:04 pm, Mher Movsisyan <mher.movsis...@...> wrote:
>>>> On Jul 30, 2011, at 12:49 AM, Phill wrote:
>>>> 
>>>>> Wacky question.. I've got a couple really big memory/cpu tasks that appear to
>>>>> be  leaking (I'm rebuilding some haystack indexes nightly).. I'm not sure
>>>>> why the memory isn't being recovered yet (since it seems to work nicely in my
>>>>> local environment).... but until I get it figured out, I'm wanting to make
>>>>> sure the process that ran the task doesn't stick around.
>>>> 
>>>>> Of course I can hook up a cron job that runs the task in a management command
>>>>> independently, but I figured I'd ask if there was a way to do this in through
>>>>> celery/celerybeat instead... just in case there was a neat way to do this.
>>>> 
>>>> CELERYD_MAX_TASKS_PER_CHILD option sets the number of tasks a worker process executes before
it’s replaced by a new process.
>>>> 
>>>> http://ask.github.com/celery/userguide/workers.html#max-tasks-per-chi...
>>>> 
>>>> Mher
>>> 
>>> --
>>> You received this message because you are subscribed to the Google Groups "celery-users" group.
>>> To post to this group, send email to celery-users@...
>>> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
>>> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
>>> 
>> 
>> --
>> You received this message because you are subscribed to the Google Groups "celery-users" group.
>> To post to this group, send email to celery-users@...
>> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
>> 
>> 
> 
> 
> 
> -- 
> Obstáculos es lo que ves cuando apartas la vista del objetivo.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "celery-users" group.
> To post to this group, send email to celery-users@...
> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
> 

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Can I use two RabbitMQ server on one django-celery?

Hi,

I have two django apps that uses celery. But, they are independent, so
the RabbitMQ servers are not the same one (They are required different
BROKER_HOST). How can I configurate that to make it work?

Thanks.

worker sever calculating results, but not returning answers - celery tutorial

Hello,

I am new to celery and the world of computer messaging, task
management, etc.  I am having trouble going through the tutorials in
the celery and django celery documentation.

Here is the short explanation of my problem. The worker server is not
returning my results. Going through the “add” tutorial, I see that the
server computes add.delay(4,4) as 8 in 0.059001232314 seconds, but the
solution is never returned to my python prompt.

What would prevent this communication back to the consumer?

Help and suggestions greatly appreciated. Thanks.

Here are the details:

I am developing on a windows machine.

I installed rabbitmq and it appears to be functioning properly. I went
through the rabbitmq tutorials without any problems.

First I added the following to my django project’s settings file:

At the top:
import djcelery
djcelery.setup_loader()

BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"

to my installed apps:
'myapp.tasks',
'djcelery',

Then I created a tasks.py file in the myapp folder in my django
project containing the following:

from celery.decorators import task

 <at> task()
def add(x, y):
    print("Executing task id %r, args: %r kwargs: %r" %
(add.request.id, add.request.args, add.request.kwargs))
    logger = add.get_logger()
    logger.info("Adding %s + %s" % (x, y))
    return x + y

When I start rabbitmq and run the celery worker  server through django
in a terminal window using this command:

python manage.py celeryd -l info

I get this message from the server:

No handlers could be found for logger “multiprocessing”

[Configuration]
- broker:      amqplib://guest <at> localhost:5672/
- loader:      djcelery.loaders.DjangoLoader
- logfile:     [stderr] <at> INFO
- concurrency: 4
- events:      OFF
- beat:        OFF

[Queues]
- celery:      exchange: celery (direct)  binding: celery

[Tasks]
 - myapp.module.tasks.add
[2011-07-27 21:17:19, 990: WARNING/MainProcess] celery <at> sequoia has
started.

 C:\Python27\lib\site-packages\django-celery-2.2.4-py2.7.egg\djcelery
\loaders.py:80:  UserWarning: Using settings.DEBUG leads to a memory
leak, neveruse this setting in production environments!
     warnings.warn(“Using settings.DEBUG leads to a memory leak,
never”

So it seems things are working.

Next I attempted the tutorial.

>>> from myproject.myapp.tasks import add
>>> result = add.delay(4,4)

When I enter that command the worker server terminal displays this

[2011-07-28 11:39:21, 706: INFO/MainProcess] Got task from broker:
myproject.myapp.tasks.add[7e794740-63c4-42fb-acd5-b9c6fcd545c3]
[2011-07-28 11:39:21, 706: INFO/MainProcess] Task
myproject.myapp.tasks.add [7e794740-63c4-42fb-acd5-b9c6fcd545c3]
succeed in 0.04600000038147s: 8

The task was received and the correct answer was computed.

But when I continue with the tutorial I don’t get answers back.

>>> result
(AsyncResult: 7auathu945gry48-a bunch of stuff)
>>> result.ready()
False
>>>                 (nothing returned)
>>> result.get()
                    (it just hangs there)

After that last line the terminal has to be closed. What is going
wrong?

Thanks for your help.

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.