Paul Choi | 16 Sep 01:03 2014
Picon

Publishing task stats to external monitoring service

Hello,

I'd like some ideas on how to accomplish the following - and perhaps someone's already done this already?
1. Overall stats on the worker itself. How many tasks are pending, have succeeded, etc. Looks like Celery Flower already implements this, and it could be done as a periodic task by Celery itself? It may be as simple as publishing results of these methods: http://docs.celeryproject.org/en/latest/reference/celery.worker.state.html?highlight=state#module-celery.worker.state

2. Each task should publish its own stats after finishing. For example, I want the task to publish its name, worker hostname, how many seconds it took to execute the task, etc. Would task signals be the right way to implement this? (http://docs.celeryproject.org/en/latest/userguide/signals.html#task-signals) Again, Celery Flower does this already, but it inspects Celery workers from outside-in, and I'd rather not run another daemon if possible. I'd like the stats publishing to be done by each individual task.

And obviously I don't want to reinvent the wheel if this has been done already.
Thanks for your help!

-Paul Choi

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Eswar Vandanapu | 14 Sep 05:00 2014

Messages stay in reserved status and are not distributed to available tasks after 3.1 upgrade.

We have 8 workers in the master queue, of which one is currently used. It is waiting for completion of 3 other messages submitted into the queue, and we can see them with inspect reserved as below.

However they are not being handed off to any worker even though 7 of them are free. I understand a single machine may pre_fetch them, but that machine itself has 3 more workers doing nothing. It only started happening upgrade to 3.1 from 3.0

We have one task that creates and waits for the other tasks to complete in the same queue.  Other tasks seems to be fine, but this one particular task that creates a large number of child tasks like that is always behaving this behavior.

Please advise us if anything can be done?

(analytics-server_env)gnana <at> ip-10-0-15-117:/opt/gnana/analytics-server$ celery -A gnana inspect reserved
-> b <at> ip-10-0-15-117: OK
- empty -
-> w <at> ip-10-0-14-184: OK
- empty -
-> m <at> ip-10-0-14-184: OK
- empty -
-> w <at> ip-10-0-15-117: OK
- empty -
-> b <at> ip-10-0-14-184: OK
- empty -
-> m <at> ip-10-0-15-117: OK
* {u'args': u"[[u'xxx', u'xxx.com', u'administrative.domain']]", u'time_start': None, u'name': u'tasks.answers.combined_results_for_day', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'master', u'exchange': u'master'}, u'hostname': u'm <at> ip-10-0-15-117', u'acknowledged': False, u'kwargs': u"{u'target_key': u'main', u'daily_dose': [], u'target': u'bookings', u'trace': u'3c63a2a8-5ddc-4054-b9f8-7c8783b27032'}", u'id': u'd405d9e0-067a-47d6-8bea-9d8999042e61', u'worker_pid': None}
* {u'args': u"[[u'xxx', u'xxx.com', u'administrative.domain']]", u'time_start': None, u'name': u'tasks.answers.combined_results_for_day', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'master', u'exchange': u'master'}, u'hostname': u'm <at> ip-10-0-15-117', u'acknowledged': False, u'kwargs': u"{u'target_key': u'main', u'daily_dose': [], u'target': u'bookings', u'trace': u'3c63a2a8-5ddc-4054-b9f8-7c8783b27032'}", u'id': u'0362d63c-45f1-45f8-b8c3-6c428a22e0f1', u'worker_pid': None}
* {u'args': u"[[u'xxx', u'xxx.com', u'administrative.domain']]", u'time_start': None, u'name': u'tasks.answers.combined_results_for_day', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'master', u'exchange': u'master'}, u'hostname': u'm <at> ip-10-0-15-117', u'acknowledged': False, u'kwargs': u"{u'target_key': u'main', u'daily_dose': [], u'target': u'bookings', u'trace': u'3c63a2a8-5ddc-4054-b9f8-7c8783b27032'}", u'id': u'5bb8fe84-e54d-414d-a312-e466445accd2', u'worker_pid': None}
(analytics-server_env)gnana <at> ip-10-0-15-117:/opt/gnana/analytics-server$

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Simon Forsberg | 12 Sep 10:29 2014
Picon

Revoking tasks with termination

Hello,

I noticed that tasks that have been terminated is completed with success 
status?

Is it supposed to function like that?

Thanks,

Lars Bowlin | 11 Sep 23:17 2014
Picon

Celery as a windows service

I am trying to install celery as a windows service on Windows Server 2008 R2 without much success. 
I'm using the following to create the process: 

sc create "celeryBeat" binpath= "C:\Python27\Scripts\celery.exe beat --workdir=\"<my working dir>\"" start= auto

The service installs successfully, but up on starting displays:
 "Error 1053: The service did not respond to the start or control request in a timely fashion". 
The same celery command runs just fine from the command line, and I've extended that timeout to ~10 minutes with no further success. I have seen suggestions for running it from task scheduler, but I would prefer to use a windows service for deployment automation reasons.

Any help would be greatly appreciated,
-L

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Nino Walker | 11 Sep 21:05 2014

Celery 2.6 to 3.1 performance degradation

We recently upgraded part of our system to 3.1.x, using the Redis backend and multiprocess concurrency.

We've observed a marked decrease in throughput under 3.1. Without flooding the thread with logs (yet), we're observing via strace sluggishness of workers in prolonged blocking READ on the pipe to the parent process (waiting for a task?), which we didn't see in 2.X. There are plenty of tasks in the redis queue.

This blocking read could last up to several seconds, and a number of short tasks would execute, and then we'd block again. Load on the host was well inside of healthy operating parameters.

As a workaround, we are multiple spawning single-process workers who do not suffer the same lag (but is suboptimal).

The stack:
billiard==3.3.0.18

celery==3.1.13

kombu==3.0.21
Ubuntu 12.04.4

Thanks in advance for any advice.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Mauricio Porras | 8 Sep 16:04 2014
Picon

error processing tasks from a federated exchange

Hey everyone! I'm trying to consume some tasks from a federated exchange in rabbitMq ...initially I got a error related to disallowed content being picked up by the worker, I'm gonna paste the initial error msg in here.

CRITICAL/MainProcess] Can't decode message body: ContentDisallowed('Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)',) [type:u'application/x-python-serialize' encoding:u'binary' headers:{u'x-received-from': [{u'redelivered': False, u'uri': u'amqp://localhost/branch', u'cluster-name': u'rabbit <at> mporras-lap', u'exchange': u'celery'}]}]
.
. more trace stack
.   
self.content_encoding, accept=self.accept)
  File "/home/mporras/serverApp_py2env/local/lib/python2.7/site-packages/kombu/serialization.py", line 174, in loads
    raise self._for_untrusted_content(content_type, 'untrusted')
ContentDisallowed: Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)

Then I find out a workaround where specifying this settings on celeryconfig (on the publisher side) that fixed that initial problem.

CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['application/json']

this settings fixed the initial problem, but now a got a new one, the task is being processes anyways, but the worker's console is showing this error messages on every task they pick up

 ERROR/MainProcess] Control command error: TypeError('dispatch() takes at least 2 arguments (1 given)',)
Traceback (most recent call last):
  File "/home/mporras/serverApp_py2env/local/lib/python2.7/site-packages/celery/worker/pidbox.py", line 38, in on_message
    self.node.handle_message(body, message)
  File "/home/mporras/serverApp_py2env/local/lib/python2.7/site-packages/kombu/pidbox.py", line 133, in handle_message
    return self.dispatch(**kwdict(body))
TypeError: dispatch() takes at least 2 arguments (1 given)


Any ideas? thanks in advance !

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Tadej Janež | 8 Sep 14:56 2014
Picon

How to properly terminate a running Celery task?

Hello,

I'm using Celery to run tasks which execute bash scripts which glue
together several independent programs. Such programs can get stuck (for
what ever reason), so I must be able to manually terminate them.

I can accomplish this by calling app.control.revoke(<task-id>,
terminate=True, signal='SIGKILL').
However, as explained in the documentation [1], this is error-prone
since it will terminate the Celery worker executing the task and that
process might have already started processing another task at that
point.

Is there any alternative / proper way to terminate already running
tasks?

Thanks and best regards,
Tadej

[1]
http://celery.readthedocs.org/en/latest/userguide/workers.html#revoke-revoking-tasks

Ask Solem | 4 Sep 17:49 2014

Re: Chain follow parents


On Sep 4, 2014, at 9:46 AM, horstibot-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org wrote:

Hi, if i use result.parent of the returned AsyncResult instance (result = chain.apply_async) i get the correct parents, but wenn i try to access AsyncResult(result.id).parent the parents are missing.
Is this correct or is it a bug? It doesn't allow you to follow workflows where chains are nested.

Thanks Stefan


The result instance is a local object, so when you get an instance from task.delay, etc it will be
instantiated with all the fields, in your case you are specifying a single field only (the id).

You can pass the result objects around serialized using celery.result.result_from_tuple(result.as_tuple())



-- 
Ask Solem
twitter.com/asksol | +44 07454281208

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

lev | 4 Sep 14:56 2014
Picon

Run celery task only if parent still alive

Hi,

I got a celery task that invokes a lot of other child tasks and waits for their results. Those child tasks can run for a long time, and due to limited resources, might start running long after they were invoked (few hours). I'd like to make sure that a child task will start to run only if it's parent is still alive.

My initial idea was for the child task to check on start if its parent is still alive (using inspect), and quit immediately otherwise.

So my question is, is there a way to get the parent tasks id from the child (so that I could check if it's still running)?
This is something I would expect to find in the
 contex, but unfortunately it is not there.

Or any other suggestions how to perform this check?


Thanks,
Lev.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
xeon Mailinglist | 4 Sep 13:05 2014
Picon

deal with concurrency with celery.

 I really need help in dealing with concurrent tasks in celery. I have several processes that execute the same tasks, and use the same queue to perform concurrent tasks. I don't know how to deal with this. Anyone here, to talk about it?

I don't know where to look for help, or read some doc about it.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Daniel Lenski | 4 Sep 00:45 2014
Picon

program structure for data warehousing apps

I am writing a Python application with data warehousing aspects, and was inspired by Roger Barnes' talk and slides (https://www.youtube.com/watch?v=AhIoAMltzVw) to try and figure out if I can use Celery to organize my application.

One of the difficulties that I am encountering is that I'm not sure how to organize and schedule my tasks given that:
  1) Many of them rely on *slow* external databases which I am scraping in batch mode; it is slow and ineffective to run multiple simultaneous instances of these tasks.
  2) Many of them should run periodically, but others are highly contingent on the outcome of previous tasks.

To deal with (1), I'm using a locking strategy similar to what is discussed here (http://stackoverflow.com/questions/20894771/celery-beat-limit-to-single-task-instance-at-a-time): I use an advisory lock on my Postgresql DB as a context manager around tasks which 

   <at> app.task(...)
  <at> pg_locked('lock_name')
  def slow_task(self, *args):
    # do something

For (2), I thought the simplest solution would be to use the chain signature to order multiple tasks, however I've found that it's not obvious how to "break the chain" and stop processing if an earlier task doesn't give any useful results. For example, how can I skip the later tasks in this chain if the earlier ones find no data that will be useful as input to the later ones?

  ch = celery.chain( slow_external_DB_task_1.s() ,
                     slow_external_DB_task_2.s() ,
                     fast_tasks_to_populate_local_DB.s(),
                     ...
                   )

Are there any examples of publicly-available data warehousing code structured around Celery which I could follow to find some possible design patterns for my application?

Thanks,
Dan Lenski

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

Gmane