Zhe Brah | 11 Nov 20:49 2014
Picon

Celery + RabbitMQ - stuck at mingle: searching for neighbors

I posted today on StackOverflow issue I've been fighting with for last two days. Have you ever had something like this?

http://stackoverflow.com/questions/26864925/celery-rabbitmq-stuck-at-mingle-searching-for-neighbors

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Manish Gill | 11 Nov 14:17 2014
Picon

apply_async gives TypeError

I've been using this particular bit of code for a while now, and until today, there had been no issues. Initially, I thought I might have broken something while modifying my code, but I made sure to revert back and the issue is still persisting.

During the executing of a scheduling function, my  "apply_async" breaks with the following traceback:

Traceback (most recent call last):
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/manish/.virtualenvs/conflow/daakiya/daakiya/views.py", line 72, in notifications
    res = rule.schedule_or_update_cycle()
  File "/Users/manish/.virtualenvs/conflow/daakiya/daakiya/rules.py", line 106, in schedule_or_update_cycle
    rv = self.event.schedule()   # Returns Celery AsyncResult
  File "/Users/manish/.virtualenvs/conflow/daakiya/daakiya/event.py", line 218, in schedule
    rv = msg.ruled_dispatch.apply_async(eta=eta)
  File "/Users/manish/.virtualenvs/conflow/lib/python2.7/site-packages/celery/app/task.py", line 462, in apply_async
    args = (self.__self__, ) + tuple(args)
TypeError: 'NoneType' object is not iterable

Here's a paste of the same: https://dpaste.de/kyWW


So, Celery is ready to accept tasks, but apply_async does not schedule it for some reason. I'd appreciate any help. :)

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Frank | 6 Nov 23:47 2014
Picon

Custom Exception Stack Traces In Logs

I'm using Celery with Loggly (a log aggregation service).

When an exception occurs our celery.log file contains long stack traces which overwhelm our Loggly usage quota.

How can I customize the Celery logger to include only the exception message but not the entire stack trace?

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
phill | 4 Nov 22:15 2014
Picon

Chord finalizer not running intermitently

celery==3.1.16
kombu==3.0.23
amqp==1.4.6
Redis broker (2.8.X)

Recently converted a couple of task groups to a chord, in which I use a finalizer to publish some stats to redis for monitoring purposes. The tasks in the chord header seem to run without issue all of the time, but about half the time my finalizer doesn't. No log output or mention of it anywhere.

I'm calling the chord like this:

# args = [[p1, p2, p3], [p1, p2, p3], [p1, p2, p3]]....
chord_header = [stats_rollup_task.s(*a) for a in args]
chord(chord_header)(stats_rollup_completed_task.si())


I'm wondering what I might be doing wrong. Right now the chord header is always about 340 tasks deep, run once per day. The one interesting thing I've noticed is that the finalizer hasn't gotten lost on the weekend (when the tasks run significantly quicker due to lighter amounts of data to summarize). Generally, they aren't fast tasks. Most of them finish in 30 seconds, but some take as long as 18 minutes.

I've tried varying a number of settings and trying to find a common thread through the failure cases but not coming up with much. Curious if anyone has tips on how to chase down what's happening here.

Thanks in advance,
Phill

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Andy Gayton | 4 Nov 01:21 2014

Best way for a forked worker to stop itself

Hi there,

Our celery forked workers can grow to consume a large amount of memory, depending on which jobs they happen to field. We'd like for them to stop themselves once they exceed a certain size. I'm investigating using a custom loader:

class Loader(BaseLoader):
    def on_process_cleanup(self):
        rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        if rss > (10*1024*1024):
            os.kill(os.getpid(), signal.SIGTERM)

However this logs an exception: raised unexpected: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',) 

So, I'm guessing this isn't a great approach :) Does anyone have any advice on the best way to approach this problem?

thanks,
Andy.


--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
David Miranda | 3 Nov 16:08 2014
Picon

Django Celery + RabbitMQ + Nginx

Hi,

My application consists of django instances that send messages to the message queue (RabbitMQ) and are consumed by workers (Celery). Right now I have configured a cluster with 3 nodes with mirror queues. 

Currently my django application and workers connects with the AMQP protocol, however I wanted to implement a prevention mechanism for node failures by making load balancing (ie nginx). 

However I can not understand how my producers and consumers can connect to the load balancer (nginx) since it is not possible to connect to nginx with the AMQP protocol. 

Has anyone done this setup that can show? 

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Sztrovacsek | 3 Nov 15:09 2014
Picon

Stack trace in celery worker

Hello everyone,

I just subscribed to this mailing list in the hope of finding a solution for a celery problem of mine... We just stared using celery at work, and so far I like it a lot. There is one thing which annoys me however, and I spent two days on reading documentation and googling, but I could not find a solution.

My problem is the following. If the python code inside a celery task throws an exception, I would like to see the stack trace printed in the log (of the worker). I don't know what settings control this behaviour. 
I am using celery==3.1.13, with Django==1.7 and with python3.

Interestingly, in a freshly made toy project it all works as I want it to (I see the stacktrace on stdout, and it is even printed in red), but in the real project I can't get it to do the same.

This is what I want to see in the log:
[2014-11-03 08:32:17,006: INFO/MainProcess] Received task: mainapp.tasks.async_task_phase1[0a531b6d-55ff-4988-b406-0a116f605e0f]
[2014-11-03 08:32:17,025: ERROR/MainProcess] Task mainapp.tasks.async_task_phase1[0a531b6d-55ff-4988-b406-0a116f605e0f] raised unexpected: NameError("name 'y' is not defined",)
Traceback (most recent call last):
  File "/home/hanna/homeworkspace/rabbitmq/env/lib/python3.4/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/hanna/homeworkspace/rabbitmq/env/lib/python3.4/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/hanna/homeworkspace/rabbitmq/mysite/mainapp/tasks.py", line 14, in async_task_phase1
    print(y)
NameError: name 'y' is not defined


But I only see this:
[2014-11-03 08:32:17,006: INFO/MainProcess] Received task: mainapp.tasks.async_task_phase1[0a531b6d-55ff-4988-b406-0a116f605e0f]

This is very annoying because I don't see any indication of problems if I make a programming error in the celery task. I had to enable CELERY_ALWAYS_EAGER = True for developing purposes, to get feedback of my programming errors. 
Can you help me?


Regards,

Sztrovacsek

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Leonid Blouvshtein | 1 Nov 00:15 2014
Picon

Perforamnce Issue: ResultSet perform several queries instead of a single one

This is an important performance issue I think.

I have a result redis backend.
I saw that my bottleneck is the queries to the backend from the client (status queries).

After a short research I saw that the 'ResultSet' class is actually initiates a query for every single AsyncResult.

This is although the redis backend implements the 'mget' method (multi get) that could be used for retrieving all the queries at once.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Mike | 30 Oct 21:33 2014
Picon

Distributed Locks & Celery (using memcached)

I just wanted to relay my experiences with celery and the need for distributed locks.  I know most of the time people post about problems they are having in general on this forum (me included), so I thought I'd change it up a little. I'm not sure if anyone has talked much about this, but I didn't see much on search.

I'm sure everyone can imagine some situations where distributed locks might be useful.  If a tasks inserts records into a database that does not have RI locking, for example (as in my case).  Or the desire to keep a task running on a poll timer but NEVER wanting to run two at the same time in case a previous execution takes longer than the poll time. Keeping tasks idempotent isn't too hard in most cases if you can guarantee that two of the same tasks are not running at the exact same time for the same parameters.

The examples of how to do this for celery (at least that I have read in the past) have involved it being a Django project.  Well, I don't personally know much about Django, but it seemed more than I needed.  Since I was already using memcached as my results backend for celery, I thought I might as well use it for handling the distributed locking as well.  

There is a very nice and simple package for doing this called DistributedLock on pypi: https://pypi.python.org/pypi/DistributedLock/1.2   I just wanted to report that this was extremely simple to use, and uses the with statement and you can make your task block or raise exception (I recommend the latter for sure).  

For example if you only want to allow a single instance of the task to ever run at one time, you set the key to something static:

<at> app.task()def mytask(): try: with distributedlock('ScheduleChecker'): do_stuff() exception LockNotAcquiredError as e: print 'Lock not acquired, skipping task execution.'
Or if you want to only allow one task to run per parameter, send in some kind of key identifier that uniquely identifies the call.  Like if the task does something with a file:
<at> app.task()def mytask(filename): try: with distributedlock('mytask:{key]'.format(key=filename)): do_stuff(filename) exception LockNotAcquiredError as e: print 'Lock not acquired, skipping task execution for {file}.'.format(file=filename)

If the filename or path is > 255 characters you might need to shorten or use hash digest.
Does anyone see any issues using this method?  It seemed pretty simple to use for me, and seems to work great.  The only thing I've noticed that you have to be careful of is the lock expiration in memcached.  Too short, and you lose your locks and tasks start overlapping. Too long, and if a bad crash occurs on a worker that doesn't release the lock, then you have to wait longer before the process can pick itself back up. 
Anyhow, just thought I would share. 

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Kristaps Rāts | 30 Oct 11:21 2014
Picon

Internal error when embedding worker

Hi,
I'm trying to embed a celery worker in a Flask app using the method described in https://groups.google.com/forum/#!topic/celery-users/Ezv1kr6bkwc and am having some trouble, perhaps someone can spot what's wrong. 

I'm wrapping the celery app inside a utility class, which constructs the app and the worker:
    self.celery = celery.Celery()
   
self.worker = self.celery.WorkController(pool_cls='solo', queues=[self.queue_name])
   
self.celery.task(self._receive_callback, name=self.callback_task_name)

The utility class has a start() method which starts the worker like this:
    t = threading.Thread(target=self.worker.start)
   
# Starting the worker in a daemonic thread so that it doesn't keep the process
   
# alive when the main thread exits
    t
.setDaemon(True)
    t
.start()

When the embedded app receives the task it crashes with the following traceback:
 CRITICAL:celery.worker.job:Task [my_task_name][cfe87fb7-373d-4082-a72c-0f44d265cc9f] INTERNAL ERROR: AttributeError("'NoneType' object has no attribute 'loader'",)

 
Traceback (most recent call last):
 
File "/virtualenvdir/lib/python2.7/site-packages/celery/app/trace.py", line 333, in trace_task
    task
.__trace__ = build_tracer(task.name, task, **opts)
 
File "/virtualenvdir/lib/python2.7/site-packages/celery/app/trace.py", line 160, in build_tracer
    loader
= loader or app.loader
 
AttributeError: 'NoneType' object has no attribute 'loader'

Any ideas?

Kristaps

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Mike | 30 Oct 07:15 2014
Picon

Substantial drift from celery <at> xxxxxxxxxxxxx may mean clocks are out of sync.


[2014-10-30 01:11:16,935: WARNING/MainProcess] Substantial drift from celery <at> xxxxxxxxxxxxx may mean clocks are out of sync.  Current drift is
53 seconds.  [orig: 2014-10-30 01:11:16.935202 recv: 2014-10-30 01:10:23.681895]


How can I possibly have a drift if all the workers are currently running on the same server?  Does this mean I may have some other underlying issue?  

I also get missed heartbeats all the time, just in case this is related.  

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

Gmane