Ask Solem | 13 Jun 18:26 2014

Re: Tasks run multiple times


On Jun 13, 2014, at 5:02 PM, Mihnea Giurgea <skipy3-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

Indeed, the cause of the issue was the celery flower instance.

Thanks for all the help and sorry for the clumsy mistake. It did not occur to me that celery flower would ever restore unacked messages.



On second thought this should be considered a bug since it means it will be very hard
to get custom visibility timeouts right, since every new consumer using the same
redis instance must always use the same visibility timeout.

It could be solved by associating a visibility timeout with every message,
instead of in the consumer like it works now.



-- 
Ask Solem
twitter.com/asksol | +44 07454281208

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

Ask Solem | 13 Jun 18:23 2014

Re: Tasks run multiple times


On Jun 13, 2014, at 5:02 PM, Mihnea Giurgea <skipy3-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

Indeed, the cause of the issue was the celery flower instance.

Thanks for all the help and sorry for the clumsy mistake. It did not occur to me that celery flower would ever restore unacked messages.

Hah, even I did not consider the fact that flower would help recover the unacknowledged tasks,
but that makes perfect sense :)

Glad it’s been resolved

-- 
Ask Solem
twitter.com/asksol | +44 07454281208

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

Ask Solem | 13 Jun 14:04 2014

Re: Tasks run multiple times


On Jun 13, 2014, at 11:36 AM, Mihnea Giurgea <skipy3-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

The following methods were not called during the experiment (from kombu.transport.redis):
  • Channel._do_restore_message
  • Channel._get
  • Channel._put
  • Channel._put_fanout
  • virtual.Channel._restore
It seems that the duplicate message has the redelivered flag set to True, although the only code that sets that flag from kombu.transport.redis was never called. What does that mean?

Also note that the message was not enqueued by any other worker, since no other worker was running and the running one never called Channel._put.



Very mysterious indeed, the only functions that perform LPUSH/RPUSH are _do_restore_message and _put.

My hunch would be that there are more workers here, one or more that you did not shut down properly
that are still configured to use the default visibility timeout of 1 hour.

It’s the only explanation that makes sense to me, and it’s very easy to over look a stray worker running
somewhere.

-- 
Ask Solem
twitter.com/asksol | +44 07454281208

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

Dmitry Malinovsky | 13 Jun 12:17 2014
Picon

How to make sure that specific queue is always in CELERY_QUEUES?

Hi Ask,

What is the preferred way to make sure that specific queue is always present in CELERY_QUEUES?
I want to be able to allow users to specify additional queues from the cli, or with configuration object, but on the client side I always need to have an auth queue.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Ludwig Schmidt-Hackenberg | 12 Jun 11:35 2014

Celery makes RabbitMQ use so much virtual memory that it crashes.

Hi, 

till now I was very pleased with celery. I use Celery to distribute task on my compute server. I started with ~10k tasks which worked fine. Now I moved to ~100k tasks and after some time RabbitMQ crashes with following error message:

eheap_alloc: Cannot allocate 1098556536 bytes of memory (of type "heap").
Aborted (core dumped)

I guess celery is creating too many queues, but I don't know which setting I sent wrongly.

This is my celeryconfig.py
from utilities import get_number_cpu_cores

# List of modules to import when celery starts.
CELERY_IMPORTS = (...)

BROKER_URL = 'amqp://guest:guest <at> localhost:5672//'
CELERY_RESULT_BACKEND = "amqp"

CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']  # Ignore other content
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Berlin'
CELERY_ENABLE_UTC = True
CELERYD_CONCURRENCY = get_number_cpu_cores() * 2

I tried to log what RabbitMQ is doing with 
 curl -i -u guest:guest http://localhost:15672/api/vhosts >> vhosts.log
 curl -i -u guest:guest http://localhost:15672/api/queues >> queues.log

This is the last entry of vhosts.log: 
Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
Date: Wed, 11 Jun 2014 19:03:34 GMT
Content-Type: application/json
Content-Length: 627
Cache-Control: no-cache

[{"message_stats":{"ack":59994,"ack_details":{"rate":0.0},"deliver":59994,"deliver_details":{"rate":0.0},"deliver_get":121603,"deliver_get_details":{"rate":0.0},"deliver_no_ack":1630,"deliver_no_ack_details":{"rate":0.0},"get":59979,"get_details":{"rate":0.0},"publish":118167,"publish_details":{"rate":0.0}},"messages":69099,"messages_details":{"rate":0.0},"messages_ready":69099,"messages_ready_details":{"rate":0.0},"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0},"recv_oct":118595149,"recv_oct_details":{"rate":0.0},"send_oct":91945666,"send_oct_details":{"rate":0.0},"name":"/","tracing":false}]

The last entry of queues.log is cut off and the one before is 64MB. 

Tell me if you need more information!
Thanks!

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
phill | 11 Jun 17:16 2014
Picon

Chunks ignoring routing, even when manually specifying (v3.011)

I ran into the same problem that was brought up last year here: https://groups.google.com/forum/#!topic/celery-users/zlGvHa1q0eQ

When I run chunks, they create starmap tasks and those aren't routed per the ROUTES map of the task being chunked... Ask suggests specifying the queue manually when issuing the chunk, but that appears to be ignored for me also. Does anyone know why or what else I can do to debug?

I'm on Celery 3.0.11 and using Redis as my backend.

Thanks in advance,
Phill

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Jan Vilhuber | 10 Jun 21:43 2014

NEWOBJ class argument has NULL tp_new error

We keep running into this, and googling hasn't really gotten me much data to work with. I understand it's multiprocessing related, but I'm lost on where in celery this could be happening/coming from.

We are running:
Celery 3.0.16
CentOS 6.4
Django 1.6.2
python 2.7.6

[2014-06-06 06:00:01,562: ERROR/MainProcess] Thread 'ResultHandler' crashed: UnpicklingError('NEWOBJ class argument has NULL tp_new',)
Traceback (most recent call last):
  File "/apps/mam/nukona/python/lib/python2.7/site-packages/billiard/pool.py", line 469, in run
    return self.body()
  File "/apps/mam/nukona/python/lib/python2.7/site-packages/billiard/pool.py", line 837, in body
    self.finish_at_shutdown()
  File "/apps/mam/nukona/python/lib/python2.7/site-packages/billiard/pool.py", line 854, in finish_at_shutdown
    ready, task = poll(1.0)
  File "/apps/mam/nukona/python/lib/python2.7/site-packages/billiard/pool.py", line 1267, in _poll_result
    return True, self._quick_get()
UnpicklingError: NEWOBJ class argument has NULL tp_new

We can't seem to make any headway on this as this doesn't appear to be from any of our tasks. We have CELERY_IGNORE_RESULT=True, so no results should ever be returned (or however celery handles this). So I'm unsure where to even start looking. We also can't seem to reproduce this, other than on production networks, where we can't add debugging code.

Any thoughts or hints on how to track this down? Better yet, is it a known issue and I can just upgrade to pull in a fix?

jan

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Phill | 10 Jun 17:21 2014
Picon

Limiting concurrency of grouped tasks to a subset of the available worker pool

'Ello!

Here's the thing, say I have a worker pool with 30 workers in it. 
Sometimes I want to run a group of tasks but limit the concurrency 
of that group to some smaller number.. Say, no more than 5 of them 
running at the same time. I'm wondering if there's a handy pattern 
for doing that, which I've overlooked.

First off, I know this is poor form. Ideally my worker pools are 
tuned for concurrency and my tasks are all dissolved into nice little 
chunks that can be run as fast as the pool can handle them. However, 
in practice from time to time I run into challenges where I have tasks 
that strain some external resource like a database or remote API 
connection and I can't afford to run them at a concurrency that's as 
high as some of the other tasks in the pool. Additionally, I'm trying 
to resist spinning up lots of new specialized worker pools that will 
end up sitting idle most of the time.

Is this a problem anyone else has tackled? Curious if there's a 
convenient way to pull this off.

Thanks in advance,
Phill

Jonathan Lefman | 10 Jun 04:39 2014

Limit rate of tasks to queue

Hi. I am using a SaaS rabbitmq service. This service limits the number of messages that can be put into the queue per second. I have many tasks that I need to put into the queue; more than can be put into the queue per second. What is the easiest and most reliable way to put tasks into the queue using a rate limiter?

Thank you.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Devang | 10 Jun 04:01 2014
Picon

Using celery worker --detach option in production for Django

Hi,

In the documentation, it is mentioned that
"In a production environment you will want to run the worker in the background as a daemon - see Running the worker as a daemon -..."

However, it looks like there is a new and easier way to daemonize the workers using the celery worker --detach option.
So from 3.1 onwards, can we skip using celeryd, celery.service, supervisord, etc.. to launch celery workers and simply call celery worker command?

I am trying to deploy Celery in AWS Elastic Beanstalk, so it would seem that using a simple celery worker command would be much easier than juggling with Custom AMIs and supervisord scripts :)

Thanks,
Devang

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Joseph Wenninger | 9 Jun 09:25 2014
Picon

rabbitmq queue cleanup?

Hi!

Is there a simple way to clean up rabbitmq result queues?

I know there is an expire timout, but I know that my task has finished and I got the results, after querying it, so I do not need it to stay in the queue till some arbitrary timeout. 

I thought AsyncResult.forget() would be good for me, but it is not implemented for the rabbit backend.

I'm using djcelery, celery 3.1.11 and rabbitmq-server 3.1.5-1 (debian)

What is the best way to clean up no longer needed results

Best regards
Joseph

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

Gmane