Ask Solem | 12 Oct 15:24 2012

Re: vmware vmotion or snapshot + rabbitmq + celery = Too many heartbeats missed


On 11 Oct 2012, at 08:01, Tom Pepper <peppernicus@...> wrote:

> Hi all:
> 
> I've noticed that during snapshot events and nightly backups (which take a snapshot), as well as
lengthier vmotion events of the rabbitmq server or nodes, that celeryd (kombu, really) will emit the following:
> 
> ERROR/MainProcess] Error in timer: ConnectionError('Too many heartbeats missed', None, None, None,
'')#012Traceback (most recent call last):#012  File
"/root/toro/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 93, in
apply_entry#012    entry()#012  File
"/root/toro/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 49, in
__call__#012    return self.fun(*self.args, **self.kwargs)#012  File
"/root/toro/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 150, in
_reschedules#012    return fun(*args, **kwargs)#012  File
"/root/toro/local/lib/python2.7/site-packages/kombu/connection.py", line 186, in
heartbeat_check#012    return self.transport.heartbeat_check(self.connection, rate=rate)#012 
File "/root/toro/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in
heartbeat_check#012    return connection.heartbeat_tick(rate=rate)#012  File
"/root/toro/local/lib/python2.7/site-packages/amqp/connection.py", line 836, in
heartbeat_tick#012    raise ConnectionError('Too many heartbeats missed')#012ConnectionError: Too
many heartbeats missed
> 
> followed shortly by:
> 
> CRITICAL/MainProcess] Couldn't ack 4, reason:error(32, 'Broken pipe')#012Traceback (most recent
call last):#012  File "/root/toro/local/lib/python2.7/site-packages/kombu/transport/base.py",
line 104, in ack_log_error#012    self.ack()#012  File
"/root/toro/local/lib/python2.7/site-packages/kombu/transport/base.py", line 99, in ack#012   
(Continue reading)

Ask Solem | 12 Oct 15:34 2012

Re: Help in "selling" Celery


On 11 Oct 2012, at 22:20, Miki Tebeka <miki.tebeka@...> wrote:

> Greetings,
> 
> We're evaluating several frameworks at work and one of the leading candidates is Celery.
> Is there material out there that will help me "sell" Celery?
> 
> I'm mostly looking for project stability, big companies using it (other than what's in the wiki), support
... 
> But anything else that will help is highly appreciated.

My policy is that it's up to the companies to reveal what they are using,
and the wiki is a place where they can do so, but I fear that most people
aren't even aware of it.

I don't have an overview of who is using it, but there are many.
You can search for job listings mentioning Celery.
I've seen Amazon, Ebay and JP Morgan Chase and many others, but I have no idea
if and if so, what they're using it for.

Being a popular tool there are many people skilled in using it,
and commercial support is available (just contact me).

--

-- 
Ask Solem
twitter.com/asksol | +44 (0)7713357179
ask@...

(Continue reading)

Miki Tebeka | 12 Oct 15:56 2012
Picon

Re: Help in "selling" Celery

Thanks!

On Friday, October 12, 2012 6:34:54 AM UTC-7, Ask Solem wrote:


On 11 Oct 2012, at 22:20, Miki Tebeka <miki....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Greetings,
>
> We're evaluating several frameworks at work and one of the leading candidates is Celery.
> Is there material out there that will help me "sell" Celery?
>
> I'm mostly looking for project stability, big companies using it (other than what's in the wiki), support ...
> But anything else that will help is highly appreciated.

My policy is that it's up to the companies to reveal what they are using,
and the wiki is a place where they can do so, but I fear that most people
aren't even aware of it.

I don't have an overview of who is using it, but there are many.
You can search for job listings mentioning Celery.
I've seen Amazon, Ebay and JP Morgan Chase and many others, but I have no idea
if and if so, what they're using it for.

Being a popular tool there are many people skilled in using it,
and commercial support is available (just contact me).

--
Ask Solem
twitter.com/asksol | +44 (0)7713357179
a...-oFFgF2hNjUOoClj4AeEUq9i2O/JbrIOy@public.gmane.org

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/U-92dFS_pAQJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Harel Malka | 12 Oct 15:59 2012

Re: Help in "selling" Celery

I can't divulge too many details but I can tell you I'm running a celery salad (with a rabbitmq queue) here hitting APIs and lifting heavy things to the amount of 4-5 million requests a day without breaking a sweat. So far the problems we encountered were our own fault and not celery's.

Harel


On Fri, Oct 12, 2012 at 2:56 PM, Miki Tebeka <miki.tebeka-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Thanks!

On Friday, October 12, 2012 6:34:54 AM UTC-7, Ask Solem wrote:

On 11 Oct 2012, at 22:20, Miki Tebeka <miki....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Greetings,
>
> We're evaluating several frameworks at work and one of the leading candidates is Celery.
> Is there material out there that will help me "sell" Celery?
>
> I'm mostly looking for project stability, big companies using it (other than what's in the wiki), support ...
> But anything else that will help is highly appreciated.

My policy is that it's up to the companies to reveal what they are using,
and the wiki is a place where they can do so, but I fear that most people
aren't even aware of it.

I don't have an overview of who is using it, but there are many.
You can search for job listings mentioning Celery.
I've seen Amazon, Ebay and JP Morgan Chase and many others, but I have no idea
if and if so, what they're using it for.

Being a popular tool there are many people skilled in using it,
and commercial support is available (just contact me).

--
Ask Solem
twitter.com/asksol | +44 (0)7713357179
a...-oFFgF2hNjUOoClj4AeEUq9i2O/JbrIOy@public.gmane.org

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/U-92dFS_pAQJ.

To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.



--

--8<----------------------------------------------------
Harel Malka
Software Architect & Developer
http://www.harelmalka.com
http://www.freecrm.com


--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Tom Pepper | 12 Oct 16:42 2012
Picon

Re: vmware vmotion or snapshot + rabbitmq + celery = Too many heartbeats missed

Thanks, Ask.

Each of the instances is running ntpd, we have a stratum 1 timeserver synced to GPS in the same facility. 
Clock drift is more or less negligible, and well under a tenth of a second worst case.

I'll try extending the heartbeat - pretty sure that's going to fix the issue, just wasn't sure where to
change config.  Thanks!

-t

On Oct 12, 2012, at 6:24 AM, Ask Solem <ask@...> wrote:

> 
> On 11 Oct 2012, at 08:01, Tom Pepper <peppernicus@...> wrote:
> 
>> Hi all:
>> 
>> I've noticed that during snapshot events and nightly backups (which take a snapshot), as well as
lengthier vmotion events of the rabbitmq server or nodes, that celeryd (kombu, really) will emit the following:
>> 
>> ERROR/MainProcess] Error in timer: ConnectionError('Too many heartbeats missed', None, None, None,
'')#012Traceback (most recent call last):#012  File
"/root/toro/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 93, in
apply_entry#012    entry()#012  File
"/root/toro/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 49, in
__call__#012    return self.fun(*self.args, **self.kwargs)#012  File
"/root/toro/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 150, in
_reschedules#012    return fun(*args, **kwargs)#012  File
"/root/toro/local/lib/python2.7/site-packages/kombu/connection.py", line 186, in
heartbeat_check#012    return self.transport.heartbeat_check(self.connection, rate=rate)#012 
File "/root/toro/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in
heartbeat_check#012    return connection.heartbeat_tick(rate=rate)#012  File
"/root/toro/local/lib/python2.7/site-packages/amqp/connection.py", line 836, in
heartbeat_tick#012    raise ConnectionError('Too many heartbeats missed')#012ConnectionError: Too
many heartbeats missed
>> 
>> followed shortly by:
>> 
>> CRITICAL/MainProcess] Couldn't ack 4, reason:error(32, 'Broken pipe')#012Traceback (most recent
call last):#012  File "/root/toro/local/lib/python2.7/site-packages/kombu/transport/base.py",
line 104, in ack_log_error#012    self.ack()#012  File
"/root/toro/local/lib/python2.7/site-packages/kombu/transport/base.py", line 99, in ack#012   
self.channel.basic_ack(self.delivery_tag)#012  File
"/root/toro/local/lib/python2.7/site-packages/amqp/channel.py", line 1556, in basic_ack#012   
self._send_method((60, 80), args)#012  File
"/root/toro/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 58, in
_send_method#012    self.channel_id, method_sig, args, content)#012  File
"/root/toro/local/lib/python2.7/site-packages/amqp/method_framing.py", line 216, in
write_method#012    write_frame(1, channel, payload)#012  File
"/root/toro/local/lib/python2.7/site-packages/amqp/transport.py", line 149, in
write_frame#012    frame_type, channel, size, payload, 0xce))#012  File
"/usr/lib/python2.7/socket.py", line 224, in meth#012    return
getattr(self._sock,name)(*args)#012error: [Errno 32] Broken pipe
>> 
>> Once this happens, the celeryd instances show in top as consuming 100% CPU per node started and no longer
process any tasks until they are restarted.
>> 
> 
> Either the broker did actually miss the heartbeat or the system time is unreliable.
> (time in virtualized environments is often unreliable, but not sure if that is
> at play here).
> 
> You could try increasing the heartbeat rate (e.g. BROKER_HEARTBEAT=30)
> 
> There also a constant in the code called AMQHEARTBEAT_RATE, there's no setting for this
> yet but you could change this in the source code to modify how often the heartbeats
> are checked.  The default is to check twice the rate of the heartbeat value, and I would guess
> decreasing this could adjust for clock instability.
> 
> 
> As for why the process uses 100% CPU I have no idea, but it sounds like a bug.
> 
> -- 
> Ask Solem
> twitter.com/asksol | +44 (0)7713357179
> 
> -- 
> You received this message because you are subscribed to the Google Groups "celery-users" group.
> To post to this group, send email to celery-users@...
> To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
> 

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Ask Solem | 12 Oct 17:00 2012

Re: djcelery leaving behind open database transactions


On 10 Oct 2012, at 18:53, Matt Long <matt@...> wrote:

> The moral of the story is that threading can have unintended side-effects if you don't fully understand
them :)
> 

I think you mean "threading have unintended side-effects and no one fully understands them" :)

--

-- 
Ask Solem
twitter.com/asksol | +44 (0)7713357179

Miki Tebeka | 12 Oct 21:24 2012
Picon

Re: Help in "selling" Celery

That's a great quote, thanks.

On Friday, October 12, 2012 6:59:10 AM UTC-7, Harel Malka wrote:

I can't divulge too many details but I can tell you I'm running a celery salad (with a rabbitmq queue) here hitting APIs and lifting heavy things to the amount of 4-5 million requests a day without breaking a sweat. So far the problems we encountered were our own fault and not celery's.
Harel


On Fri, Oct 12, 2012 at 2:56 PM, Miki Tebeka <miki....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Thanks!

On Friday, October 12, 2012 6:34:54 AM UTC-7, Ask Solem wrote:

On 11 Oct 2012, at 22:20, Miki Tebeka <miki....-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Greetings,
>
> We're evaluating several frameworks at work and one of the leading candidates is Celery.
> Is there material out there that will help me "sell" Celery?
>
> I'm mostly looking for project stability, big companies using it (other than what's in the wiki), support ...
> But anything else that will help is highly appreciated.

My policy is that it's up to the companies to reveal what they are using,
and the wiki is a place where they can do so, but I fear that most people
aren't even aware of it.

I don't have an overview of who is using it, but there are many.
You can search for job listings mentioning Celery.
I've seen Amazon, Ebay and JP Morgan Chase and many others, but I have no idea
if and if so, what they're using it for.

Being a popular tool there are many people skilled in using it,
and commercial support is available (just contact me).

--
Ask Solem
twitter.com/asksol | +44 (0)7713357179
a...-oFFgF2hNjUOoClj4AeEUq9i2O/JbrIOy@public.gmane.org

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/U-92dFS_pAQJ.

To post to this group, send email to celery...-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to celery-users... <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.



--

--8<----------------------------------------------------
Harel Malka
Software Architect & Developer
http://www.harelmalka.com
http://www.freecrm.com


--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/ZZNgsgQZa-MJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Gmane