Salvatore Iovene | 2 Dec 06:41 2011
Picon

Model object losing set attributes when passed to callback function

Hi,

in one of my tasks I set some property on a model object: Image. Here's some relevant extract of the code:

<at> task()
def store_image(image, callback = None):
  (w, h) = save_image_to_amazon_s3(image)
  image.w = w
  image.h = h
  print image.w, image.h

  if callback is not None:
    subtask(callback).delay(image)


And here is the callback:

<at> task()
def image_stored_cb(image):
  print image.w, image.h

And this is how I call the first task:

store_image.delay(image, callback = image_stored_cb)


Now, as you guessed from the subject of my message, when I print image.w and image.h in the callback, they are both empty, even though they are printed correctly in the first task.
If, in the first task, I do image.save() before calling the callback, everything works fine, but the thing is that I can't save the image there, because the user might have deleted it in the meantime.

Is this the expected behavior or is there something I'm doing wrong? Do you have any recommendations?

Thanks!

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/IbAbqQG_tvUJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Andrew | 2 Dec 14:55 2011
Picon

Using the new x-ha-policy option in RabbitMQ w/ Celery

So, it looks like there's a BROKER_TRANSPORT_OPTIONS accepts arguments
for initializing the connection, but I'm having trouble finding out
how we can pass in generic queue parameters.  In particular, the `x-ha-
policy` option.  Is there a way of passing this parameter to Celery on
queue creation?

I appreciate the help.

rd-london | 2 Dec 15:44 2011
Picon

RDB suffers a timeout

Am experimenting with debugging using RDB (see
http://ask.github.com/celery/tutorials/debugging.html).

Looking at the celeryd logfile, the code stops, but then very shortly
times out (say 1-2 seconds). I see the following in the logfile:

**************** logfile snippet **************************

[2011-12-02 14:40:42,203: WARNING/PoolWorker-2] Remote Debugger:6901:
Waiting for client...
[2011-12-02 14:40:47,224: ERROR/MainProcess] Task
build.tasks.build_minify[d90fc0e2-5010-4251-a573-e43527c93394] raised
exception: timeout('timed out',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/execute/
trace.py", line 47, in trace
    return cls(states.SUCCESS, retval=fun(*args, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/celery/app/task/
__init__.py", line 247, in __call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/
__init__.py", line 175, in run
    return fun(*args, **kwargs)
  File "/home/roland/WORK/DEV/DJANGO/apps/build/tasks.py", line 21, in
build_minify
    rdb.set_trace()
  File "/usr/local/lib/python2.7/dist-packages/celery/contrib/rdb.py",
line 156, in set_trace
    return debugger().set_trace(frame)
  File "/usr/local/lib/python2.7/dist-packages/celery/contrib/rdb.py",
line 148, in debugger
    rdb = _current[0] = Rdb()
  File "/usr/local/lib/python2.7/dist-packages/celery/contrib/rdb.py",
line 101, in __init__
    self._client, address = self._sock.accept()
  File "/usr/lib/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
timeout: timed out
Traceback (most recent call last):
**************** logfile snippet **************************

Anyone any suggestions?

Many thanks,
R

Redmumba | 2 Dec 16:51 2011
Picon

Re: Using the new x-ha-policy option in RabbitMQ w/ Celery

Despite searching all morning, I ended up finding it immediately after posting, hidden away on the issues page:

https://github.com/ask/celery/issues/483

Hope this helps anyone else who is looking around!

Andrew

On Fri, Dec 2, 2011 at 8:55 AM, Andrew <redmumba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
So, it looks like there's a BROKER_TRANSPORT_OPTIONS accepts arguments
for initializing the connection, but I'm having trouble finding out
how we can pass in generic queue parameters.  In particular, the `x-ha-
policy` option.  Is there a way of passing this parameter to Celery on
queue creation?

I appreciate the help.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
David | 3 Dec 01:06 2011
Picon

Strategies for dealing w/ database deadlock & lock wait timeout in tasks

I'm trying to devise the best strategy for dealing with database deadlock and lock wait timeouts within tasks that occasionally will all try to update the same row (based on events from an external API). 

I'm running MySQL 5.1.41 in Read Committed, which is the only thing that the Celery FAQ mentions: http://celery.readthedocs.org/en/latest/faq.html#mysql-is-throwing-deadlock-errors-what-can-i-do). Innodb_lock_wait_timeout has been bumped up to 300, but I'm wondering if a better strategy would be to make this value something really low, add a decorator for tasks that need to update a row, and upon detecting a deadlock or lock wait, just rollback and retry the task. Thanks!

(Running Celery 2.4.4 and Django-celery 2.4.2)

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/Abg3rCsWV0UJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Harel Malka | 3 Dec 01:25 2011

Re: Strategies for dealing w/ database deadlock & lock wait timeout in tasks

Might not be viable for your scenario, but perhaps you can decouple the database update as a separate task running on a separate queue. 
Although unlike your scenario, I've had a case where a very large amount of inserts would stall the entire database. I took the actual insert action into a separate task and a dedicated queue with 1 or two instances trickling the inserts slowly and reducing the impact on the database. 


Also, are you running under Django? If so try to run it in auto commit mode. For me Django was destroying the database and effectively killing apache due to idle transactions when I ran it without auto commit. 

Harel

On Sat, Dec 3, 2011 at 12:06 AM, David <dtran320-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
I'm trying to devise the best strategy for dealing with database deadlock and lock wait timeouts within tasks that occasionally will all try to update the same row (based on events from an external API). 

I'm running MySQL 5.1.41 in Read Committed, which is the only thing that the Celery FAQ mentions: http://celery.readthedocs.org/en/latest/faq.html#mysql-is-throwing-deadlock-errors-what-can-i-do). Innodb_lock_wait_timeout has been bumped up to 300, but I'm wondering if a better strategy would be to make this value something really low, add a decorator for tasks that need to update a row, and upon detecting a deadlock or lock wait, just rollback and retry the task. Thanks!

(Running Celery 2.4.4 and Django-celery 2.4.2)

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/Abg3rCsWV0UJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.



--

--8<----------------------------------------------------
Harel Malka
Software Architect & Developer
http://www.harelmalka.com
http://www.freecrm.com


--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
David | 3 Dec 01:41 2011
Picon

Re: Strategies for dealing w/ database deadlock & lock wait timeout in tasks

Thanks for the response Harel!


Yes, running under Django - isn't autocommit the default?

Decoupling the update sounds like an interesting short-term solution - rather than simply trickling inserts/updates slowly, this approach would give you the opportunity to batch together some operations or re-order them to reduce deadlock, although this would be a little more involved than just factoring out database code and putting them in separate tasks. Any other thoughts/approaches?

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/1samlDmd53EJ.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Harel Malka | 3 Dec 01:57 2011

Re: Strategies for dealing w/ database deadlock & lock wait timeout in tasks

Django, as I painfully learned is not on autocommit by default. 

Add 'autocommit': True to your database config dict. I have no idea why this is not the default because really, django is a database & apache killer without it.
As far as the decoupled update task - try to have the decoupled task be the final update/insert statements. Try to make it atomic in the sense that the task receives simple values and puts them in a table. 

Another option is to limit the number of workers running your problematic task, and yet another option is to use something like memcached to lock out the process from other tasks based on some criteria. 
I.e., if you're working on record types A and B you can place a key in memcached A-12-B-34=True which then means that record A with id of 12 and record B with id of 34 are locked for updates so any task that attempts to work on the same objects will skip it and move on. You clear the lock when you're finished.

On Sat, Dec 3, 2011 at 12:41 AM, David <dtran320-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Thanks for the response Harel!

Yes, running under Django - isn't autocommit the default?

Decoupling the update sounds like an interesting short-term solution - rather than simply trickling inserts/updates slowly, this approach would give you the opportunity to batch together some operations or re-order them to reduce deadlock, although this would be a little more involved than just factoring out database code and putting them in separate tasks. Any other thoughts/approaches?

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/1samlDmd53EJ.

To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.



--

--8<----------------------------------------------------
Harel Malka
Software Architect & Developer
http://www.harelmalka.com
http://www.freecrm.com


--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Aron Griffis | 4 Dec 02:23 2011
Picon

Re: singleton tasks

Hi Dmitry, did you come up with something for this? I've been curious about this, particularly to coalesce concurrent duplicate GETs to a backend HTTP/REST service.


Thanks,
Aron

On Wed, Oct 26, 2011 at 2:28 PM, dmitry b <dmitry.maven <at> gmail.com> wrote:
I remember some talk a while ago about the possibility of having
singleton tasks (tasks which are guaranteed to only have a single
concurrently running instance).  Is there some work being done in this
area or should I just create my own abstraction for that?


Thanks
D.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.


--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
Kirill Panshin | 4 Dec 15:11 2011

Re: how to set up shared Celery/Django logging

Harel, that's actually a very good approach. If you could make a
package that would be great. I can contribute if you'll decide to do
this.

BR, Kirill Panshin

On Oct 25, 6:05 pm, Harel Malka <ha...@...> wrote:
> I've taken a different approach to logging: I have many a growing number of
> celery servers and I wanted a shared logging mechanism for them. I ended up
> writing a custom object that logs all data into a mongodb database. Mongodb
> inserts are 'fire and forget' so they are very quick. I have a (crude but
> effective) admin page I use to view the log which lets me filter items by
> logger, server, related object ids etc. The object is quite specialized to
> my application but can be easily adapted as a generic logger.
> For me this has worked very well, and I don't have to go into servers to dig
> into text log files. Its all there in one place.
> Harel
>
>
>
>
>
>
>
>
>
> On Fri, Oct 21, 2011 at 7:54 PM, dmitry b <dmitry.ma...@...> wrote:
> > Hi,
>
> > I have shared code base for both celery and django functionality.
> > This means that some library level functions may get called from
> > Celery or from Django.  How do I use (set up) logging so that when I
> > do logging from such a shared library function, the output ends up
> > being handled according to Celery or Django logging preferences
> > depending on the context of the call chain?
>
> > Thanks
> > Dmitry
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "celery-users" group.
> > To post to this group, send email to celery-users@...
> > To unsubscribe from this group, send email to
> > celery-users+unsubscribe@...
> > For more options, visit this group at
> >http://groups.google.com/group/celery-users?hl=en.
>
> --
>
> --8<----------------------------------------------------
> Harel Malka
> Software Architect & Developerhttp://www.harelmalka.comhttp://www.freecrm.com

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.


Gmane