Re: ALWAYS_EAGE and 2.5.0

Re: ALWAYS_EAGE and 2.5.0

celery countdown task with flask and redis

I have celery 2.5 and redis working with python 2.7. I am able to run
an async task with no problem. However, when I try to use the
countdown argument, the task gets scheduled as evidenced by the tasks
showing up in celeryev. But each of these tasks shows up with state
set to "RECEIVED" and the state never changes, even when the date/time
passes the countdown mark after 60 seconds. Here are the relevant
snips of code:

 <at> celery.task(name='app.whatever')
def whatever(app_key, member_list, msg, content_type):
    return do_it(app_key, member_list, msg, content_type=content_type)

res = whatever.apply_async((app_key, member_list, msg, 'text/plain'),
None, countdown=60)

Any ideas what I might be doing wrong?

Thanks,
David

Re: Running celeryd-multi as a service

I have found the problem, in the init script i have added this "/usr/local/bin/" 
as there i have the celeryd-multi

export PATH="${PATH:+$PATH:}/usr/sbin:/sbin:/usr/local/bin/"

Thank you anyway

Re: ALWAYS_EAGE and 2.5.0

2.5.1 keeps spawning poolWorkers

Cheers,

just tryed 2.5.1 with my project and it seems i've missed something?
After starting celeryd, two pool workers are started each second.
And it wont stop when reaches BROKER_POOL_LIMIT, 400 idle workers and 
still counting ;)
...
[2012-03-02 16:31:31,736: INFO/PoolWorker-352] child process calling 
self.run()
[2012-03-02 16:31:32,549: INFO/PoolWorker-354] child process calling 
self.run()
[2012-03-02 16:31:32,551: INFO/PoolWorker-353] child process calling 
self.run()
...
Backend is RabbitMQ, i've deleted the old exchange, purged the queues 
and followed
http://celery.readthedocs.org/en/latest/whatsnew-2.5.html

somebody got a hint for me?

yours
julien

Re: ALWAYS_EAGE and 2.5.0

I found a better solution which was to use a custom loader:

class CustomLoader(DjangoLoader):

    def on_task_init(self, task_id, task):
        """Called before every task."""
        request = getattr(task, 'request', None)
        # Don't close DB for eager tasks
        if not getattr(request, 'is_eager', False):
            self.close_database()

If this seems like a reasonable solution to everyone I'd be happy to
put together a pull request for django-celery.

On Mar 2, 10:14 am, Mark Lavin <markdla...@...> wrote:
> I ran into this issue as well and last night I was able to trace it down to
> this change in celery:
>
> https://github.com/ask/celery/commit/9b94556b8c0bf0889d785058e29e7ca3...
>
> This refactored the task execution and created a tracer for all tasks
> including when running as ALWAYS_EAGER. With this change eager tasks still
> call loader.on_task_init which if you are using django-celery closes the DB
> connection. Since the tests run in a transaction and you are likely not
> running READ UNCOMMITED the task's new connection cannot access any fixture
> data created in the test case. For me this breaks my tasks which index data
> with haystack via the celery queue (raising DoesNotExist errors for data I
> just created). Disabling the django-celery loader while running the test
> suite works around the issue but it's far from ideal. Hopefully someone
> will have a better idea on how to solve this issue.
>
>
>
>
>
>
>
> On Thursday, March 1, 2012 4:26:36 PM UTC-5, Steve wrote:
>
> > On Thu, Mar 1, 2012 at 7:24 AM, Steve Schwarz <agilityn...@...>wrote:
>
> >> On Wed, Feb 29, 2012 at 1:22 PM, Zoltan Szalai <defaultd...@...>wrote:
>
> >>> i'm not sure how (yet) but i'll try to narrow it to a simple example if
> >>> i got some spare time.
>
> >>> till then, thanks for the answer
>
> >> FWIW I ran into a problem with some tests failing yesterday after
> >> upgrading as well. I'll be working on them in the next couple hours and
> >> will post if I find anything. I don't have the code here but IIRC the tests
> >> were creating and storing objects using the .create() model method and
> >> verifying that post_save signaled functions were being called. I don't
> >> think any celery tasks were even involved... I was going to see if the
> >> change in test runner had something to do with it.
>
> > I've spent a little more time on this and here's what I found.
>
> > I did pip install --upgrade django-celery in my virtualenv (and
> > consequently automatically upgraded celery and and related packages) and
> > got django-celery 2.5.1/celery 2.5.1
> > Tests invoking tasks (via .delay()) were failing (irregardless of CELERY_ALWAYS_EAGER
> > settings) the behavior was querysets for model instances created prior to
> > invoking the task always returned the empty set...
> > If if reverted only celery: pip install celery==2.4.6 all tests passed.
>
> > Best Regards,
> > Steve
> >http://tech.agilitynerd.com/
>
> On Thursday, March 1, 2012 4:26:36 PM UTC-5, Steve wrote:
>
> > On Thu, Mar 1, 2012 at 7:24 AM, Steve Schwarz <agilityn...@...>wrote:
>
> >> On Wed, Feb 29, 2012 at 1:22 PM, Zoltan Szalai <defaultd...@...>wrote:
>
> >>> i'm not sure how (yet) but i'll try to narrow it to a simple example if
> >>> i got some spare time.
>
> >>> till then, thanks for the answer
>
> >> FWIW I ran into a problem with some tests failing yesterday after
> >> upgrading as well. I'll be working on them in the next couple hours and
> >> will post if I find anything. I don't have the code here but IIRC the tests
> >> were creating and storing objects using the .create() model method and
> >> verifying that post_save signaled functions were being called. I don't
> >> think any celery tasks were even involved... I was going to see if the
> >> change in test runner had something to do with it.
>
> > I've spent a little more time on this and here's what I found.
>
> > I did pip install --upgrade django-celery in my virtualenv (and
> > consequently automatically upgraded celery and and related packages) and
> > got django-celery 2.5.1/celery 2.5.1
> > Tests invoking tasks (via .delay()) were failing (irregardless of CELERY_ALWAYS_EAGER
> > settings) the behavior was querysets for model instances created prior to
> > invoking the task always returned the empty set...
> > If if reverted only celery: pip install celery==2.4.6 all tests passed.
>
> > Best Regards,
> > Steve
> >http://tech.agilitynerd.com/
>
> On Thursday, March 1, 2012 4:26:36 PM UTC-5, Steve wrote:
>
> > On Thu, Mar 1, 2012 at 7:24 AM, Steve Schwarz <agilityn...@...>wrote:
>
> >> On Wed, Feb 29, 2012 at 1:22 PM, Zoltan Szalai <defaultd...@...>wrote:
>
> >>> i'm not sure how (yet) but i'll try to narrow it to a simple example if
> >>> i got some spare time.
>
> >>> till then, thanks for the answer
>
> >> FWIW I ran into a problem with some tests failing yesterday after
> >> upgrading as well. I'll be working on them in the next couple hours and
> >> will post if I find anything. I don't have the code here but IIRC the tests
> >> were creating and storing objects using the .create() model method and
> >> verifying that post_save signaled functions were being called. I don't
> >> think any celery tasks were even involved... I was going to see if the
> >> change in test runner had something to do with it.
>
> > I've spent a little more time on this and here's what I found.
>
> > I did pip install --upgrade django-celery in my virtualenv (and
> > consequently automatically upgraded celery and and related packages) and
> > got django-celery 2.5.1/celery 2.5.1
> > Tests invoking tasks (via .delay()) were failing (irregardless of CELERY_ALWAYS_EAGER
> > settings) the behavior was querysets for model instances created prior to
> > invoking the task always returned the empty set...
> > If if reverted only celery: pip install celery==2.4.6 all tests passed.
>
> > Best Regards,
> > Steve
> >http://tech.agilitynerd.com/
>
> On Thursday, March 1, 2012 4:26:36 PM UTC-5, Steve wrote:
>
> > On Thu, Mar 1, 2012 at 7:24 AM, Steve Schwarz <agilityn...@...>wrote:
>
> >> On Wed, Feb 29, 2012 at 1:22 PM, Zoltan Szalai <defaultd...@...>wrote:
>
> >>> i'm not sure how (yet) but i'll try to narrow it to a simple example if
> >>> i got some spare time.
>
> >>> till then, thanks for the answer
>
> >> FWIW I ran into a problem with some tests failing yesterday after
> >> upgrading as well. I'll be working on them in the next couple hours and
> >> will post if I find anything. I don't have the code here but IIRC the tests
> >> were creating and storing objects using the .create() model method and
> >> verifying that post_save signaled functions were being called. I don't
> >> think any celery tasks were even involved... I was going to see if the
> >> change in test runner had something to do with it.
>
> > I've spent a little more time on this and here's what I found.
>
> > I did pip install --upgrade django-celery in my virtualenv (and
> > consequently automatically upgraded celery and and related packages) and
> > got django-celery 2.5.1/celery 2.5.1
> > Tests invoking tasks (via .delay()) were failing (irregardless of CELERY_ALWAYS_EAGER
> > settings) the behavior was querysets for model instances created prior to
> > invoking the task always returned the empty set...
> > If if reverted only celery: pip install celery==2.4.6 all tests passed.
>
> > Best Regards,
> > Steve
> >http://tech.agilitynerd.com/
>
> On Thursday, March 1, 2012 4:26:36 PM UTC-5, Steve wrote:
>
> > On Thu, Mar 1, 2012 at 7:24 AM, Steve Schwarz <agilityn...@...>wrote:
>
> >> On Wed, Feb 29, 2012 at 1:22 PM, Zoltan Szalai <defaultd...@...>wrote:
>
> >>> i'm not sure how (yet) but i'll try to narrow it to a simple example if
> >>> i got some spare time.
>
> >>> till then, thanks for the answer
>
> >> FWIW I ran into a problem with some tests failing yesterday after
> >> upgrading as well. I'll be working on them in the next couple hours and
> >> will post if I find anything. I don't have the code here but IIRC the tests
> >> were creating and storing objects using the .create() model method and
> >> verifying that post_save signaled functions were being called. I don't
> >> think any celery tasks were even involved... I was going to see if the
> >> change in test runner had something to do with it.
>
> > I've spent a little more time on this and here's what I found.
>
> > I did pip install --upgrade django-celery in my virtualenv (and
> > consequently automatically upgraded celery and and related packages) and
> > got django-celery 2.5.1/celery 2.5.1
> > Tests invoking tasks (via .delay()) were failing (irregardless of CELERY_ALWAYS_EAGER
> > settings) the behavior was querysets for model instances created prior to
> > invoking the task always returned the empty set...
> > If if reverted only celery: pip install celery==2.4.6 all tests passed.
>
> > Best Regards,
> > Steve
> >http://tech.agilitynerd.com/

--

-- 
You received this message because you are subscribed to the Google Groups "celery-users" group.
To post to this group, send email to celery-users@...
To unsubscribe from this group, send email to celery-users+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

Asynchronous & Results Status for Progress Bar?

celeryctl filter tasks

Re: Asynchronous & Results Status for Progress Bar?