Andrew Pashkin | 23 Nov 17:16 2015
Picon
Gravatar

Interface of Inspect.active/.scheduled/.reserved

Hello, everybody!

I'm experiencing an issue, that Inspect.active() returns None sometimes, which I hadn't expected, b/c it's not covered in documentation. It's also hard to comprehend from the source, what it should return.

I also have a doubt about return value from the example:
>>> i.active()
[{'worker1.example.com':
    [{'name': 'tasks.sleeptask',
      'id': '32666e9b-809c-41fa-8e93-5ae0c80afbbf',
      'args': '(8,)',
      'kwargs': '{}'}]}]
Why it's a list? Can dicts, contained in the list have more then one key?

Can somebody clarify interface for these functions?
-- With kind regards, Andrew Pashkin. cell phone - +7 (985) 898 57 59 Skype - waves_in_fluids e-mail - andrew.pashkin-puGfsi27rH1aa/9Udqfwiw@public.gmane.org

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

sma | 22 Nov 04:42 2015

context switches / cores / optimal works

I've got about 16 cores on my server, I'm creating 14 worker processes with 1 level of concurrency as they're long tasks and I want to distribute them to all works equally.  I've seen that celery creates a parent process and then a child process actually does the work.  I'm guessing this parent process is responsible for fetching from the broker.  My question is it going to cause a lot of context switches between parents/child if i create alot of worker/processes matching my cores because the parents will keep checking the broker?

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Adi Krishnan | 16 Nov 14:22 2015
Picon
Gravatar

Possible incorrect Celery implementation - Need help with review and feedback.

I'm using Celery for some tasks but it doesn't look like the perfect solution I have implemented. Here is what I am trying to do -

1. I need to have a job like concept which abstracts over all individual tasks. So, task completion and job completion are two separate concepts.
2. I need to be able to fetch the individual task states and results.

For the ones who need more specifics, here is a gist of what I am trying to achieve -

1. A job is triggered by a user or periodically. Each job can be broken down as: Job = Task 1 + Task 2 + .... Task n.
2. User can upload multiple jobs.
3. User can, if necessary, check the completion rate of each job (much on the lines of Jobtastic).

Noteworthy points:

1. I cannot use "group" and "chain" constructs because there are tasks which interacts with an external system which has indefinite execution times. I haven't tested the code but I am sure that there'll be an easier way to go about this.
2. I have to manually keep track of each task result because a few tasks break down the input to achieve parallelism. [These are the same tasks I described in the point above].
3. I have not come across concept of job or job completion in Celery.

I was thinking of writing a wrapper class called Job and implementing Celery tasks as part of that but I'm guessing I will have to handle things myself too. 

Any thoughts on improving this?

As an aside, if someone is aware of some other tool which I could use here, that would be really helpful.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Andy Billy | 15 Nov 06:45 2015
Picon

Does task.apply_async return immidiately , despite networking is blocked?

I am confued by the implementation for async task of celery.

 Does task.apply_async return immidiately , despite networking is blocked?

Thanks.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
andrew | 12 Nov 13:29 2015

Celery events does not disable

Hello! I'm facing a problem. I have about 180 celery workers and my rabbitmq server feels high load from celeryev, it works fine at the moment and the node's average load is 4.0-4.5. But I'm planning to increase the count of workers, let's say to 500. And I guess when it happens my rabbitmq server will go down. I've looked into rabbitmq management console and found out that I have 15k messages/second which, I guess, are coming from celeryev.* queues. So I went forward and added to celery config `CELERY_SEND_EVENTS=False` and reconfigured all my workers to run with `--without-heartbeat --without-mingle`. Then I restarted the whole system and was expecting celeryev.* queues to disappear, but nothing changed, can you give me an advice what to do in this situation? Thank you in advance! 

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Mark Rogers | 11 Nov 23:46 2015
Picon

Celery tasks sometimes mysteriously vanishing

Hey all,

I have a bit of an interesting scenario, and I thought I'd reach out to everyone here to see if anyone's seen something similar or has any ideas. We've used Celery at my company for 4-5 years now in conjunction with Django. When this codebase started off, we were using Django 1.0, so it's grown "organically" let's call it :). We're currently on Django 1.6 and celery 3.1.7, but I'm in the process of an upgrade to Django 1.8 and Celery 3.1.18. 

On the old versions, everything works fine and has for well over a year without any issue. After I've upgraded I've ran into some snags with some tasks just dropping off the face of the Earth, and I don't have many ideas on how to track the problem down. Our setup is a bit funky, and I feel like it almost certainly plays into the issue, but I can't quite sort out where it's going wrong.

We've built a wrapper of sorts around the Task calling, so that we can fire tasks a little bit easier after a database transaction is committed. Effectively, if we're in a transaction, we keep a list of of tasks that need to be fired off, and when we commit, we iterate over the list firing off the requested tasks with their given arguments. We have a number of places in our code base where we need to do this, calling the same tasks, so making a wrapper made sense, I think. A Django signal is used to trigger the tasks to be executed. 

In our setup we're behind a load balancer, in the exact same load balancer configuration as it was in pre-upgrade. RabbitMQ sits behind a simple load balancer, and Celery on the 2 separate servers talks to Rabbit via that load balanced IP. The 2 celery nodes also have Apache/mod_wsgi running our Django stack as well. I have Selenium test setup, and I run it through about 80 "checkouts" through our app and about ~20 of those fail after the upgrade, 0 before. Tracking in the logs, I can see our wrapper calling the delay method of a particular task, but that's the last I see of it. Logging statement right before the call dumps out all the info I expect to see, the next line executes, RabbitMQ never sees it, and the log statement at the beginning of the task itself is never executed. I also verified that there isn't any form of exception happening on the call either, and nothing of note in the Celery logs themselves.

Finally, the task this seems to always happen on is a task that's fired by a different task. Task A acts as a router, it executes, and uses our wrapper to call Task B's delay method (not in a transaction at this point so it's passed on through, no hold or anything special). 

Any thoughts on what I could have missed in the upgrade that could have caused something like this? Best I can tell the task is never called in any way, as I don't see anything useful in tcp dumps so it doesn't look like a communication problem with Rabbit. I'm hesitant to believe in a communication issue, simply because I can revert these upgrades at get back to 100% execution on my tests. Rabbit's working just fine, I feel like it has to be something else. That said, I've been proven wrong before, and I'm willing to be proven wrong again :)

Thanks,
Mark

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Dave Wood | 10 Nov 16:00 2015
Picon

How to control a chord's callback polling frequency

Hi All,

I've been using Canvas to control some distributed computing workflows. After a bit of a learning curve, it's working well. It's a neat tool!

I would like to change the polling frequency for chord callbacks, essentially the interval for the celery.chord_unlock task. By default the retry frequency is set to 1s and as a consequence, tens of thousands of tasks are created and retried before the chord can be unlocked.

My chords are a bit complex and feature nested chains of tasks. See below for an example structure of one.

workflow = chord(
(
map_A.s(job) |
map_B.s() |
map_C.s()
) for item in my_list
)(
reduce_A.s() |
reduce_B.s(job) |
reduce_C.s()
)

I have tried adding an 'interval=X' keyword argument to the callback part of the chord, but that seemed to break the workflow: the callback subtasks were called before the group subtasks were complete.

Please could someone advise me how I can change the callback polling frequency of this chord?
Is it possible to change the default polling frequency?

Many thanks

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
sma | 7 Nov 19:39 2015

delay() / apply_async() not returning right away

I have significant latency when sending a task, it's about 30sec per task.  I can't figure out why it's taking so long to put it on the broker.  I have plenty of workers, anyway to debug this?

I've tried everything, please help!


from celery import Celery
from kombu import Exchange, Queue

# RabbitMQ
BROKER_URL = 'amqp://something'
BROKER_POOL_LIMIT = 50

# Redis
CELERY_RESULT_BACKEND = 'redis://something'
CELERY_RESULT_PERSISTENT = True

CELERY_DISABLE_RATE_LIMITS = True

# Pool
CELERYD_CONCURRENCY = 3
CELERYD_POOL_RESTARTS = True

CELERYD_PREFETCH_MULTIPLIER = 1

CELERY_QUEUES = (
Queue('celery', routing_key='celery', delivery_mode=1),
Queue('transient', routing_key='transient', delivery_mode=1),
)

CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml', 'application/x-python-serialize']
CELERY_MESSAGE_COMPRESSION = 'zlib'


CELERY_IMPORTS = (
'task'
)

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Zhihui Luo | 5 Nov 03:58 2015
Picon

celery logging to rsyslog

from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
handler = logging.handlers.SysLogHandler(address=('127.0.0.1', 514))
formatt = “[%(asctime)s:%(levelname)s/%(processName)s] %(message)s"
handler.setFormatter(logging.Formatter(formatt))
handler.setLevel(logging.INFO)
logger.addHandler(handler)
redirect_stdouts_to_logger(logger)

print "Hello ......................" #1
logger.error("OCCCCCCCCCC Error") #2

<at> app.task(bind=True)
def delete(self, filename):
    logger.error("22222<><><><><><><><><><><>") # 3
    print logger.handlers #4


rsyslog received #1 #2, but can't #3 #4  why?

how to logging to rsyslog?

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Mike Lowe | 4 Nov 21:08 2015
Picon

Redis loosing connection when used with Celery groups or chain

Please if anyone have the time please reply to my SO post, I have simulated my problem with using chord or groups which throws connection lost over and over again.

Here is my SO post:  http://stackoverflow.com/questions/33530781/redis-loosing-connection-when-used-with-celery-groups-or-chain-throwing-error 


Basically when i have up to 1000 or more task as group or chord i noticed my redis connection get lost. At first i thought it was my code so i modified to basic add.s() task and the problem persists.

Thanks in advance for your help.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Radha Dharmaraajan | 4 Nov 19:28 2015
Picon

How to Configure celeryd script to run celery works for different django projects

I have 2 different django projects (not connected to one another) in same server. 
Lets call them Project A and Project B
Project A was already using celery. I already see /etc/init.d/celeryd script in the server. 
CELERY_BIN - it is staring celery using manage.py found under Project A
and
CELERYD_CHDIR - is pointing to the path of Project A

Now, I have a new project, Project B, which also needs to use Celery.

How to change the celeryd script, in a way the workers will know which task they have to pickup. (project A workers to pick up tasks related to project A and same for Project B)

I did not find any documentation or examples for above case.

Any help is much appreciated. 

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.

Gmane