Robert Stewart | 14 Aug 00:10 2014
Picon

System hangs on tasks to non-existent workers.

Error Condition:
    If tasks are sent to a non-existent worker, the entire system
    seems to stop accepting messages until a corresponding worker is
    created to handle the messages.

Guess:
    Is rabbitmq applying "Per-Connection Flow Control" on single
    celery feeder queue, effectively blocking all celery tasks?
    (http://www.rabbitmq.com/memory.html)

Setup:
    Each worker is unique. Each worker creates a separate queue.
    Clients request work depending on the worker.

Questions:
    - What are possible causes of celery freezing the system due to a
      single worker going offline?
    - What are the tools to diagnose this kind of issue? How should they
      be applied?
    - Is there a convenient diagram showing celery's architecture when
      using rabbitmq?
    - If rabbitmq *is* applying "per connection flow control", what
      steps should be taken to prevent a single worker freezing the
      system?

Versions:
    Celery (3.1) routes tasks from clients to workers.
    Rabbitmq (2.7) is the message broker.
    Ubuntu 12.04 is the underlying OS.

(Continue reading)

Ask Holme | 12 Aug 09:28 2014
Picon

Support for publisher confirms

Hi list!

I'v been checking out wherever celery support the rabbitmq publisher confirms - however I cannot find any ressources around that topic.

It's clear that some support has been implemented in py-aqmp but it doesn't seems like it has moved up into kombu and celery?

Has anyone tried to make celery use publisher confirms when sending messages? 

Ps. I am aware that publisher confirms will hit performance and also that I will need some kind of async setup to avoid it being too "hard" on the system. So i'm not looking for advice on publisher confirms or not, but rather if i can use the feature via celery

Best regards
Ask

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Leonard Bedner | 12 Aug 04:52 2014
Picon

Using The "memory" Backend Doesn't Return Results To "group_result.join()"

Celery Version = 3.0.23

-------------------------------------------------------------

Hello all.

I have a batch job that downloads a file that contains JSON on each line. I take this file, iterate over it line by line, and pass each bit of JSON to celery. Here is the pertinent code:

    #iterate over each line in json file, pass to celery, then wait until all items have been completed
    logger
.log("Queueing up tasks...")
    file
= open(json_file)
    group_result
= group(parse_and_persist.s(line) for line in file)()
    file
.close()
    logger
.log("Waiting for tasks to finish being processed...")
    group_result
.join()

Now, this works fine when using RabbitMQ as the backend. However, as I'm sure everyone is aware, Rabbit will create a separate queue for each result, and when I ran a full job that had over 470K items, things didn't go so well. So I've been spending the day looking for another backend.

The key thing to realize is that I actually don't care about the state of the results at all. I just need to know they've all been processed before sending out an email to the appropriate people when the job finishes.

At any rate, I started looking into memcached, and was able to get that up running just fine, using the following config:

BROKER_URL            = 'amqp://guest:guest <at> localhost:5672//'
CELERY_RESULT_BACKEND
= 'cache'
CELERY_CACHE_BACKEND  
= 'memcached://127.0.0.1:11211/'

However, when reading the "Cache Backend Settings" section of the documentation, I noticed a bit at the end that said I could use "memory" as the cache backend. To me, that sounded perfect, because as stated above, I just need to know when the all the results have returned. In fact, once the job is done, the celery workers are shut down, so I wouldn't even have to worry about losing the results in memory.

Here is my config file:

BROKER_URL            = 'amqp://guest:guest <at> localhost:5672//'
CELERY_RESULT_BACKEND
= 'cache'
CELERY_CACHE_BACKEND  
= 'memory'

Once I made this changed, I fired up the workers, and started the job. I monitored RabbitMQ and noticed that all the tasks were successfully processed by the celery workers, however, they never returned the results, so the job just hung on the "group_results.join()" line.

Is there anything I am missing here? Am I not understanding how this is supposed to work? My assumption here is that the "memory" cache backend is something celery handles internally, not something like memcached.

Thanks.

Regards,

Leonard Bedner

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
asili22 | 11 Aug 21:27 2014
Picon

Having troubles with Django and concurrent tasks in Celery

Hi There,

I have a complicated scenario I need to tackle.

I'm using Celery to run tasks in parallel, my tasks involve with HTTP requests and I'm planning to use Celery along with eventlet for such purpose.

Let me explain my scenario:

I have 2 tasks that can run in parallel and third task that needs to work on the output of those 2 tasks therefore I'm using Celery group to run the 2 tasks and Celery chain to pass the

output to the third task to work on it when they finish.

Now it gets complicated, the third task needs to spawn multiple tasks that I would like to run in parallel and I would like to collect all outputs together and to process it in another task.

So I created a group for the multiple tasks together with a chain to process all information.


I guess I'm missing basic information about Celery concurrent primitives, I was having a 1 celery task that work well but I needed to make it faster.


This is a simplified sample of the code:

<at> app.task

def task2():

       return "aaaa"


<at> app.task

def task3():

       return "bbbb"


<at> app.task

def task4():

    work = group(...) | task5.s(...)                                                          

    work()


<at> app.task

def task1(weight, length, width, height, value, to_country, is_envelope=True, postcode=None):

   

        tasks = [

            task2.s(a, b),

            task3.s(c, d)

        ]

    work = group(tasks) | task4.s()

    return work()


This is how I start this operation:

            task = tasks1.apply_async(

                kwargs=kwargs,

                queue='queue1')


I save task.id and pull the server every 30 seconds to see if results available by doing:

results = tasks1.AsyncResult(task_id)

if results.ready():

                res = results.get()

.....



Please help me putting the pieces back together.


10x

 

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Dhana Raghu | 11 Aug 12:18 2014
Picon

Logging Celery3.1.13 logs to Syslog

Hi,
     Can somebody give me a sample code for writing Celery logs to Syslogs.

Regards
Dhana

Joseph Metzinger | 10 Aug 02:53 2014

Expiration Times for Chrod Callbacks

Hey guys,

I have a question about using celery chords in conjunction with task expiration. Basically, I want to know what happens for the callback task of the chord if I set expires=10 for the task. Does the clock for the callback start as soon as the chord is added to the queue, or does the 10 second clock start once the header tasks have all completed?

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
tommaso barbugli | 9 Aug 15:48 2014
Picon

contrib.batches and celery eager

Hi,
I am experimenting with the batches contib module; I rewrite one task to run as in the example from docs.
It seems that when running the code with celery always eager, the task function doesn't get a list of parameters (as there would be a batch of 1 task to flush) but instead it gets the plain signature.
Are there any clean workarounds for this?
Since I am here, do I need to mark tasks as completed at the end of the batch processing even if I don't collect results?

Thanks,
Tommaso

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Paul Choi | 8 Aug 22:45 2014
Picon

Chord of chords reporting SUCCESS when a subtask threw an exception

This is a contrived example, but it reflects the workflow that we are trying to achieve in our actual code.

We're running Celery 3.1.12, with RabbitMQ.


I have code as such. Task cook_and_eat is a chord, which calls 2 other subtasks - cook, eat. They each are a subtask that return the AsyncResult of a chord.


import celeryconfig

from celery import Celery, chord, group, chain



app = Celery('tasks')

app.config_from_object(celeryconfig)


<at> app.task

def cook_and_eat(food):

   return chord(

       cook.si(food),

       eat.si(food)

   ).apply_async()


######################### cooking

<at> app.task

def chop_with_knife(food):

   print "Chopping {} with knife".format(food)

   return True



<at> app.task

def throw_into_pan(food):

   print "Throwing {} into pan".format(food)

   return True



<at> app.task

def heat(food):

   print "heat {}".format(food)

   return True



<at> app.task

def plate(food):

   print "plate {}".format(food)

   return True



<at> app.task

def cook(food):

   return chord(

       chain(

           chop_with_knife.s(food),

           throw_into_pan.si(food),

           heat.si(food)

       ),

       plate.si(food)

   ).apply_async()



######################### eating


<at> app.task

def bite(food):

   if food == 'salad':

       raise Exception('Throwing up. I hate {}'.format(food))

   print "bite {}...".format(food)

   return True



<at> app.task

def chew(food):

   print "chewing {}...".format(food)

   return True



<at> app.task

def mix_with_saliva(food):

   print "mixing {} with saliva...".format(food)

   return True



<at> app.task

def swallow(food):

   print "swallowing {}...".format(food)

   return True


<at> app.task

def eat(food):

   return chord(

       chain(

           bite.si(food),

            chew.si(food),

            mix_with_saliva.si(food)

       ),

       swallow.si(food)).apply_async()



When I pass in a value - steak - everything succeeds as expected.


>>> import foo
>>> res = foo.cook_and_eat.delay('steak')

>>> res

<AsyncResult: 8d007ec5-9caa-4e67-8710-a9b5471f9cfc>

>>> res.get()

[[u'53e6b4ac-09eb-4f22-b656-6af7125c6198', [u'6be070e5-25eb-4d53-9c54-dc1f161b4d75', [[[u'461b5f45-5575-4adf-bb51-b8c08efcf2fa', None], None]]]], None]

>>> res.status

u'SUCCESS'



If I pass in a value that generates an exception by bite subtask - the value is salad, bite will raise an exception as expected. And the chord stops as expected. But cook_and_eat returns SUCCESS, which is incorrect. Am I missing something here?


>>> import foo
>>> res = foo.cook_and_eat.delay('salad')

>>> res.status

u'SUCCESS'

>>> res.ready()

True

>>>



And the celery worker output looks as such (for both steak and salad):

$ celery -A foo worker


 -------------- celery-B0OyYtC1EkCdBw3G0RLmbYDVIPR50AEW@public.gmane.orgl v3.1.12 (Cipater)

---- **** -----

--- * ***  * -- Darwin-13.3.0-x86_64-i386-64bit

-- * - **** ---

- ** ---------- [config]

- ** ---------- .> app:         tasks:0x104f3b290

- ** ---------- .> transport:   amqp://guest:** <at> localhost:5672//

- ** ---------- .> results:     amqp

- *** --- * --- .> concurrency: 24 (prefork)

-- ******* ----

--- ***** ----- [queues]

 -------------- .> celery           exchange=celery(direct) key=celery



[2014-08-08 13:10:54,651: WARNING/MainProcess] celery-B0OyYtC1EkCdBw3G0RLmbRFnWt+6NQIA@public.gmane.org ready.

[2014-08-08 13:11:13,777: WARNING/Worker-15] Chopping steak with knife

[2014-08-08 13:11:13,779: WARNING/Worker-4] Throwing steak into pan

[2014-08-08 13:11:13,782: WARNING/Worker-19] heat steak

[2014-08-08 13:11:15,844: WARNING/Worker-10] bite steak...

[2014-08-08 13:11:15,844: WARNING/Worker-20] plate steak

[2014-08-08 13:11:15,847: WARNING/Worker-11] chewing steak...

[2014-08-08 13:11:15,865: WARNING/Worker-18] mixing steak with saliva...

[2014-08-08 13:11:17,915: WARNING/Worker-15] swallowing steak...

[2014-08-08 13:11:34,394: WARNING/Worker-3] Chopping salad with knife

[2014-08-08 13:11:34,396: WARNING/Worker-1] Throwing salad into pan

[2014-08-08 13:11:34,399: WARNING/Worker-17] heat salad

[2014-08-08 13:11:36,461: WARNING/Worker-16] plate salad

[2014-08-08 13:11:36,495: ERROR/MainProcess] Task foo.bite[2684cd88-aab2-4670-ac7d-46ebff3cdca1] raised unexpected: Exception('Throwing up. I hate salad',)

Traceback (most recent call last):

  File "/Users/paulchoi/env/isd-python/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task

    R = retval = fun(*args, **kwargs)

  File "/Users/paulchoi/env/isd-python/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__

    return self.run(*args, **kwargs)

  File "/Users/paulchoi/celerytest/foo.py", line 71, in bite

    raise Exception('Throwing up. I hate {}'.format(food))

Exception: Throwing up. I hate salad




I do have CELERY_CHORD_PROPAGATES=True. This is my celeryconfig.py:

$ cat celeryconfig.py

# config file for Celery Daemon


# default RabbitMQ broker

BROKER_URL = 'amqp://'


# default RabbitMQ backend

CELERYD_CONCURRENCY=24

CELERYD_HIJACK_ROOT_LOGGER=False

CELERY_RESULT_BACKEND='amqp'

CELERY_TASK_SERIALIZER='json'

CELERY_RESULT_SERIALIZER='json'

CELERY_SEND_EVENTS=True

CELERY_ACCEPT_CONTENT=['json']

CELERY_CHORD_PROPAGATES=True



Thanks for your help.

(And if it happens to be a repost, my apologies. I tried posting earlier, but I didn't see my post).

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
mingle | 7 Aug 20:54 2014

Confusing Celery concurrency issues

Hi,

I have one worker created like so:

celery worker --app=cleopatra -l INFO --pidfile=/var/run/proj/cleopatra-celery.pid -Q celery -c 4

I'm using redis for results and transport.

My assumptions:

1. This should create 5 processes.  One main and 4 workers.
2. This should consume and work on 4 tasks at a time (max)

It does #1 but not always #2.

I have a task called barbaz that uses time.sleep to sleep for 30 seconds.

If I run barbaz.delay manually from a console this is what happens:

barbaz.delay(0)
<AsyncResult: f181ee08-1cd0-407b-907a-8ffead457c41>

barbaz.delay(1)
<AsyncResult: 891c14af-b842-4667-9e6c-c3df2caefd20>

barbaz.delay(2)
<AsyncResult: ca7a3e5c-5835-42eb-b418-630278967c28>

barbaz.delay(3)
<AsyncResult: 43fdb27d-1a1d-46d5-a729-2fec260a3cfe>

Task 0 [2014-08-07 18:26:10,892: INFO/MainProcess] Received task: cleopatra.jobs.foobar.barbaz[f181ee08-1cd0-407b-907a-8ffead457c41]
Task 0 [2014-08-07 18:26:10,894: INFO/Worker-1] Beginning task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 1 [2014-08-07 18:26:12,300: INFO/MainProcess] Received task: cleopatra.jobs.foobar.barbaz[891c14af-b842-4667-9e6c-c3df2caefd20]
Task 1 [2014-08-07 18:26:12,302: INFO/Worker-4] Beginning task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 2 [2014-08-07 18:26:13,660: INFO/MainProcess] Received task: cleopatra.jobs.foobar.barbaz[ca7a3e5c-5835-42eb-b418-630278967c28]
Task 3 [2014-08-07 18:26:15,099: INFO/MainProcess] Received task: cleopatra.jobs.foobar.barbaz[43fdb27d-1a1d-46d5-a729-2fec260a3cfe]
Task 0 [2014-08-07 18:26:40,925: INFO/Worker-1] Successful task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 2 [2014-08-07 18:26:40,925: INFO/Worker-1] Beginning task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 0 [2014-08-07 18:26:40,926: INFO/MainProcess] Task cleopatra.jobs.foobar.barbaz[f181ee08-1cd0-407b-907a-8ffead457c41] succeeded in 30.0318708755s
Task 1 [2014-08-07 18:26:42,333: INFO/Worker-4] Successful task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 3 [2014-08-07 18:26:42,333: INFO/Worker-4] Beginning task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 1 [2014-08-07 18:26:42,334: INFO/MainProcess] Task cleopatra.jobs.foobar.barbaz[891c14af-b842-4667-9e6c-c3df2caefd20] succeeded in 30.0311090592s
Task 2 [2014-08-07 18:27:10,956: INFO/Worker-1] Successful task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 2 [2014-08-07 18:27:10,957: INFO/MainProcess] Task cleopatra.jobs.foobar.barbaz[ca7a3e5c-5835-42eb-b418-630278967c28] succeeded in 30.0318741249s
Task 3 [2014-08-07 18:27:12,364: INFO/Worker-4] Successful task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 3[2014-08-07 18:27:12,365: INFO/MainProcess] Task cleopatra.jobs.foobar.barbaz[43fdb27d-1a1d-46d5-a729-2fec260a3cfe] succeeded in 30.0314799445s

It seems only Worker-1 and Worker-4 start working on the tasks even though 4 tasks were received.  Why didn't Worker-2 and Worker-3 work on them?  This is the same behavior if I repeat this over and over.

If I run barbaz.delay in a loop from the console this is what happens:

[barbaz.delay(x) for x in xrange(4)]
[<AsyncResult: 9d678be0-ef3b-4f02-aefa-672a3709e2cb>, <AsyncResult: 20444d92-dae8-4235-82a6-411eb680dd93>, <AsyncResult: 28891609-2e0c-437c-805f-b6ee6b17e71b>, <AsyncResult: a7671df9-ed11-4fcb-85dd-d56f86efcb7e>]

Task 0 [2014-08-07 18:35:34,997: INFO/MainProcess] Received task: cleopatra.jobs.foobar.barbaz[9d678be0-ef3b-4f02-aefa-672a3709e2cb]
Task 1 [2014-08-07 18:35:34,999: INFO/MainProcess] Received task: cleopatra.jobs.foobar.barbaz[20444d92-dae8-4235-82a6-411eb680dd93]
Task 0 [2014-08-07 18:35:35,001: INFO/Worker-1] Beginning task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 1 [2014-08-07 18:35:35,003: INFO/Worker-2] Beginning task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 2 [2014-08-07 18:35:35,004: INFO/MainProcess] Received task: cleopatra.jobs.foobar.barbaz[28891609-2e0c-437c-805f-b6ee6b17e71b]
Task 3 [2014-08-07 18:35:35,007: INFO/MainProcess] Received task: cleopatra.jobs.foobar.barbaz[a7671df9-ed11-4fcb-85dd-d56f86efcb7e]
Task 2 [2014-08-07 18:35:35,008: INFO/Worker-3] Beginning task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 0 [2014-08-07 18:36:05,031: INFO/Worker-1] Successful task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 3 [2014-08-07 18:36:05,031: INFO/Worker-1] Beginning task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 0 [2014-08-07 18:36:05,032: INFO/MainProcess] Task cleopatra.jobs.foobar.barbaz[9d678be0-ef3b-4f02-aefa-672a3709e2cb] succeeded in 30.0304881223s
Task 1 [2014-08-07 18:36:05,032: INFO/Worker-2] Successful task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 1 [2014-08-07 18:36:05,034: INFO/MainProcess] Task cleopatra.jobs.foobar.barbaz[20444d92-dae8-4235-82a6-411eb680dd93] succeeded in 30.0309955431s
Task 2 [2014-08-07 18:36:05,039: INFO/Worker-3] Successful task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 2 [2014-08-07 18:36:05,040: INFO/MainProcess] Task cleopatra.jobs.foobar.barbaz[28891609-2e0c-437c-805f-b6ee6b17e71b] succeeded in 30.031243939s
Task 3 [2014-08-07 18:36:35,062: INFO/Worker-1] Successful task < <at> task: cleopatra.jobs.foobar.barbaz of cleopatra:0x24fde50> execution
Task 3 [2014-08-07 18:36:35,065: INFO/MainProcess] Task cleopatra.jobs.foobar.barbaz[a7671df9-ed11-4fcb-85dd-d56f86efcb7e] succeeded in 30.0330389757s

It seems Worker-1, Worker-2, and Worker-3 worked on the tasks.  Sometimes all four workers work on them but never less than 3.

Here are my settings:

{'CELERY_RESULT_DB_TABLENAMES': None, 'BROKER_TRANSPORT_OPTIONS': {}, 'BROKER_USE_SSL': False, 'CELERY_BROADCAST_QUEUE': 'celeryctl', 'EMAIL_USE_TLS': False, 'CELERY_STORE_ERRORS_EVEN_IF_IGNORED': False, 'CELERY_CREATE_MISSING_QUEUES': True, 'CELERY_DEFAULT_QUEUE': 'celery', 'CELERY_SEND_TASK_SENT_EVENT': False, 'CELERYD_TASK_TIME_LIMIT': None, 'BROKER_URL': 'redis://127.0.0.1:12035/0', 'CELERY_EVENT_QUEUE_EXPIRES': None, 'CELERY_DEFAULT_EXCHANGE_TYPE': 'direct', 'CELERYBEAT_SCHEDULER': 'celery.beat:PersistentScheduler', 'CELERY_MAX_CACHED_RESULTS': 100, 'CELERY_RESULT_PERSISTENT': None, 'CELERYD_POOL': 'prefork', 'CELERYD_AGENT': None, 'EMAIL_HOST': 'localhost', 'CELERY_CACHE_BACKEND_OPTIONS': {}, 'BROKER_HEARTBEAT': None, 'CELERY_RESULT_ENGINE_OPTIONS': None, 'CELERY_RESULT_SERIALIZER': 'pickle', 'CELERYBEAT_SCHEDULE_FILENAME': 'celerybeat-schedule', 'CELERY_REDIRECT_STDOUTS_LEVEL': 'WARNING', 'CELERY_IMPORTS': ['cleopatra.lib.delivery.email', 'cleopatra.lib.delivery.sms', 'cleopatra.lib.delivery.voice', 'cleopatra.lib.twilio_lib', 'cleopatra.views.tests', 'cleopatra.lib.email_utils', 'cleopatra.jobs.cleanup', 'cleopatra.jobs.index_city_mass_text_strings', 'cleopatra.jobs.run_delayed_jobs', 'cleopatra.jobs.sender_ids', 'cleopatra.jobs.foobar'], 'SERVER_EMAIL': 'celery <at> localhost', 'CELERYD_TASK_LOG_FORMAT': '[%(asctime)s: %(levelname)s/%(processName)s] %(task_name)s[%(task_id)s]: %(message)s', 'CELERY_SECURITY_CERTIFICATE': None, 'CELERYD_LOG_COLOR': None, 'CELERY_RESULT_EXCHANGE': 'celeryresults', 'CELERY_TRACK_STARTED': False, 'CELERY_REDIS_PASSWORD': None, 'BROKER_USER': None, 'CELERY_COUCHBASE_BACKEND_SETTINGS': None, 'CELERY_RESULT_EXCHANGE_TYPE': 'direct', 'CELERY_REDIS_DB': None, 'CELERYD_TIMER_PRECISION': 1.0, 'CELERY_REDIS_PORT': None, 'BROKER_TRANSPORT': None, 'CELERYMON_LOG_FILE': None, 'CELERYD_CONCURRENCY': 0, 'CELERYD_HIJACK_ROOT_LOGGER': True, 'BROKER_VHOST': None, 'CELERY_DEFAULT_EXCHANGE': 'celery', 'CELERY_DEFAULT_ROUTING_KEY': 'celery', 'CELERY_ALWAYS_EAGER': False, 'EMAIL_TIMEOUT': 2, 'CELERYD_TASK_SOFT_TIME_LIMIT': None, 'CELERY_WORKER_DIRECT': False, 'CELERY_REDIS_HOST': None, 'CELERY_QUEUE_HA_POLICY': None, 'BROKER_PORT': None, 'CELERYD_AUTORELOADER': 'celery.worker.autoreload:Autoreloader', 'BROKER_CONNECTION_TIMEOUT': 4, 'CELERY_ENABLE_REMOTE_CONTROL': True, 'CELERY_RESULT_DB_SHORT_LIVED_SESSIONS': False, 'CELERY_EVENT_SERIALIZER': 'json', 'CASSANDRA_DETAILED_MODE': False, 'CELERY_REDIS_MAX_CONNECTIONS': None, 'CELERY_CACHE_BACKEND': None, 'CELERYD_PREFETCH_MULTIPLIER': 1, 'BROKER_PASSWORD': None, 'CELERY_BROADCAST_EXCHANGE_TYPE': 'fanout', 'CELERY_EAGER_PROPAGATES_EXCEPTIONS': False, 'CELERY_IGNORE_RESULT': False, 'CASSANDRA_KEYSPACE': None, 'EMAIL_HOST_PASSWORD': None, 'CELERYMON_LOG_LEVEL': 'INFO', 'CELERY_DISABLE_RATE_LIMITS': False, 'CELERY_TASK_PUBLISH_RETRY_POLICY': {'interval_start': 0, 'interval_max': 1, 'max_retries': 3, 'interval_step': 0.2}, 'CELERY_SECURITY_KEY': None, 'CELERY_MONGODB_BACKEND_SETTINGS': None, 'CELERY_DEFAULT_RATE_LIMIT': None, 'CELERYBEAT_SYNC_EVERY': 0, 'CELERY_EVENT_QUEUE_TTL': None, 'CELERYD_POOL_PUTLOCKS': True, 'CELERY_TASK_SERIALIZER': 'pickle', 'CELERYD_WORKER_LOST_WAIT': 10.0, 'CASSANDRA_SERVERS': None, 'CELERYD_POOL_RESTARTS': False, 'CELERY_TASK_PUBLISH_RETRY': True, 'CELERY_ENABLE_UTC': True, 'CELERY_SEND_EVENTS': False, 'BROKER_CONNECTION_MAX_RETRIES': 100, 'CELERYD_LOG_FILE': None, 'CELERYD_FORCE_EXECV': False, 'CELERY_CHORD_PROPAGATES': True, 'CELERYD_AUTOSCALER': 'celery.worker.autoscale:Autoscaler', 'CELERYD_STATE_DB': None, 'CELERY_ROUTES': {'cleopatra.jobs.cleanup.clean_all': {'queue': 'low'}, 'cleopatra.jobs.cleanup.delete_job': {'queue': 'low'}}, 'CELERYD_TIMER': None, 'ADMINS': (), 'BROKER_HEARTBEAT_CHECKRATE': 3.0, 'CELERY_ACCEPT_CONTENT': ['pickle', 'json', 'msgpack', 'yaml'], 'BROKER_LOGIN_METHOD': None, 'BROKER_CONNECTION_RETRY': True, 'CELERY_TIMEZONE': None, 'CASSANDRA_WRITE_CONSISTENCY': None, 'CELERYBEAT_MAX_LOOP_INTERVAL': 0, 'CELERYD_LOG_LEVEL': 'WARN', 'CELERY_REDIRECT_STDOUTS': True, 'BROKER_POOL_LIMIT': 10, 'CELERY_SECURITY_CERT_STORE': None, 'CELERYD_CONSUMER': 'celery.worker.consumer:Consumer', 'CELERY_INCLUDE': (), 'CELERYD_MAX_TASKS_PER_CHILD': None, 'CELERYD_LOG_FORMAT': '[%(asctime)s: %(levelname)s/%(processName)s] %(message)s', 'CELERY_ANNOTATIONS': None, 'CELERY_MESSAGE_COMPRESSION': None, 'CASSANDRA_READ_CONSISTENCY': None, 'EMAIL_USE_SSL': False, 'CELERY_SEND_TASK_ERROR_EMAILS': False, 'CELERY_QUEUES': None, 'CELERY_ACKS_LATE': False, 'CELERYMON_LOG_FORMAT': '[%(asctime)s: %(levelname)s] %(message)s', 'CELERY_TASK_RESULT_EXPIRES': datetime.timedelta(1), 'BROKER_HOST': None, 'EMAIL_PORT': 25, 'BROKER_FAILOVER_STRATEGY': None, 'CELERY_RESULT_BACKEND': 'redis', 'CELERY_BROADCAST_EXCHANGE': 'celeryctl', 'CELERYBEAT_LOG_FILE': None, 'CELERYBEAT_SCHEDULE': {'provision_missing_sender_ids': {'task': 'cleopatra.jobs.sender_ids.provision_missing_sender_ids', 'schedule': <crontab: 0 */12 * * * (m/h/d/dM/MY)>}, 'cleanup-tasks': {'task': 'cleopatra.jobs.cleanup.clean_all', 'schedule': <crontab: */20 * * * * (m/h/d/dM/MY)>}, 'run-delayed-jobs-task': {'task': 'cleopatra.jobs.run_delayed_jobs.run_scheduled_jobs', 'schedule': <crontab: */1 * * * * (m/h/d/dM/MY)>}, 'cleanup-job-progress': {'task': 'cleopatra.jobs.cleanup.clean_job_progress', 'schedule': <crontab: */10 * * * * (m/h/d/dM/MY)>}, 'city-mass-text-task': {'task': 'cleopatra.jobs.index_city_mass_text_strings.index_mass_text_strings', 'schedule': <crontab: 0 */3 * * * (m/h/d/dM/MY)>}}, 'CELERY_RESULT_DBURI': 'redis://127.0.0.1:12035/0', 'CELERY_DEFAULT_DELIVERY_MODE': 2, 'CELERYBEAT_LOG_LEVEL': 'INFO', 'CASSANDRA_COLUMN_FAMILY': None, 'EMAIL_HOST_USER': None}

Any ideas what's up?  I can post more information if needed.  Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
sandeep G Kurdagi | 7 Aug 09:14 2014
Picon

Adding tasks to celery for Scheduling

Background: I am using celery for building a scheduling system to Crawl the websites on daily basis.We are crawling about 10 lakhs urls (approx) daily. So it's becoming dificult to handle and manage the things at micro level. Celery is one where we thought could handle the current system in much better way than what it is now.

Problem: Let's say i have 1000 urls for a domain. Currently, what i am thinking to do is like 1000 urls are equally divided into some n equal chunks and then for each chunk, create a task and schedule it using celery.To do this, am not able to create (register) the tasks dynamically. And also i need to ensure the politeness policy over here. How to create the tasks on the fly in celery. There is no documentation for the same.

Am i going in right direction in solving this. If any better approach. Please add your insights. Your suggestions will be highly appreciated.

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to celery-users+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To post to this group, send email to celery-users-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
Visit this group at http://groups.google.com/group/celery-users.
For more options, visit https://groups.google.com/d/optout.
Quân Nguyễn Hồng | 7 Aug 04:05 2014

List waiting periodic tasks

Hello everybody

How can I list waiting periodic tasks, similar with the the command 
"celeryctl inspect scheduled"?

Thanks


Gmane