headers
ui | 1 May 2010 21:41
Picon

Possible bug when inserting 179366 jobs and then deleting them

Summary:  Server returns internal err if we restart, put 179366 jobs,
then try to reserve+delete those same jobs.   This happens only with -
b

I hope what follows isn't too much info. Pls. yell if I can be of any
assistance.

Beanstalkd-1.4.4  with -b

If we stop  beanstalkd, wipe the binlog folder, restart beanstalkd
then run a script that adds 179366  jobs and then run another that
reserves and deletes all of those jobs, the last delete returns an
INTERNAL ERROR.

I see this: "/tmp/beanstalkd-1.4.4/beanstalkd: binlog.c:473 in
binlog_write_job: failed to use next binlog"   .   Prot.c:1302 returns
the err ( r == 0 )
pheanstalk throws an exception.

It takes at least that many jobs (on this linux machine) ; adding and
then deleting 179365 jobs doesnt seeem to do it.

The 179366th job does seem to get erased.

We invoke beanstalkd like this
beanstalkd -l 74.54.253.178 -p 11300  -b /tmp/beanstalkd.binlogs/

the PUSH script that adds the jobs to the test tube:

<?php
(Continue reading)

Permalink | Reply | 2 comments |
headers
Keith Rarick | 1 May 2010 22:18
Picon
Favicon
Gravatar

Re: Possible bug when inserting 179366 jobs and then deleting them

On Sat, May 1, 2010 at 12:41 PM, ui <uivatury@...> wrote:
> Summary:  Server returns internal err if we restart, put 179366 jobs,
> then try to reserve+delete those same jobs.   This happens only
> with -b

That is a bug. At first glance, it looks like something to do with our
code for allocating binlog space.

> I hope what follows isn't too much info. Pls. yell if I can be of any
> assistance.

It's excellent info! Thanks for being so thorough. Here's a ticket to
track the bug:

http://github.com/kr/beanstalkd/issues/issue/38

kr

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
Permalink | Reply | 1 comment |
headers
waratuman | 2 May 2010 00:32
Picon
Gravatar

Introspection

Got a few questions as far as looking into what the system is doing.
I've used Resque in the past and am in the process of reading up on
beanstalk and seeing if it is a viable replacement.

1. Is it possible to get a list of jobs that are in a tube?
2. What happens with failed jobs? (this may be client side)
3. Is it possible to get a list of queues that a worker is watching?
4. Is it possible to get the current job that a worker is working on?

Thanks,
 - James

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
Permalink | Reply | 3 comments |
headers
Kurt Seifried | 2 May 2010 03:14
Gravatar

Re: Introspection

> Got a few questions as far as looking into what the system is doing.
> I've used Resque in the past and am in the process of reading up on
> beanstalk and seeing if it is a viable replacement.
>
> 1. Is it possible to get a list of jobs that are in a tube?

You can reserve them to look at them and then let them be returned
(i.e. don't delete the job).

> 2. What happens with failed jobs? (this may be client side)
http://nubyonrails.com/articles/about-this-blog-beanstalk-messaging-queue
failed in what sense? A client reserves a job. a client does the work.
the work fails. The client could then delete the job and you handle
the failure in the program, or you could bury the job so it's in the
queue but not available until someone specifically pokes it.
Personally I prefer to handle failures in my applications and delete
the job since it's unlikely the job is going to succeed if run again.
If you're going to re-run jobs you also want to make sure that won't
mess up the system (you may need to back out of transactions/etc based
on what you are doing). If you're going to rerun jobs you should make
sure they are idempotent. http://en.wikipedia.org/wiki/Idempotence

> 3. Is it possible to get a list of queues that a worker is watching?

Yup, you ask the client.

> 4. Is it possible to get the current job that a worker is working on?

Well you ask the client and it says "I am working on job id X"
(Continue reading)

Permalink | Reply | 1 comment |
headers
James R. Bracy | 2 May 2010 10:27
Picon
Gravatar

Re: Introspection


>> Got a few questions as far as looking into what the system is doing.
>> I've used Resque in the past and am in the process of reading up on
>> beanstalk and seeing if it is a viable replacement.
>> 
>> 1. Is it possible to get a list of jobs that are in a tube?
> 
> You can reserve them to look at them and then let them be returned
> (i.e. don't delete the job).

What if there are thousands of jobs in the queue? It doesn't seem right to go reserving a thousand jobs just to
put them back on the queue.

>> 2. What happens with failed jobs? (this may be client side)
> http://nubyonrails.com/articles/about-this-blog-beanstalk-messaging-queue
> failed in what sense? A client reserves a job. a client does the work.
> the work fails. The client could then delete the job and you handle
> the failure in the program, or you could bury the job so it's in the
> queue but not available until someone specifically pokes it.
> Personally I prefer to handle failures in my applications and delete
> the job since it's unlikely the job is going to succeed if run again.
> If you're going to re-run jobs you also want to make sure that won't
> mess up the system (you may need to back out of transactions/etc based
> on what you are doing). If you're going to rerun jobs you should make
> sure they are idempotent. http://en.wikipedia.org/wiki/Idempotence
> 

In my case I just need a back trace of what went wrong. Resque provides a great interface for this, an let's me
manually requeue the job.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Keith Rarick | 3 May 2010 00:24
Picon
Favicon
Gravatar

Re: Introspection

On Sat, May 1, 2010 at 3:32 PM, waratuman <waratuman@...> wrote:
> 1. Is it possible to get a list of jobs that are in a tube?
> 4. Is it possible to get the current job that a worker is working on?

Short answer: client-side logging already provides this information,
plus a whole lot more on top, so it's usually easiest and more helpful
just to use that.

In beanstalkd, to provide a fast system under high load, we've chosen
to avoid operations that can bog down. If there are millions of jobs
in the queue, you don't want other clients to have to wait while
beanstalkd performs an Θ(n log n) sort and copy.

Unfortunately, these features are hard to provide in a way that is
always fast and still useful. (Note that a worker may have arbitrarily
many current jobs.)

Fortunately, these features aren't really necessary in the first
place. In my experience, in production, introspection is superfluous
-- the application and workers should not require hand-holding. If
they do, something is wrong with the application design. In
development, tail -f on the client's log file does a pretty good job.

That having been said, I'm not opposed to adding more introspection
abilities; I just want to do it in a way that's in keeping with our
high standard for simplicity and speed. I'll send another message soon
to discuss using a radix tree instead of a binary heap, so we can have
efficient paging for the contents of a tube.

> 2. What happens with failed jobs? (this may be client side)
(Continue reading)

Permalink | Reply | 0 comments |
headers
Keith Rarick | 5 May 2010 00:40
Picon
Favicon
Gravatar

radix trees

I've been thinking about the possibility of using radix trees (aka
crit bit trees or patricia trees) in two places in beanstalkd.

1. Replace the global job hashtable. This would give us constant-time
worst-case insertion, where a hashtable only gives *amortized*
constant-time insertion.

2. Replace the binary heaps. This would let us answer range queries on
the ready queue, for better introspection. This prospect makes me a
little sad, not for any rational reason, but just because the binary
heap is such a beautiful structure.

The disadvantage of a radix tree would be slightly more complex code,
more memory fragmentation, worse memory access locality, and (I think)
slightly slower average access times. The allocation overhead could be
mitigated somewhat by keeping a free list of tree node structs.

Hashtables and binary heaps are extremely simple data structures, but
radix trees are only a little more complex. Nowhere near as bad as
something like AVL trees or red-black trees.

Once I get issue 38 fixed I'm going to do an initial implementation of
this and compare the performance. If speed says comparable to what we
have now, I'll go ahead and make the switch.

In the meantime, if anyone has questions or comments about this, as
always, chime in!

kr
(Continue reading)

Permalink | Reply | 2 comments |
headers
Cody Caughlan | 5 May 2010 00:46
Picon
Gravatar

Re: radix trees

Keith, you rock.

I love Beanstalk for many reasons, certainly its simplicity and
ease-of-use. That being said, I would love some introspection
abilities. I have read your earlier posts about how introspection and
speed can be mutually exclusive, so I am aware of the trade-offs. But
still, having better introspection abilities would be a great missing
feature.

Thus, if you do this test implementation and your benchmarks show it
comparable to the existing hashtable approach then I welcome my new
radix tree overlords.

Keep up the great work!

Thanks
/Cody

On Tue, May 4, 2010 at 3:40 PM, Keith Rarick <kr@...> wrote:
> I've been thinking about the possibility of using radix trees (aka
> crit bit trees or patricia trees) in two places in beanstalkd.
>
> 1. Replace the global job hashtable. This would give us constant-time
> worst-case insertion, where a hashtable only gives *amortized*
> constant-time insertion.
>
> 2. Replace the binary heaps. This would let us answer range queries on
> the ready queue, for better introspection. This prospect makes me a
> little sad, not for any rational reason, but just because the binary
> heap is such a beautiful structure.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Kurt Seifried | 6 May 2010 06:55
Gravatar

Re: radix trees

> Once I get issue 38 fixed I'm going to do an initial implementation of
> this and compare the performance. If speed says comparable to what we
> have now, I'll go ahead and make the switch.
>
> In the meantime, if anyone has questions or comments about this, as
> always, chime in!

I'd say this is all moot until you implement and benchmark it, I mean
if this is a single digit (percentage wise) performance hit than go
for it, but if it's like 50% slower it might not be a good idea =). I
haven't actually bothered to benchmark beanstalkd in my application
because I find it is pretty much instantaneous for all intents and
purpose (which is one reason I like it). Personally I'd rather see a
light weight authentication layer (username/password) and/or TLS
support added so it can be exposed to non-trusted networks (for use in
Amazon EC2, etc.).

> kr

-Kurt

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
Permalink | Reply | 0 comments |
headers
Keith Rarick | 10 May 2010 23:46
Picon
Favicon
Gravatar

Re: Possible bug when inserting 179366 jobs and then deleting them

This is fixed in git along with a test case. Would you mind trying it
out? If someone can confirm the fix, I'll make another release.

kr

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
Permalink | Reply | 0 comments |

Gmane