headers
Parand Darugar | 5 Aug 2009 02:22
Picon

State of persistence in Beanstalkd


Hello,

I've been out of the loop on beanstalkd development for a while,
wanted to see what the latest is on persistence. Is persistence in the
latest version of beanstalkd from github production ready and
recommended? Apologies if this information exists somewhere already
but I wasn't able to find it. I'd also love a pointer to some usage
guides on how to enable/recover/examine persistence or some test cases
so I can build my own docs.

Much thanks,

Parand
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en
-~----------~----~----~----~------~----~------~--~---
Permalink | Reply | 11 comments |
headers
Nicolas Fouché | 5 Aug 2009 19:43
Picon
Gravatar

Re: Amazon SQS / comparisons


You can consider that SQS is more reliable than beanstalkd only
because SQS has persistences.

But we found some very very annoying things with SQS. Here is a
sample:

http://docs.amazonwebservices.com/AWSSimpleQueueService/2009-02-01/SQSDeveloperGuide/
"You can delete a queue at any time, whether it is empty or not. You
should be aware that queues have a time limit; SQS deletes messages
that have been in the queue for more than 4 days.
We reserve the right to delete a queue without notification if one of
the following actions hasn't been performed on it for 30 consecutive
days"

http://docs.amazonwebservices.com/AWSSimpleQueueService/2009-02-01/SQSDeveloperGuide/
"SQS stores copies of your messages on multiple servers for redundancy
and high availability. On rare occasions, one of the servers storing a
copy of a message might be unavailable when you receive or delete the
message. If that occurs, the copy of the message will not be deleted
on that unavailable server, and you might get that message copy again
when you receive messages. Because of this, you must design your
application to be idempotent (i.e., it must not be adversely affected
if it processes the same message more than once)."

SQS is distributed, so you inherit the drawbacks of distribution...
But you know that, it really depends on your project.

Last but not least, the 'bury' feature of beanstalkd is awesome.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Keith Rarick | 6 Aug 2009 03:24
Picon
Favicon
Gravatar

Re: State of persistence in Beanstalkd


On Tue, Aug 4, 2009 at 5:22 PM, Parand Darugar<darugar@...> wrote:
> I've been out of the loop on beanstalkd development for a while,
> wanted to see what the latest is on persistence.

I have been busy with a contract project recently, but I am almost
done with it. Then I'll finish up persistence and make a release.

If anyone else wants to work on persistence right away, I am aware of
two things that need to be done:

1. Allow beanstalkd to reserve arbitrary amounts of binlog space.
http://github.com/kr/beanstalkd/issues#issue/14

2. Go over the actual format and make sure it is robust against errors
in the data on disk, especially file truncation.
http://github.com/kr/beanstalkd/issues/#issue/15

3. Add appropriate fsync calls, with optional throttling.
http://github.com/kr/beanstalkd/issues/#issue/16

> Is persistence in the
> latest version of beanstalkd from github production ready and
> recommended?

Not quite, because of the three things above. But if you trust your
disks and power supply and don't generate big queues and don't mind
binlog files being incompatible with the official release, go for it!
:)
(Continue reading)

Permalink | Reply | 10 comments |
headers
Tom | 8 Aug 2009 21:14
Picon

Architecturally lost!?


Good evening,

I am a queue newbie and are completely lost in the plethora of
possibilities. My question is, what I will draft here, the best and
MOST simple way to reach my goal - and especially is beanstalk the way
to go? Any pointers really appreciated!

Case:
I am developing a Ruby on Rails app that basically works as a message
hub for transport drivers. The system talks to the drivers, the
drivers talk to each other and the system via the system. I expect
thousands of messages a day, later a few million max a day. Messages
are externally in different formats, like twitter, xmpp, aim and SMS.
So latency is a big problem, the message turnaround is important.
Persistence is not a big problem, because once the messages are older
than a few minutes they are worthless anyway. Simplicity and easy
integration are currently key!

Planned way to go:
My business logic runs on a full blown rails stack. This stack is
connected to a multitude of very simple, single protocol workers
(written in ruby) in their own processes via one ore more queues (in
and out). It would be ideal if the messaging solution already has
consumers that are able to convert to http/rest, xmpp etc. (RabbitMQ
seems to have at least http, but it seems like overkill to me...)

Concrete questions:
1.) Should I have two queues per protocol (incoming and outgoing). So
I end up having 6 queues if I use twitter, sms and xmpp?
(Continue reading)

Permalink | Reply | 5 comments |
headers
Tom | 8 Aug 2009 21:21
Picon

Re: Architecturally lost!?


Forgot to add, read a few comments about this being an option as well:
The messages are quite short, mostly 200 characters or less, max
5kByte. So I would prefer a messaging system where I can send the
actual message and not just a pointer, would alleviate the need for a
DB connection for everyone.

On Aug 8, 9:14 pm, Tom <qkl...@...> wrote:
> Good evening,
>
> I am a queue newbie and are completely lost in the plethora of
> possibilities. My question is, what I will draft here, the best and
> MOST simple way to reach my goal - and especially is beanstalk the way
> to go? Any pointers really appreciated!
>
> Case:
> I am developing a Ruby on Rails app that basically works as a message
> hub for transport drivers. The system talks to the drivers, the
> drivers talk to each other and the system via the system. I expect
> thousands of messages a day, later a few million max a day. Messages
> are externally in different formats, like twitter, xmpp, aim and SMS.
> So latency is a big problem, the message turnaround is important.
> Persistence is not a big problem, because once the messages are older
> than a few minutes they are worthless anyway. Simplicity and easy
> integration are currently key!
>
> Planned way to go:
> My business logic runs on a full blown rails stack. This stack is
> connected to a multitude of very simple, single protocol workers
> (written in ruby) in their own processes via one ore more queues (in
(Continue reading)

Permalink | Reply | 4 comments |
headers
Keith Rarick | 9 Aug 2009 11:13
Picon
Favicon
Gravatar

Re: Architecturally lost!?


On Sat, Aug 8, 2009 at 12:21 PM, Tom<qklxmf@...> wrote:
> On Aug 8, 9:14 pm, Tom <qkl...@...> wrote:
>> I am a queue newbie and are completely lost in the plethora of
>> possibilities. My question is, what I will draft here, the best and
>> MOST simple way to reach my goal - and especially is beanstalk the way
>> to go? Any pointers really appreciated!

Hi Tom,

Your situation sounds like a perfect fit for beanstalkd.

Simplicity is one thing beanstalkd has over every AMQP program.

Several million messages per day throughput is easy.

If you want to get started with basics, there is a tutorial in ruby at
http://devver.net/blog/2008/10/ruby-beanstalkd-distributed-worker-basics/
. It has lots of example code.

> Concrete questions:
> 1.) Should I have two queues per protocol (incoming and outgoing). So
> I end up having 6 queues if I use twitter, sms and xmpp?

If you have a process that only knows how to handle, say, outgoing
sms, it should get its own tube (aka queue).

How are the incoming jobs created? Can you generate them all in the
same format? If so, you could make one incoming tube, and have one
type of worker that handles all incoming jobs, and puts appropriate
(Continue reading)

Permalink | Reply | 3 comments |
headers
Tom | 9 Aug 2009 22:16
Picon

Re: Architecturally lost!?


Hey Keith,

thank you for your prompt reply! I am currently experimenting with the
devver examples, looks promising so far.

> If you have a process that only knows how to handle, say, outgoing
> sms, it should get its own tube (aka queue).
>
> How are the incoming jobs created? Can you generate them all in the
> same format? If so, you could make one incoming tube, and have one
> type of worker that handles all incoming jobs, and puts appropriate
> outgoing jobs into all three outgoing tubes.

I will most probably have one worker per protocol, so that i can
handle e.g. incoming rest-calls from sms or twitter via the same
worker and xmpp with another.

> I have never heard of anyone outgrowing the performance of a single
> beanstalkd process on a single box. So, probably, the only reason you
> would want more than one is for high availability of the service -- if
> a box goes down, you can still submit jobs to the other one.

Thats sounds inviting, even if performance is not my main problem.

shares an/a few internal ruby queue(s)?
>
> I'd recommend beanstalkd, but maybe I'm biased. :) Seriously, though,
> this is just what beanstalkd was designed for. A textbook example.
(Continue reading)

Permalink | Reply | 2 comments |
headers
Keith Rarick | 9 Aug 2009 22:33
Picon
Favicon
Gravatar

Re: Architecturally lost!?


On Sun, Aug 9, 2009 at 1:16 PM, Tom<qklxmf@...> wrote:
> Do you have any tutorial or examples on async_observer and its uses?
> It seems like a natural fit because I am on RoR or would you start
> with the plain beanstalk-client gem and later (when?) 'upgrade' to the
> async_observer? Or use it right away?

If you plan on using it eventually, it's easier if you start with
async_observer.

I don't know of any tutorials other than the very basic guide on
http://async-observer.rubyforge.org/ . If you get that much working,
the rest is pretty easy.

> Did I get it right, that it can make most method calls asynchronous,
> without any additional work on the backend (just a little config)?

That's the idea. :)

kr

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en
-~----------~----~----~----~------~----~------~--~---
Permalink | Reply | 1 comment |
headers
trak3r | 13 Aug 2009 15:13
Picon

is there a limit on the number of jobs in the queue?


(yes i checked the faq and searched the discussion group first :-)

i've got a workload that's going to generate about 3mm jobs and i want
to make sure beanstalkd can store them all.

thanks.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en
-~----------~----~----~----~------~----~------~--~---
Permalink | Reply | 2 comments |
headers
Jeremy Hinegardner | 13 Aug 2009 16:29

Re: is there a limit on the number of jobs in the queue?


On Thu, Aug 13, 2009 at 06:13:50AM -0700, trak3r wrote:
> 
> (yes i checked the faq and searched the discussion group first :-)
> 
> i've got a workload that's going to generate about 3mm jobs and i want
> to make sure beanstalkd can store them all.
> 
> thanks.

As far as I know, you can put jobs into beanstalkd until you run out of memory
or you reach your ulimit setting.  We were testing OUT_OF_MEMORY errors
yesterday by setting ulimit appropriately and we were able to force
OUT_OF_MEMORY errors in beanstalk.

I have had a beanstalk with at least 3mm jobs in it and it managed them.
It also took up (in this case) about 3GB of RAM :-)

enjoy,

-jeremy

--

-- 
========================================================================
 Jeremy Hinegardner                              jeremy@... 

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
(Continue reading)

Permalink | Reply | 1 comment |

Gmane