Jon Cooper | 1 Aug 2011 16:46
Picon

Re: beanstalk-client ruby gem -- FD_CLOEXEC doesn't exist in windows

This has been working for me with various builds of Ruby 1.8.7:

     if RUBY_PLATFORM.match(/i386-mswin32/) != nil ||
RUBY_PLATFORM.match(/i386-mingw32/) != nil ||
RUBY_PLATFORM.match(/cygwin/) != nil
       -aeTkqpBoj2WDhNf7Hbop5Q@public.gmane.org(Fcntl::F_SETFL, Fcntl::O_NONBLOCK)
     else
       -aeTkqpBoj2WDhNf7Hbop5Q@public.gmane.org(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
     end

>> Anyways, commenting that one line renders a gem that seems to work for
>> me for development purposes in windows. 

The FD_CLOEXEC flag is part of the POSIX API, which Windows doesn't support, but it does  provide analogous functionality. 


You can implement a patch with https://github.com/jarib/childprocess/ which appears to provide a consistent facade. 


Cheers,
Jon

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
rstradling | 2 Aug 2011 21:03
Picon

Reliability

I am looking to use beanstalk on our next project and have a lot of
questions about reliability and/or how others handle it.  The reason I
am here is I am curious if binlogs work as well.  I have seen a decent
amount of posts saying they do and some saying they are experimental.

But when I think of this I am looking at
Is sending of the job reliable to the server?  I believe this is a no
and the best way to handle this would be via a master slave type
arrangement where they both watch the same queue.  The slave would
monitor the master and if the master is down would start processing
the queue.  Is that what people would recommend?

Are tubes fault tolerant?  If the machine with the tubes fail what
happens?  How is redundancy handled for this?  I believe that is where
bin logs enter the equation.  Correct?

I appreciate any help or advice.

Thanks,
Ryan

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

Chad Kouse | 2 Aug 2011 21:40
Picon
Gravatar

Re: Reliability

We've been using beanstalkd for a year or two now with really very minimal problems.  That said we've never had one of our beanstalkd servers totally die (knocking on wood).  We have had the process crash once or twice and when we restarted it it read the binlogs and started back up without incident.


The way our setup is -- we have 3 beanstalkd servers all identical and all to be used for holding the same tubes.  Then in our producers we randomly distribute jobs to one of the 3 servers -- skipping over any server if it happens to be down.

On the consumer side we have 3 servers all running the same consumers, but pointed at particular servers.  so consumer A points to beanstalkd server A, consumer B to beanstalkd server B, and so forth.

In this way we are able to reliably keep job production running even in the event of a beanstalkd server failing, and we are similarly able to keep consumption going in the same situation.  We monitor the backlog in each of our tubes on each server to make sure the consumers are still running and able to keep up with production.

Hope it helps!
--chad

On Tue, Aug 2, 2011 at 3:03 PM, rstradling <ryanstradling-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
I am looking to use beanstalk on our next project and have a lot of
questions about reliability and/or how others handle it.  The reason I
am here is I am curious if binlogs work as well.  I have seen a decent
amount of posts saying they do and some saying they are experimental.

But when I think of this I am looking at
Is sending of the job reliable to the server?  I believe this is a no
and the best way to handle this would be via a master slave type
arrangement where they both watch the same queue.  The slave would
monitor the master and if the master is down would start processing
the queue.  Is that what people would recommend?

Are tubes fault tolerant?  If the machine with the tubes fail what
happens?  How is redundancy handled for this?  I believe that is where
bin logs enter the equation.  Correct?

I appreciate any help or advice.

Thanks,
Ryan

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.


--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
Matt Hampton | 7 Aug 2011 20:30
Picon

Persistent Reservations

Hi

I have been playing with adding a new set of features to beanstalkd
and wanted to know if there were anyone else is interested (so that I
can finish the code and tidy it up).

A bit of history: I updating an application that processes inbound
emails from customers.  These need to be processed in order of arrival
and are handled by a web application.  I already use beanstalkd for
other components of the system (and it works really well ;-) ) so
wanted to replace the database based queue with beanstalkd.  The issue
(obviously) is that each request that initiates the connection to the
server, reserves a job and then closes the connection - instantly
releasing the job.

So I have added a new command to the grammar of the protocol.
"preserve".  This is only valid after a a job is reserved (in the same
way as "bury" works - however the preservation is only valid until the
existing reservation. It releases in the same way as a normal
reservation (i.e. when the expiry time is reached or if the server is
restarted and binary logging is enabled) but not if the connection is
dropped.

I have got this most of this working now so wanted to know how
interested people were.

cheers

Matt

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

Dustin Norlander | 10 Aug 2011 02:26
Picon

Re: Persistent Reservations

Very cool.  The job per connection paradigm is the biggest problem
with beanstalkd, would be great to have a solution.

Please post your fork when you're ready.

-Dustin

On Sun, Aug 7, 2011 at 11:30 AM, Matt Hampton
<matt.hampton.uk@...> wrote:
> Hi
>
> I have been playing with adding a new set of features to beanstalkd
> and wanted to know if there were anyone else is interested (so that I
> can finish the code and tidy it up).
>
> A bit of history: I updating an application that processes inbound
> emails from customers.  These need to be processed in order of arrival
> and are handled by a web application.  I already use beanstalkd for
> other components of the system (and it works really well ;-) ) so
> wanted to replace the database based queue with beanstalkd.  The issue
> (obviously) is that each request that initiates the connection to the
> server, reserves a job and then closes the connection - instantly
> releasing the job.
>
> So I have added a new command to the grammar of the protocol.
> "preserve".  This is only valid after a a job is reserved (in the same
> way as "bury" works - however the preservation is only valid until the
> existing reservation. It releases in the same way as a normal
> reservation (i.e. when the expiry time is reached or if the server is
> restarted and binary logging is enabled) but not if the connection is
> dropped.
>
> I have got this most of this working now so wanted to know how
> interested people were.
>
> cheers
>
> Matt
>
> --
> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
> To post to this group, send email to beanstalk-talk@...
> To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>
>

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

Daniel | 10 Aug 2011 15:12
Picon

Deleting a Delayed job

Is there any way to delete a delayed job, or modify a delayed job? I
am trying to run several commands several minutes after user action,
but if there is a duplicate job in delay, I do not want it to run
twice.

I know beanstalkd does not support deleting delayed jobs currently,
but is there any sort of work around?

- Daniel

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

Chad Kouse | 10 Aug 2011 23:46
Picon
Gravatar

Re: Deleting a Delayed job

Nothing I can think of that's too elegant. You could use memcached
though and stick a record in there relating to the delayed job. Then
when you want to delete the job just delete the associated memcached
object. And when you are consuming jobs check for the associated
record in memcached first.

--chad

On Aug 10, 2011, at 5:25 PM, Daniel <dolfelt@...> wrote:

> Is there any way to delete a delayed job, or modify a delayed job? I
> am trying to run several commands several minutes after user action,
> but if there is a duplicate job in delay, I do not want it to run
> twice.
>
> I know beanstalkd does not support deleting delayed jobs currently,
> but is there any sort of work around?
>
> - Daniel
>
> --
> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
> To post to this group, send email to beanstalk-talk@...
> To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>

--

-- 
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

Daniel Olfelt | 11 Aug 2011 02:21
Picon

Re: Deleting a Delayed job

How hard would it be to add deleting of delayed jobs? Is there a reason it isn't an option? If it's just time, I might consider trying my hand at adding it.

--Daniel

On Aug 10, 2011 4:46 PM, "Chad Kouse" <chad.kouse-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Nothing I can think of that's too elegant. You could use memcached
> though and stick a record in there relating to the delayed job. Then
> when you want to delete the job just delete the associated memcached
> object. And when you are consuming jobs check for the associated
> record in memcached first.
>
> --chad
>
> On Aug 10, 2011, at 5:25 PM, Daniel <dolfelt-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> Is there any way to delete a delayed job, or modify a delayed job? I
>> am trying to run several commands several minutes after user action,
>> but if there is a duplicate job in delay, I do not want it to run
>> twice.
>>
>> I know beanstalkd does not support deleting delayed jobs currently,
>> but is there any sort of work around?
>>
>> - Daniel
>>
>> --
>> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
>> To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
>> To unsubscribe from this group, send email to beanstalk-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
>> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
> To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
> To unsubscribe from this group, send email to beanstalk-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
Chad Kouse | 11 Aug 2011 02:46
Picon
Gravatar

Re: Deleting a Delayed job

From reading the protocol doc - maybe you could work around it if you know the id. You might be able to bury and then delete the job. Not sure. 

--chad

On Aug 10, 2011, at 8:21 PM, Daniel Olfelt <dolfelt-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

How hard would it be to add deleting of delayed jobs? Is there a reason it isn't an option? If it's just time, I might consider trying my hand at adding it.

--Daniel

On Aug 10, 2011 4:46 PM, "Chad Kouse" <chad.kouse-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Nothing I can think of that's too elegant. You could use memcached
> though and stick a record in there relating to the delayed job. Then
> when you want to delete the job just delete the associated memcached
> object. And when you are consuming jobs check for the associated
> record in memcached first.
>
> --chad
>
> On Aug 10, 2011, at 5:25 PM, Daniel <dolfelt-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> Is there any way to delete a delayed job, or modify a delayed job? I
>> am trying to run several commands several minutes after user action,
>> but if there is a duplicate job in delay, I do not want it to run
>> twice.
>>
>> I know beanstalkd does not support deleting delayed jobs currently,
>> but is there any sort of work around?
>>
>> - Daniel
>>
>> --
>> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
>> To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
>> To unsubscribe from this group, send email to beanstalk-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
>> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
> To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
> To unsubscribe from this group, send email to beanstalk-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
Daniel Olfelt | 11 Aug 2011 04:29
Picon

Re: Deleting a Delayed job

Yeah, I thought that would work, but bury works on the same objects that delete does. I do know the ID of the job I am trying to delete from the delay queue, but when I try to delete it, even though I know it is in the queue, it returns NOT_FOUND.


--Daniel

On Wed, Aug 10, 2011 at 7:46 PM, Chad Kouse <chad.kouse-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
From reading the protocol doc - maybe you could work around it if you know the id. You might be able to bury and then delete the job. Not sure. 

--chad

On Aug 10, 2011, at 8:21 PM, Daniel Olfelt <dolfelt-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

How hard would it be to add deleting of delayed jobs? Is there a reason it isn't an option? If it's just time, I might consider trying my hand at adding it.

--Daniel

On Aug 10, 2011 4:46 PM, "Chad Kouse" <chad.kouse-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Nothing I can think of that's too elegant. You could use memcached
> though and stick a record in there relating to the delayed job. Then
> when you want to delete the job just delete the associated memcached
> object. And when you are consuming jobs check for the associated
> record in memcached first.
>
> --chad
>
> On Aug 10, 2011, at 5:25 PM, Daniel <dolfelt-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> Is there any way to delete a delayed job, or modify a delayed job? I
>> am trying to run several commands several minutes after user action,
>> but if there is a duplicate job in delay, I do not want it to run
>> twice.
>>
>> I know beanstalkd does not support deleting delayed jobs currently,
>> but is there any sort of work around?
>>
>> - Daniel
>>
>> --
>> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
>> To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
>> To unsubscribe from this group, send email to beanstalk-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
>> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
> To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
> To unsubscribe from this group, send email to beanstalk-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
> For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

Gmane