We're using pybeanstalk 0.11.1, python 2.6.5, Ubuntu 10.04.3 LTS. The main loop of our multi-threaded beanstalk client looks like:
while True:
m = mq.reserve()
jid = m['jid']
try:
# process job
except Exception:
logger.error("skipped job %d" % jid)
finally:
logger.info("deleted job %d" % jid)
mq.delete(jid)
Every once in a while (yesterday it happened once in 60k jobs processed), the delete() call in the finally block raises a NotFound exception. I've included a stack dump below.
2012-04-02T04:56:57+00:00 cluster1.songza.com 2012-04-02 04:56:57,497 - scrobble_mill - Caught NotFound('Server returned: NOT_FOUND',) in <Thread(Thread-9, started 139969929115392)>
Traceback (most recent call last):
File "/home/songza/deploy/current/scrobble/scrobble_mill.py", line 98, in listen_wrapper
listen(host, port)
File "/home/songza/deploy/current/scrobble/scrobble_mill.py", line 162, in listen
mq.delete(jid)
File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 94, in caller
*getattr(protohandler, 'process_%s' % attr)(*args, **kw))
File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 68, in _do_interaction
return self._get_response(handler)
File "/usr/lib/pymodules/python2.6/beanstalk/serverconn.py", line 58, in _get_response
res = handler(recv)
File "/usr/lib/pymodules/python2.6/beanstalk/protohandler.py", line 56, in __call__
return self.__h(val)
File "/usr/lib/pymodules/python2.6/beanstalk/protohandler.py", line 68, in handler
checkError(response)
File "/usr/lib/pymodules/python2.6/beanstalk/errors.py", line 35, in checkError
raise err
NotFound: Server returned: NOT_FOUND
Looking over the logs, it appears that two different threads got the same job in their reserve() calls. Is this a bug? I was assuming that beanstalk was thread safe, but maybe not?
--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/beanstalk-talk/-/GMkJoHdsoOgJ.
To post to this group, send email to beanstalk-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.