headers
Dave Syer | 1 Nov 2010 01:47
Picon
Favicon
Gravatar

Re: Missing features making me look at moving off RabbitMQ


On Oct 29, 6:46 pm, Matthias Radestock <matth...@...> wrote:
> We are working on better and more integrated HA. Meanwhile, the
> instructions for the existing active/passive HA may look a bit
> daunting, but it's actually not that hard to set up.

I also found the existing HA guide rather hard going.  I suppose that
doesn't help much, but it's good to share the pain sometimes.

I think I would probably want to try and have the hypervisor deal with
HA (e.g. shadow VMs in vSphere).

> > -- Support custom authentication schemes <e.g. LDAP>

Spring Security and Spring Integration already work together (and LDAP
is a dime a dozen for Spring Security users), so Spring AMQP with
Spring Integration might provide a nice high-level wrapper for this.

> > -- Support message-level permissions
> That sounds like a pretty hard problem. Are there any messaging systems
> out there that do something close to what you need here?

Spring Security is pretty good for that kind of requirement also. We
could try it pretty soon once Spring AMQP and Spring Integration get
properly synched up (should be any time, now that Spring Integration
2.0 is getting close to final).

Dave.
Permalink | Reply | 2 comments |
headers
Simon MacMullen | 1 Nov 2010 11:56
Favicon
Gravatar

Re: RabbitMQ and Splunk

On 30/10/10 01:13, Michael Vierling wrote:
> We're developing a Splunk plugin for RabbitMQ.  Splunk is an excellent
> log search engine and we highly recommend it. Anyway, while Splunk can
> ingest almost any log files, it prefers a key-value pair format. You can
> see this page for more details:
>
> http://www.splunk.com/wiki/Apps:Common_Information_Model
>
> So in that spirit, I'd like to propose the following patch to the
> rabbitmqadmin script, which ships with your management 2.1.1 plugin.
> This patch adds a key-value pair Splunk compatible option (kvp) to the
> script. It would be very helpful if this could be incorporated into the
> official Management plugin.

<snip>

Hi Michael.

First of all, it's nice to know someone's using rabbitmqadmin - I think 
you're the first to ask about it...

In order to accept your patch, I'd need to get you to sign our 
contributor agreement (yes, even for something this small). Or I could 
reimplement it; that might be easier.

But before that, can we clear up a few issues:

* I'd like to not call it kvp since it's really timestamp - key/value, 
which seems quite Splunk-specific.
* I hadn't thought of using rabbitmqadmin to generate log files. I
(Continue reading)

Permalink | Reply | 6 comments |
headers
jiri | 1 Nov 2010 13:22

Protection against slow consumers

Hello together

I'm looking for a way how to protect against slow consumers.

We are publishing messages to a topic exchange as part of a public  
interface. The subscribers are applications developed by our clients  
(outside of our control).

Our concern is that one slow consumer could potentially force the  
broker to throttle our publisher, affecting all other consumers.

We encourage the consumers to use a small prefetch buffer, but have no  
way how to enforce this. In general, we cannot assume that all  
consumers cooperate.

For instance, a client that is consuming but not acking will (I  
believe) impact the broker memory consumption.

Your ideas are welcome.

Cheers
Jiri Krutil
Deutsche Boerse Systems AG
Permalink | Reply | 21 comments |
headers
Eric@SPIN | 1 Nov 2010 12:58
Favicon

Why do I get unackowledged messages?


This question may have an obvious answer but I don't know what it is. I've
got about 10 queues on an exchange. I'm enqueing/dequeing via the .NET
client version 2.1.1.0. The behaviour that i'm seeing is that messages stay
in the unacknowledged queue for long (and seemingly random) periods of time.
The system is not under any significant load. For some of the queues the
messages are eventually processed for others they remain for hours. I use
the same code for all the queues. What could be going wrong?
--

-- 
View this message in context: http://old.nabble.com/Why-do-I-get-unackowledged-messages--tp30104183p30104183.html
Sent from the RabbitMQ mailing list archive at Nabble.com.
Permalink | Reply | 1 comment |
headers
Alexandru Scvorţov | 1 Nov 2010 14:10
Favicon

Re: Is the "master" node mandatory ?

> To summarize, the list of exchanges and queues are recreated at 
> reconnection time by producers and consumers, thus, the "state" of 
> the cluster can be lost. That's why I think we can use RAM node 
> only. But maybe the "state" of the cluster is not limited to the 
> list of exchanges and queues ?

The "state" of the cluster is just the names of the nodes currently in
the cluster and the queues/exchanges declared on said nodes.

All nodes, whether ram or disc record the names of the other nodes in
the cluster, so even ram nodes will rejoin the cluster when restarted.

Ram nodes don't store most of their mnesia tables on disk, so they won't
remember queues/exchanges/messages on restart.

Your scenario seems fine: if you don't mind having to redeclare
queues/exchanges on every use and losing messages on node failure, using
a cluster of all ram nodes is ok.

Also, sorry for the late reply.

Cheers,
Alex

On Wed, Oct 20, 2010 at 09:19:30AM +0200, Vincent Barat wrote:
>   Thank you for this long reply, I was not aware about that.
> 
> In our use case, all queues are not persistent. Thus, when a node 
> fails, all messages of all queues it handles are lost (since 
> messages are not replicated between nodes).
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jon Brisbin | 1 Nov 2010 14:42
Favicon
Gravatar

Re: Understanding RabbitMQ clustering


On Oct 30, 2010, at 3:28 AM, Kshitiz Garg wrote:

> Now if I use a HA proxy and have independent RabbitMQs sitting behind
> that proxy, how will that be different from using a HA proxy and have
> clustered RabbitMQs sitting behind that proxy. What is that these
> clustered RabbitMQs have in common?  Actually I want to understand
> "clustering" in this regard.

If the brokers aren't clustered, your individual RabbitMQ brokers won't share users, exchanges, queues,
or any other resource. I'm not sure how that affects applications because I haven't tried it. I suspect it
wouldn't work very well with round-robin load balancing because consumers might get connected to a
different broker than the producers.

If you're using only the failover configuration, though, it would be better. You'd just have to
reestablish all your Channels if one of them gets disconnected, assuming the next connection will go to
the other broker.

"Clustered" RabbitMQ brokers look the same no matter which actual broker you contact. Messages delivered
to one broker will be delivered to consumers on another broker. Without this clustering, you have to
create some kind of store-and-forward (maybe using the shovel plugin) or your application has to manage
that itself.

If your brokers are clustered, upgrading versions of the broker is, in my experience, fragile. Every
version upgrade I've done (we're running it on Ubuntu Linux) results in a dead broker that has to be
manually restarted. I've also had to re-configure the broker back into the cluster more times than I'd
like. 

<soapbox>
The broker upgrade process could use some attention, IMHO. The broker should restart after upgrades
(Continue reading)

Permalink | Reply | 5 comments |
headers
Simon MacMullen | 1 Nov 2010 14:47
Favicon
Gravatar

Re: Understanding RabbitMQ clustering

On 01/11/10 13:42, Jon Brisbin wrote:
> <soapbox> The broker upgrade process could use some attention, IMHO.
> The broker should restart after upgrades without manual intervention
> and new versions should ALWAYS be able to migrate data so I don't
> have to reconfigure all my users and clustering configuration.
> </soapbox>

We're working on it. It's very likely 2.0.0 -> 2.1.0 will be the last 
such reconfiguration-requiring upgrade.

Cheers, Simon
--

-- 
Simon MacMullen
Staff Engineer, RabbitMQ
SpringSource, a division of VMware
Permalink | Reply | 2 comments |
headers
Emile Joubert | 1 Nov 2010 14:51
Favicon

Re: Lose messages between restarts of broker


Hi Rudi,

Op 28/10/2010 19:50, het Rudi Wirth geskryf:
>> If you really need messages to go safely to disk, then you should
>> put the channel into transactional mode (tx.select), and issue a
>> tx.commit
>
> What about unacknowledged messages?
>
> A Producer (rabbit 2.1.1 java client, not Spring Rabbit AMQP) commits
> some persistent messages on a durable queue/exchange.
>
> The Consumer is neither ack'ing nor rejecting the message to leave it
> on the queue (and let recovery retry it later), but I am committing
> periodically, e.g.:
>
> QueueingConsumer.Delivery delivery = consumer.nextDelivery(); //
> neither channel.basicReject or Ack channel.txCommit();
>
> Per ctl:
>
> $ rabbitmqctl list_queues name messages messages_unacknowledged
> messages_ready Listing queues ... testQueue       5       5       0
>
> Using a soft "rabbitmqctl stop", a Broker restart will make the
> unacknowledged messages ready again (as expected). testQueue       5
> 0       5
>
> With a Broker hardkill, the unack'd messages disappear. testQueue
(Continue reading)

Permalink | Reply | 2 comments |
headers
Emile Joubert | 1 Nov 2010 15:12
Favicon

Re: Why do I get unackowledged messages?


Hi Eric,

Op 01/11/2010 11:58, het Eric <at> SPIN geskryf:
>
> This question may have an obvious answer but I don't know what it is. I've
> got about 10 queues on an exchange. I'm enqueing/dequeing via the .NET
> client version 2.1.1.0. The behaviour that i'm seeing is that messages stay
> in the unacknowledged queue for long (and seemingly random) periods of time.
> The system is not under any significant load. For some of the queues the
> messages are eventually processed for others they remain for hours. I use
> the same code for all the queues. What could be going wrong?

The most likely cause is that consumers are not acknowledging messages. 
You could set the noAck flag when consuming messages or call the 
BasicAck() method after processing a message.

Regards

Emile
Permalink | Reply | 0 comments |
headers
Simon MacMullen | 1 Nov 2010 15:45
Favicon
Gravatar

Re: Understanding RabbitMQ clustering

On 01/11/10 13:47, Simon MacMullen wrote:
> On 01/11/10 13:42, Jon Brisbin wrote:
>> <soapbox> The broker upgrade process could use some attention, IMHO.
>> The broker should restart after upgrades without manual intervention
>> and new versions should ALWAYS be able to migrate data so I don't
>> have to reconfigure all my users and clustering configuration.
>> </soapbox>
>
> We're working on it. It's very likely 2.0.0 -> 2.1.0 will be the last
> such reconfiguration-requiring upgrade.

Oops, got slightly ahead of myself there. What we're currently working 
on only works in the single-node case (although obviously we hope to 
extend that in the future).

Cheers, Simon

--

-- 
Simon MacMullen
Staff Engineer, RabbitMQ
SpringSource, a division of VMware
Permalink | Reply | 0 comments |

Gmane