Should requent queue creation-attachment-deletion be avoided?

Hi,

I am thinking how to model my problem with RabbitMQ and AMQP. Let me
describe it briefly.

There are something like 1000 entities about which central site should
publish and broadcast information. There are also something like
100-1000 clients which can choose only one of those items and receive
pushed updates to its state. I was thinking about setting an exchange
to which updates are going to be published, and each client would get
its private queue attached to it to fetch updates from. The updates
need not to be persisted or ack-ed.

Now the clients might at some times quite frequently change the entity
they are monitoring, maybe even once per second if they are edgy. Over
the period of 6-10 hours that might cumulate to quite large number of
private queues being created, attached to exchanges and deleted
subsequently. Is this going to be costly and possible time and
resource drain?

Maybe queues need not to be deleted but they could be just reattached
with different keyword pattern? Is that possible and if yes, would it
also present performance and resource issue?

Or maybe this should be modeled in some other way?

Many thanks!

Davorin Rusevljan
http://www.cloud208.com/

Re: Should requent queue creation-attachment-deletion be avoided?

Davorin

You are definitely thinking along the right lines here.

* Queue creation, binding, etc, carries a small cost in time and space
* For some use cases (eg managing a lot of chat style sessions) it
would be great to have a way to 'hide' bindings temporarily, instead
of deleting them
* A mechanism akin to 'presence' has been mooted but not implemented yet

Space costs:

* The space cost is usually the more critical to considerations of
scale in our practical experience with real apps, to date
* Each binding takes a couple of 100 bytes, last time I asked the
engineering team about this, but this may have improved recently, or
be about to improve with the new persistor - you'll have to ask
Matthew
* Obviously if you have a large machine (8Gb RAM or more) this goes a
long way to helping.
* Memory consumption in RabbitMQ 1.7 is the cost of the entities plus
the bindings plus any messages in memory (messages are stored at most
once per node if they are larger than 60 bytes)
* Per message memory consumption will decrease for persistent
messages, with the new persistor
* For more on this, read
http://www.lshift.net/blog/2009/12/07/rabbitmq-at-the-skills-matter-functional-programming-exchange

Time costs:

Re: Encoding message payload for multiple languages

On 12/31/09 11:35 AM, Jeremy Dunck wrote:
> 
> 
> On Dec 31, 2009, at 1:17 PM, Garrett Smith <g <at> rre.tt> wrote:
>> Binary representation of Erlang terms is darned near perfect (fast and
>> flexible), but it's not as easy to consume from other language
>> environments.
>>
> 
> Have you seen Bert?
> http://github.com/blog/531-introducing-bert-and-bert-rpc
> 

Another vote for BERT.  I'm using it quite happily:
http://github.com/b/bertrem .  Piece of cake to encode/decode as
evidenced by the proliferation of client libraries.

b

On 12/31/09 11:35 AM, Jeremy Dunck wrote:
> 
> 
> On Dec 31, 2009, at 1:17 PM, Garrett Smith <g <at> rre.tt> wrote:
>> Binary representation of Erlang terms is darned near perfect (fast and
>> flexible), but it's not as easy to consume from other language
>> environments.
>>
> 

Re: Encoding message payload for multiple languages

On 12/31/09 11:51 AM, Alexis Richardson wrote:
> On Thu, Dec 31, 2009 at 7:35 PM, Jeremy Dunck <jdunck@...> wrote:
>>
>>
>> On Dec 31, 2009, at 1:17 PM, Garrett Smith <g <at> rre.tt> wrote:
>>> Binary representation of Erlang terms is darned near perfect (fast and
>>> flexible), but it's not as easy to consume from other language
>>> environments.
>>>
>>
>> Have you seen Bert?
>> http://github.com/blog/531-introducing-bert-and-bert-rpc
> 
> And not wholly unrelatedly: http://github.com/b/bertem
> 

FYI, I inconveniently changed the name to bertrem so it could be
pronounced as a name, as is the case with most other BERT bits.  Also
for the lulz.

http://github.com/b/bertrem

b

On 12/31/09 11:51 AM, Alexis Richardson wrote:
> On Thu, Dec 31, 2009 at 7:35 PM, Jeremy Dunck <jdunck@...> wrote:
>>
>>

Re: Rabbit-MQ crashing

We're experiencing this problem as well.  Just from reading the list
it seems that the new persister stuff committed around 12/1 is
supposed to fix it.

I'm curious:

- has this bug indeed been fixed?
- is the code somewhere past bleeding edge?  Would you consider it to be stable?
- please excuse my ignorance, but is there any bug or ticket url that
I can follow to keep track of the fix / when it's been released / etc
?

Regards,
Steve

On Thu, Nov 12, 2009 at 6:25 AM, Tony Garnock-Jones <tonyg@...> wrote:
> Hi Derek,
>
> (Replying here as well for the archive)
>
> Derek Burdick wrote:
>> I have a durable, fanout exchange and queue.  I post persistent messages
>> to the exchange.  After sending messages for some time, the ERL process
>> stops and I can not restart RabbitMQ without deleting the contents of
>> the Database.  Here is the log from startup:
>> [...]
>>                               {{{badmatch,eof},
>> [{rabbit_persister,internal_load_snapshot,2},
>> [...]
>

AMQP message publishing - I wish it was synchronous

Before I begin let me state for the record that I have read the 0-8  
and 0-9 AMQP specs and realise that they will not change just because  
I believe that it might be a good idea. Neither do I advocate  
deviation from the specs for the sake of expedience.

OK, I've been looking at including Channel.Flow method processing in  
the Bunny client (again) and have come to the conclusion that  
publishing should be synchronous in AMQP terms. My reasoning is as  
follows -

The AMQP Channel.Flow method is a device to try to prevent servers  
getting swamped by message producing clients. The server sends a  
Channel.Flow method with :active argument set to false to a producer  
in order to get the producer to stop sending messages when server  
memory is getting critically low.

When a client application publishes a message it passes that message  
into the care of the server. If something goes wrong the client  
application needs to be notified, but it is equally important for the  
client to know that the server has the message and will do its level  
best to deliver it to an appropriate destination queue. It seems to  
me that the second piece of information, namely a "yes I have your  
message", is missing. This absence of information means that the  
client has to make a judgement on the success of the publish  
operation based on what it does not receive, rather than what is does  
receive.

If, after issuing a Basic.Publish method, a client does not receive  
an error indication or Channel.Flow method, it has to assume that a  
publish has succeeded. For a simple single-threaded synchronous  

Re: AMQP message publishing - I wish it was synchronous

It would be nice to have a lazy ack and things like that will be in
future versions of amqp.

For now, have you looked at the TX class?

alexis

On Mon, Jan 4, 2010 at 9:26 PM, Chris Duncan <celldee@...> wrote:
> Before I begin let me state for the record that I have read the 0-8
> and 0-9 AMQP specs and realise that they will not change just because
> I believe that it might be a good idea. Neither do I advocate
> deviation from the specs for the sake of expedience.
>
> OK, I've been looking at including Channel.Flow method processing in
> the Bunny client (again) and have come to the conclusion that
> publishing should be synchronous in AMQP terms. My reasoning is as
> follows -
>
> The AMQP Channel.Flow method is a device to try to prevent servers
> getting swamped by message producing clients. The server sends a
> Channel.Flow method with :active argument set to false to a producer
> in order to get the producer to stop sending messages when server
> memory is getting critically low.
>
> When a client application publishes a message it passes that message
> into the care of the server. If something goes wrong the client
> application needs to be notified, but it is equally important for the
> client to know that the server has the message and will do its level
> best to deliver it to an appropriate destination queue. It seems to
> me that the second piece of information, namely a "yes I have your

Re: AMQP message publishing - I wish it was synchronous

Hi Alexis,

I have looked at it, but I obviously didn't understood it correctly.  
So, let's see if I grok this. Suppose I send a tx.select, send my  
basic.publish and then send a tx.commit. If I receive a tx.commit-ok  
all is well, otherwise I'll receive a channel exception from the server.

Not as straightforward as a publish-ok, but hey, if it works ... :)

Thanks,

Chris

On 4 Jan 2010, at 22:11, Alexis Richardson wrote:

> It would be nice to have a lazy ack and things like that will be in
> future versions of amqp.
>
> For now, have you looked at the TX class?
>
> alexis
>
>
> On Mon, Jan 4, 2010 at 9:26 PM, Chris Duncan <celldee@...>  
> wrote:
>> Before I begin let me state for the record that I have read the 0-8
>> and 0-9 AMQP specs and realise that they will not change just because
>> I believe that it might be a good idea. Neither do I advocate
>> deviation from the specs for the sake of expedience.
>>

Re: AMQP message publishing - I wish it was synchronous

Roughly.

Using TX allows you to get a synchronous ack.  But it's comparitively
slow obviously.  You can do acks asyncly but that has a weaker SLA of
course.

I am being vague in the hope that a better-informed person will make a
more precise and possibly helpful statement about how to make the most
of this 

Meanwhile I suggest you experiment based on what you glean from the
spec and searching the rabbitmq.com site...

alexis

On Mon, Jan 4, 2010 at 10:57 PM, Chris Duncan <celldee@...> wrote:
> Hi Alexis,
>
> I have looked at it, but I obviously didn't understood it correctly. So,
> let's see if I grok this. Suppose I send a tx.select, send my basic.publish
> and then send a tx.commit. If I receive a tx.commit-ok all is well,
> otherwise I'll receive a channel exception from the server.
>
> Not as straightforward as a publish-ok, but hey, if it works ... :)
>
> Thanks,
>
> Chris
>
> On 4 Jan 2010, at 22:11, Alexis Richardson wrote:

Re: AMQP message publishing - I wish it was synchronous

OK. I'll keep playing. Thanks for your input.

Regards,

Chris

On 4 Jan 2010, at 23:03, Alexis Richardson wrote:

> Roughly.
>
> Using TX allows you to get a synchronous ack.  But it's comparitively
> slow obviously.  You can do acks asyncly but that has a weaker SLA of
> course.
>
> I am being vague in the hope that a better-informed person will make a
> more precise and possibly helpful statement about how to make the most
> of this 
>
> Meanwhile I suggest you experiment based on what you glean from the
> spec and searching the rabbitmq.com site...
>
> alexis
>
>
>
>
> On Mon, Jan 4, 2010 at 10:57 PM, Chris Duncan <celldee@...>  
> wrote:
>> Hi Alexis,
>>