Ben Browitt | 1 Feb 06:58 2010
Picon

PubSub - all bindings have to be in memory?

Hi

I want to build a pubsub system with a finite number of queues and exchanges and a growing number of bindings.
A simple example is users (queues) that subscribe (bindings) to articles (exchanges).
In the first 24H since an article has been published we want subscribers to get updates (comments) in real time
but after that when the frequency of new comments will decrease it's ok if updates will be delivered in a delay.
I don't want to limit the number of articles a user can subscribe to, so over time the bindings can grow exponentially.

Do I have to keep all bindings in memory or is there a mode where rabbitmq can read binding keys from disk when needed?

Ben


<div><div dir="ltr">Hi<br><br>I want to build a pubsub system with a finite number of queues and exchanges and a growing number of bindings.<br>A simple example is users (queues) that subscribe (bindings) to articles (exchanges).<br>
In the first 24H since an article has been published we want subscribers to get updates (comments) in real time<br>but after that when the frequency of new comments will decrease it's ok if updates will be delivered in a delay.<br>
I don't want to limit the number of articles a user can subscribe to, so over time the bindings can grow exponentially.<br><br>Do I have to keep all bindings in memory or is there a mode where rabbitmq can read binding keys from disk when needed?<br><br>Ben<br><br><br>
</div></div>
Brian Sullivan | 1 Feb 07:41 2010

routing threads on a rabbitmq node

Hi,

I am curious if anyone on the rabbitmq team can confirm/clarify what we are seeing with respect to some throughput issues on our RMQ cluster.

The config:
- 2-node RMQ cluster, running a topic-based exchange
- 8 publishers, running on different hosts
- dozens of consumers, ~75 wildcard topic bindings, mostly running on different hosts (there are a couple running on the RMQ hosts for stats, etc)

The issue:
When we publish at a higher rate than normal, there appears to be a significant delay in the pipeline between when we publish the messages and when we receive them on the consuming side.  Since publishing is asynchronous, the publisher applications send as fast as they can, meanwhile we see an increasing delay in when we see those same messages come out on the other side.  My guess (gathered from http://www.rabbitmq.com/faq.html#node-per-CPU-core) is that there is either a single routing thread per publisher (channel), or even worse a single routing bottleneck per node.  Either way, this thread cannot route fast enough in a topic exchange (we have about 75 bindings, using wildcards) and there is a backup of messages to be routed.

This is dangerous and hard to control, since we have seen memory grow in such a way that we have a hard time stopping it.  In certain failure modes, when the publisher disconnects, all of the messages pending to be routed are discarded - possibly hours of data in our environment.  It also limits the scaling we've been able to do, since if we fire up all 8 publishers at high volume, this backup problem spreads across all streams.

The question:
Can you please elaborate on where the routing backup could be occurring, and what steps might be best to prevent this from happening?  It appears from the fact that I am waiting on the routing to happen that using flags like "mandatory" on messages is not going to help me here (though I have not tested this).

One idea:
If it is truly the case that a single thread per node might be causing this problem, then perhaps we can run a small rabbitmq node on each publisher (joined to the cluster), with the sole purpose of doing the routing load?  If we publish locally, all it would need to do is keep up with it's own routing load, not the combine routing load of 3 other publishers.  It doesn't really prevent the problem from happening though, if I can produce messages faster in a single thread than even a dedicated node can route.  Would this even help?

Thanks,
Brian

<div><p>Hi,<br><br>I am curious if anyone on the rabbitmq team can confirm/clarify what we are seeing with respect to some throughput issues on our RMQ cluster.<br><br>The config:<br>- 2-node RMQ cluster, running a topic-based exchange<br>

- 8 publishers, running on different hosts<br>- dozens of consumers, ~75 wildcard topic bindings, mostly running on different hosts (there are a couple running on the RMQ hosts for stats, etc)<br><br>The issue:<br>When we publish at a higher rate than normal, there appears to be a significant delay in the pipeline between when we publish the messages and when we receive them on the consuming side.&nbsp; Since publishing is asynchronous, the publisher applications send as fast as they can, meanwhile we see an increasing delay in when we see those same messages come out on the other side.&nbsp; My guess (gathered from <a href="http://www.rabbitmq.com/faq.html#node-per-CPU-core" target="_blank">http://www.rabbitmq.com/faq.html#node-per-CPU-core</a>) is that there is either a single routing thread per publisher (channel), or even worse a single routing bottleneck per node.&nbsp; Either way, this thread cannot route fast enough in a topic exchange (we have about 75 bindings, using wildcards) and there is a backup of messages to be routed.<br><br>This is dangerous and hard to control, since we have seen memory grow in such a way that we have a hard time stopping it.&nbsp; In certain failure modes, when the publisher disconnects, all of the messages pending to be routed are discarded - possibly hours of data in our environment.&nbsp; It also limits the scaling we've been able to do, since if we fire up all 8 publishers at high volume, this backup problem spreads across all streams.<br><br>The question:<br>Can you please elaborate on where the routing backup could be occurring, and what steps might be best to prevent this from happening?&nbsp; It appears from the fact that I am waiting on the routing to happen that using flags like "mandatory" on messages is not going to help me here (though I have not tested this).<br><br>One idea:<br>If it is truly the case that a single thread per node might be causing this problem, then perhaps we can run a small rabbitmq node on each publisher (joined to the cluster), with the sole purpose of doing the routing load?&nbsp; If we publish locally, all it would need to do is keep up with it's own routing load, not the combine routing load of 3 other publishers.&nbsp; It doesn't really prevent the problem from happening though, if I can produce messages faster in a single thread than even a dedicated node can route.&nbsp; Would this even help?<br><br>Thanks,<br>Brian<br><br></p></div>
Rajkumar S | 1 Feb 10:47 2010
Picon

Disabling channel.flow

Hi,

How can I disable memory-based flow control ?

I am using stomp to send messages  to rabbit and the rabbit server
also hosts mysql and other daemons. So I occasionally get
channel.flow error. Since stomp plugin cannot handle this, I loose
messages. I am okay with latency, but cannot tolerate message loss. So
is it possible to disable memory-based flow control completely?

I have seen http://www.rabbitmq.com/extensions.html#memsup and have tried with

-os_mon system_memory_high_watermark 0
-rabbit memory_alarms false

Both these does not seems to disable channel.flow.

with regards,

raj

Matthew Sackman | 1 Feb 11:17 2010
Picon

Re: Disabling channel.flow

Hi Rajkumar,

On Mon, Feb 01, 2010 at 03:17:16PM +0530, Rajkumar S wrote:
> How can I disable memory-based flow control ?

Which version of Rabbit are you using? - the details for disabling
memory monitoring have changed between 1.7.0 and 1.7.1.

I advise against disabling memory monitoring. If you do disable it, and
then Rabbit runs out of memory, Rabbit will crash, likely losing more
messages.

Matthew

Matthew Sackman | 1 Feb 11:56 2010
Picon

RabbitMQ-shovel

Morning,

Several people have recently been asking about a shovel for Rabbit,
which can consume messages from one consumer and deliver them to
another. Several of our clients have also been asking for such message
relocation equipment, and thus we have built one: the RabbitMQ-shovel
which is available from http://hg.rabbitmq.com/rabbitmq-shovel/

Documentation is in the form of a README, which is duplicated below.

This code has not yet been through QA, so there may be bugs in it.
We are, as always, very happy to receive bug reports. But certainly, our
testing suggests it works for us ;)

It is licensed under the MPL v1.1

Have fun!

RabbitMQ-shovel
===============

Introduction
------------

This is a plug-in for RabbitMQ that shovels messages from a queue on
one broker to an exchange on another broker. The two brokers may be
the same. The plug-in allows several shovels to be specified at the
same time. Each shovel may have a number of source and destination
brokers specified, and one of each is chosen whenever the shovel
attempts to make a connection: this permits simple round-rabbit load
balancing.

Resources can be declared upon connection to both the source and
destination brokers, and parameters can be specified for both the
reception and publishing of messages.

Requirements
------------

Currently, you must build the server from source, under branch
bug16653. You must also have checked out the rabbitmq-public-umbrella
hg repository, and have the rabbitmq-erlang-client built. From
scratch, the following commands should build RabbitMQ with the shovel
plug-in:

hg clone http://hg.rabbitmq.com/rabbitmq-public-umbrella
cd rabbitmq-public-umbrella
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-erlang-client
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-shovel
cd rabbitmq-server
hg up -C bug16653
make -j
mkdir -p plugins
cd plugins
ln -s ../../rabbitmq-erlang-client
ln -s ../../rabbitmq-shovel
cd ../../rabbitmq-erlang-client
make
cd ../rabbitmq-shovel
make
cd ../rabbitmq-server
./scripts/rabbitmq-activate-plugins
make cleandb run

Configuration
-------------

The RabbitMQ configuration file specifies the shovel
configurations. This exists by default, in
/etc/rabbitmq/rabbitmq.config under Linux systems,
%RABBITMQ_BASE%\rabbitmq.config under Windows or somewhere else under
OS X. This file configures both RabbitMQ-server and all the plugins
installed in it. It is an Erlang-syntax file of the form:

[{section1, [section1-config]},
 {section2, [section2-config]},
 ...
 {sectionN, [sectionN-config]}
].

thus a list of tuples, where the left element of each tuple names the
applications being configured. Don't forget the last element of the
list doesn't have a trailing comma, and don't forget the full-stop is
needed after closing the list. Hence if you configure RabbitMQ-server
and the RabbitMQ-shovel, then the configuration file may have a
structure like this:

[{rabbit,        [configuration-for-RabbitMQ-server]},
 {rabbit-shovel, [configuration-for-RabbitMQ-shovel]}
].

A full example of the shovel configuration is:

{rabbit_shovel,
  [{shovels,
    [{my_first_shovel,
      [{sources,      [{brokers,
                          ["amqp://fred:secret@.../my_vhost",
                           "amqp://john:secret@.../my_vhost"
                          ]},
                       {declarations,
                          ['queue.declare',
                           {'queue.bind',
                                  [{exchange, <<"my_exchange">>},
                                   {queue,    <<>>}]}
                          ]}]},
       {destinations, [{broker, "amqp://"},
                       {declarations,
                          [{'exchange.declare',
                                  [{exchange, <<"my_exchange">>},
                                   {type, <<"direct">>},
                                   durable]}
                          ]}]},
       {queue, <<>>},
       {qos, 10},
       {auto_ack, false},
       {tx_size, 0},
       {delivery_mode, keep},
       {publish_fields, [{exchange, <<"my_exchange">>},
                         {routing_key, <<"from_shovel">>}]},
       {reconnect, 5}
      ]}
     ]
   }]
}

Firstly, all shovels are named. Here we have one shovel, called
'my_first_shovel'. We can have multiple shovels if you wish. Every
shovel must have all sub-fields specified: sources, destinations, qos,
auto_ack, delivery_mode, publish_fields, reconnect.

Sources and Destinations
------------------------

Sources and destinations specify respectively where messages are
fetched from and delivered too. One of 'broker' and 'brokers' must be
specified, and 'broker' is simply shorthand for when only one broker
needs specifying. Using 'brokers' allows a list of brokers to be
specified: whenever the connection to a broker is lost, another one is
chosen at random from the list and a connection attempt is made to
that. The syntax for broker URIs is:

amqp://username:password <at> host:port/vhost

If username or password are omitted, the default values of guest and
guest are used. If the vhost is omitted, the default value of / is
used. If the host is omitted, then the plugin uses the "direct"
connection internally rather than a network connection: this means it
connects to the RabbitMQ-server node on which it is running without
going through the network stack. This is much more efficient. If port
is omitted then the default value is used (5672 or 5671 if SSL is
used).

SSL is implemented, for which additional parameters are needed:

amqps://username:password <at> host:port/vhost?cacertfile=/path/to/cacert.pem&certfile=/path/to/certfile.pem&keyfile=/path/to/keyfile.pem&verify=verifyOption&fail_if_no_peer_cert=failOption

All five parameters (3 paths: cacertfile, certfile and keyfile; 2
options: verify, fail_if_no_peer_cert) must be specified. See the SSL
guide at http://www.rabbitmq.com/ssl.html#configure-erlang for details
of SSL in RabbitMQ in general and specifically for the Erlang client
(on which the shovel is built).

Note that SSL cannot be used with the direct connection (i.e. a host
must be specified when using SSL), and that it is preferable to use
the non-SSL direct connection to using SSL to connect to the same node
that's running the shovel.

Resource Declarations
---------------------

Both sources and destinations can have an optional 'declarations'
clause. The value of this is a key, consisting of AMQP Methods. If
default values are sufficient, then the method name alone can be
specified - e.g. 'queue.declare'. If parameters need to be set then
the method should be given as a tuple, with the right hand side a
proplist specifying which fields need altering from their default
values. E.g:
    {'exchange.declare',[{exchange, <<"my_exchange">>},
                         {type, <<"direct">>},
                         durable]},

One very useful feature here is the Most-Recently-Declared-Queue
feature, in which RabbitMQ remembers the name of the most recently
declared queue. This means that you can declare a private queue, and
then bind it to exchanges without ever needing to know its name.

queue :: binary
---------------

This feature specifies the name of the queue on the source brokers to
consume from. This queue must exist. Use the resource declarations to
create the queue (or ensure it exists) first. Note again that the
Most-Recently-Declared-Queue feature can be used here, thus an
anonymous queue can be used.

qos :: non-negative-integer
---------------------------

The shovel consumes from a queue. The QoS controls how many messages
are sent to the shovel in advance of the message the shovel is
currently processing.

auto_ack :: boolean
-------------------

Setting this to 'true' turns on the no_ack flag when subscribing to
the source queue.

tx_size :: non-negative-integer
-------------------------------

When set to 0, transactions are not used. Other values make publishes
transactional, with a commit every N messages. In lieu of the auto-ack
option, when transactions are not used, messages are acknowledged to
the source immediately after every publish. When transactions are
used, acks are only issued to the source on receipt of the commit-ok
message from the destination. This can thus be used to guarantee that
messages are only acknowledged (and thus forgotten about by the source
broker) when they are guaranteed to have been received by the
destination broker.

delivery_mode :: 'keep' | 0 | 2
-------------------------------

This affects the delivery_mode field when publishing to the
destination. A value of 'keep' means that the same delivery_mode
should be used as when the message was originally published to the
source broker. 0 and 2 override the original setting.

publish_fields
--------------

This is a list of tuples which override fields in the publish method
when publishing to the destination. This can be used to direct
messages to a particular exchange on the destination, for example, or
change the routing key. By default, the routing key of the message as
it is received by the shovel is passed through, but this can be
overridden as necessary.

reconnect :: non-negative-integer
---------------------------------

When an error occurs, the shovel will disconnect from both the source
and destination broker immediately. This will force uncommitted
transactions at the destination to be rolled back, and delivered but
unacknowledged messages from the source to be requeued. The shovel
will then try connecting again. If this is unsuccessful, then it's not
a good idea for the shovel to very quickly and repeatedly try to
reconnect. The value specified here is the number of seconds to wait
between each connection attempt.

Note that if set to 0, the shovel will never try to reconnect: it'll
stop after the first error.

Obtaining shovel statuses
-------------------------

From the broker Erlang prompt, call
rabbit_shovel_status:status(). This will return a list, with one row
for each configured shovel. Each row has three fields: the shovel
name, the shovel status, and the timestamp (a local calendar time of
{{YYYY,MM,DD},{HH,MM,SS}}). There are 3 possible statuses:

'starting': The shovel is starting up, connecting and creating
resources.

'running': The shovel is up and running, shovelling messages.

{'terminated', Reason}: Something's gone wrong. The Reason should give
a further indication of where the fault lies.

John Apps | 1 Feb 12:41 2010
Picon

Re: rabbitmqctl often hangs at the end of a query, e.g., list_queues, for a good 60 seconds or more (Windows 7)

Matthias,

  I tried this out on another Windows 7 X64 machine I have and was unable to reproduce the behaviour there. Rather than have this entry hang around, please disregard for now.
When I have time I will pursue the matter and let you know what the cause was.

Thank you for the support, John

On Thu, Jan 28, 2010 at 00:35, Matthias Radestock <matthias-F+y6HrC/rsCsTnJN9+BGXg@public.gmane.org> wrote:
John,

John Apps wrote:
Whilst not being a bug per se, it is annoying behaviour when rabbitmqctl 'sits there' for quite some time.

How long is 'quite some time'?

And what rabbitmqctl command were you trying to run? list_queues (as per the subject line)? If so, how many queues did you have at the time?

And when you say (in the subject line) that rabbitmqctl hangs *at the end* of a query, do you mean it displays the results and *then* hangs?

Interestingly enough, it does not accept <CTRL>C either.

I'll have to find a Windows machine to try to reproduce that.

Are you running the standard rabbitmqctl.bat script? In the other thread you posted some rabbitmqctl output which contained progress log information, which is not something we see in testing and for which the most plausible explanation is that some additional options have been passed to the erl executable.

This wait is only encountered when activity is going on with
RabbitMQ, no matter what it is.

So this only happens when rabbit is really busy?

Rabbit does prioritise handling of rabbitmqctl processing over most other tasks, but when the server is under heavy load it can still take some time to handle these requests. There is not much we can do about that really, since we wouldn't want to stop message processing altogether.


Regards,

Matthias.



--
---
John Apps
(49) 171 869 1813
Sent from Prien am Chiemsee, Bavaria, Germany
<div>
<p>Matthias,</p>
<div>&nbsp;&nbsp;I tried this out on another Windows 7 X64 machine I have and was unable to reproduce the behaviour there. Rather than have this entry hang around, please disregard for now.</div>
<div>When I have time I will pursue the matter and let you know what the cause was.<br><br>
</div>
<div>Thank you for the support, John</div>
<div>
<br><div class="gmail_quote">On Thu, Jan 28, 2010 at 00:35, Matthias Radestock <span dir="ltr">&lt;<a href="mailto:matthias@...">matthias@...</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">John,<br><br>
John Apps wrote:<br><blockquote class="gmail_quote">
Whilst not being a bug per se, it is annoying behaviour when rabbitmqctl 'sits there' for quite some time.<br>
</blockquote>
<br>
How long is 'quite some time'?<br><br>
And what rabbitmqctl command were you trying to run? list_queues (as per the subject line)? If so, how many queues did you have at the time?<br><br>
And when you say (in the subject line) that rabbitmqctl hangs *at the end* of a query, do you mean it displays the results and *then* hangs?<br><br><blockquote class="gmail_quote">
Interestingly enough, it does not accept &lt;CTRL&gt;C either.<br>
</blockquote>
<br>
I'll have to find a Windows machine to try to reproduce that.<br><br>
Are you running the standard rabbitmqctl.bat script? In the other thread you posted some rabbitmqctl output which contained progress log information, which is not something we see in testing and for which the most plausible explanation is that some additional options have been passed to the erl executable.<br><br><blockquote class="gmail_quote">
This wait is only encountered when activity is going on with<br>
RabbitMQ, no matter what it is.<br>
</blockquote>
<br>
So this only happens when rabbit is really busy?<br><br>
Rabbit does prioritise handling of rabbitmqctl processing over most other tasks, but when the server is under heavy load it can still take some time to handle these requests. There is not much we can do about that really, since we wouldn't want to stop message processing altogether.<br><br><br>
Regards,<br>
<br>
Matthias.<br>
</blockquote>
</div>
<br><br clear="all"><br>-- <br>---<br>John Apps<br>(49) 171 869 1813<br>Sent from Prien am Chiemsee, Bavaria, Germany
</div>
</div>
Holger Hoffstaette | 1 Feb 14:23 2010

Re: Poor performance using a single RabbitMQ connection on high-latency networks


Followup reply from Vladimir. Apparently the problem is something else.
Any of the Rabbit devs got an idea?

Holger

-------- Original Message --------
Subject: Re: [rabbitmq-discuss] Poor performance using a single RabbitMQ
connection on high-latency networks
Date: 	Sun, 31 Jan 2010 13:52:27 +0300
From: 	Vladimir Balandin <vbalandin@...>
To: 	Holger Hoffstaette <holger.hoffstaette@...>

Holger, thank you a lot for reply.

We are tried to use different send/receive buffers (from 8kb to 16 Mb),
but it don't improve performance. We saw extremely large throughput
until buffer is filled (when use 1 Mb buffer we can send about 1 Mb data
with no delay, when use 256 kB buffer we can send about 256 kB, etc) but
then we see poor performance again - no more than 20 kB/s. We have
tested with Windows XP and Windows Vista.

Also we have created a simple client-server application which simply
writes data to stream and measure performance: we have seen throughput
is about 350 kB/s (it is about 40% of network usage, what is a pretty
good for us, because we can use three or four connections on this
network to arrive at least 90% of network usage). We don't understand
why RabbitMQ have poor performance using single TCP connection on our case.

Matthew Sackman | 1 Feb 14:33 2010
Picon

Re: Poor performance using a single RabbitMQ connection on high-latency networks

Thanks for forwarding that to the list, Holger.

Vladimir, we have sometimes seen very strange things with Windows and
some firewall products doing deep inspection on traffic that's going to
the internet at large - XML based content often triggers this. Could you
make sure all firewalls are off, or try using SSL?

You don't state which client you're using. If you can cut the code down
to a small example which demonstrates this problem we can certainly take
a look at that.

We know of several clients who are publishing to Rabbit at high speed
over a WAN - there is nothing inherently tricky here.

On Mon, Feb 01, 2010 at 02:23:37PM +0100, Holger Hoffstaette wrote:
> Also we have created a simple client-server application which simply
> writes data to stream and measure performance: we have seen throughput
> is about 350 kB/s (it is about 40% of network usage, what is a pretty
> good for us, because we can use three or four connections on this
> network to arrive at least 90% of network usage). We don't understand
> why RabbitMQ have poor performance using single TCP connection on our case.

I have a suspicion this is a firewall issue - with your simple
streaming, do I assume the data is all nulls, or random? And with your
Rabbit tests, is the data XML?

Matthew

Matthew Sackman | 1 Feb 14:43 2010
Picon

Re: PubSub - all bindings have to be in memory?

Hi Ben,

On Mon, Feb 01, 2010 at 07:58:52AM +0200, Ben Browitt wrote:
> Do I have to keep all bindings in memory or is there a mode where rabbitmq
> can read binding keys from disk when needed?

I'm afraid bindings are all kept in RAM. The binding record is quite
small (dependent on the size of the exchange name, binding key, queue
name and any args - minimising these will help reduce memory foot print)
so you should be able to fit quite a lot of them in RAM:

> erts_debug:flat_size(#binding{exchange_name = <<"my_exchange">>, queue_name = <<"my_queue">>,
key = <<"my_key">>, args = []}).
24

So just 24 bytes for that record, though obviously with padding and
longer field values, this will grow. Fixed size values here would help -
if for example you're able to use md5sums to get the exchange name and
queue name, and only use fanout or direct exchanges, and thus have an
fixed sized hash for the binding key too, then that'll make estimations
easier.

If you can estimate given, say, twice your expected number of visitors
and twice the expected number of articles they subscribe to how many
bindings that would result in, then you may be able to satisfy yourself
whether or not this is going to be a problem.

Whilst the new persister is able to page messages out to disk, we have
no plans just at the moment to be able to do the same for bindings.

Please let us know how you get on.

Matthew

Vladimir Balandin | 1 Feb 14:47 2010

Re: Poor performance using a single RabbitMQ connection on high-latency networks

Here is a code example:

        final AMQP.BasicProperties props = MessageProperties.BASIC;
        final byte[] body = new byte[64000];

        long startTime = System.nanoTime();
        for (int i = 0; i < MESSAGE_COUNT; i++) {
            try {
               channel.basicPublish("",queueName, false, false, props, body);
            } catch (Exception cause) {
               cause.printStackTrace();
               return null;
            }
         }
        long endTime = System.nanoTime();

On Mon, Feb 1, 2010 at 4:33 PM, Matthew Sackman <matthew-F+y6HrC/rsCsTnJN9+BGXg@public.gmane.org> wrote:
Thanks for forwarding that to the list, Holger.

Vladimir, we have sometimes seen very strange things with Windows and
some firewall products doing deep inspection on traffic that's going to
the internet at large - XML based content often triggers this. Could you
make sure all firewalls are off, or try using SSL?

You don't state which client you're using. If you can cut the code down
to a small example which demonstrates this problem we can certainly take
a look at that.

We know of several clients who are publishing to Rabbit at high speed
over a WAN - there is nothing inherently tricky here.

On Mon, Feb 01, 2010 at 02:23:37PM +0100, Holger Hoffstaette wrote:
> Also we have created a simple client-server application which simply
> writes data to stream and measure performance: we have seen throughput
> is about 350 kB/s (it is about 40% of network usage, what is a pretty
> good for us, because we can use three or four connections on this
> network to arrive at least 90% of network usage). We don't understand
> why RabbitMQ have poor performance using single TCP connection on our case.

I have a suspicion this is a firewall issue - with your simple
streaming, do I assume the data is all nulls, or random? And with your
Rabbit tests, is the data XML?

Matthew

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss <at> lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



--
Vladimir Balandin
Grid Dynamics
<div>
<p>Here is a code example:<br><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; final AMQP.BasicProperties props = MessageProperties.BASIC;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; final byte[] body = new byte[64000];<br><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; long startTime = System.nanoTime();<br>
&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for (int i = 0; i &lt; MESSAGE_COUNT; i++) {<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; try {<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; channel.basicPublish("",queueName, false, false, props, body);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; } catch (Exception cause) {<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; cause.printStackTrace();<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return null;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; long endTime = System.nanoTime();<br><br></p>
<div class="gmail_quote">On Mon, Feb 1, 2010 at 4:33 PM, Matthew Sackman <span dir="ltr">&lt;<a href="mailto:matthew@...">matthew@...</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">Thanks for forwarding that to the list, Holger.<br><br>
Vladimir, we have sometimes seen very strange things with Windows and<br>
some firewall products doing deep inspection on traffic that's going to<br>
the internet at large - XML based content often triggers this. Could you<br>
make sure all firewalls are off, or try using SSL?<br><br>
You don't state which client you're using. If you can cut the code down<br>
to a small example which demonstrates this problem we can certainly take<br>
a look at that.<br><br>
We know of several clients who are publishing to Rabbit at high speed<br>
over a WAN - there is nothing inherently tricky here.<br><div class="im">
<br>
On Mon, Feb 01, 2010 at 02:23:37PM +0100, Holger Hoffstaette wrote:<br>
&gt; Also we have created a simple client-server application which simply<br>
&gt; writes data to stream and measure performance: we have seen throughput<br>
&gt; is about 350 kB/s (it is about 40% of network usage, what is a pretty<br>
&gt; good for us, because we can use three or four connections on this<br>
&gt; network to arrive at least 90% of network usage). We don't understand<br>
&gt; why RabbitMQ have poor performance using single TCP connection on our case.<br><br>
</div>I have a suspicion this is a firewall issue - with your simple<br>
streaming, do I assume the data is all nulls, or random? And with your<br>
Rabbit tests, is the data XML?<br><br>
Matthew<br><div>
<div></div>
<div class="h5">
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss <at> lists.rabbitmq.com</a><br><a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div>
</div>
</blockquote>
</div>
<br><br clear="all"><br>-- <br>Vladimir Balandin<br>Grid Dynamics<br>
</div>

Gmane