sctp performance

_______________________________________________
tsvwg mailing list
tsvwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/tsvwg

Re: sctp performance

Salvatore,

Performance tests on a clean network do not exercise SCTP's strong
points.  SCTP was not really intended to outperform TCP on a clean
network with a single interface: it was intended to outperform TCP on a
real network with noise, failures and reconfiguration.  If you look at
even single stream tests under noise SCTP rapidly outperforms TCP in
dirty networks.

The SOCK_SEQPACKET test deviation from theoretical is probably more of a
software glitch in bundling, Nagle'ing and Minshall'ing in the OpenSS7
stack more than a problem with the protocol (that was noted in the
results).

The SOCK_STREAM tests on a single interface did outperform TCP, however,
note that the tests saturated the network and both are running very
close to theoretical maximums.

The interesting thing about the OpenSS7 tests were the multiple
interface tests on SOCK_STREAM, that showed that max throughput was a
multiple of the number of available interfaces.  Once TCP saturates
a single interface it has nowhere else to go.  SCTP can load multiple
interfaces to saturation.

These were throughput tests.  Latency tests show SCTP has comparable
results to TCP (with slight improvements) on clean networks, and
SCTP exhibits significant reduced latencies to TCP under even as little
as 5% noise.  On distrupted networks (50% interface failure on a 1 or
2 second duty cycle) TCP stops and SCTP continues.

There are a lot of good papers out there testing more real environments
as well as NS-2 simulations that demonstrate the benefits of SCTP over
TCP for its original intended application, telephony signalling.  Search
on "SCTP Performance" and you can read for the better part of a day.

--brian

On Thu, 07 Oct 2004, Salvatore Loreto wrote:

> 
>    Hi all,
> 
> 
> 
>    reading the experiments on SCTP performance
> 
> 
> 
>    LKSCTP
> 
>    [1]http://datatag.web.cern.ch/datatag/WP3/sctp/tests.htm
> 
>    OPENSS7
>    [2]http://www.openss7.org/sctp_perf.html
> 
> 
> 
>    If  we  ignore  the  gap  underline between  TCP  and  SCTP in LKSCTP,
>    becose due   to   implementation   problem   ( and  I  think  now  the
>    situation is  better,  isn't  it? does someone have up to date results
>    about?)
> 
>    and use just the result underline on OPENSSS7
> 
> 
> 
>    I've some questions/doubts and I'd like know you opinion about:
> 
> 
> 
>    1)  the  test  SCTP with 1 interface shows a substanzial equivalent to
>    TCP.
> 
>    Is  it  possible SCTP with 1 interface is TCP, or if you want if there
>    is    not HoL block there are not difference between two protocols due
>    to the same congestion algorithm.
> 
> 
> 
>    2)  SCTP  with  2  interface has a better performance just for message
>    size greater than 64-byte.
> 
>    But  SCTP was born for Signaling transport, and signaling messages are
>    yes lager than 64-bye but not very large.
> 
> 
> 
>    when in the experiment is used SOCK_SEQPACKET
> 
>    3)  the  real  start up with 1 or 2 interfaces are both worse than the
>    Theoretical  one.  The  strange  is the reale one with 2 interfaces is
>    worse than the reale one with just one interface.
> 
>    4)  similar  considerations  ( done above ) could be done also for the
>    other phases . So SOCK_SEQPACKET seems not so good.
> 
> 
> 
> 
> 
>    regards
> 
>    sal
> 
> 
> 
> 
>      _________________________________________________________________
> 
>    [3]Yahoo!  Mail  ti  protegge  gratis  dalla  posta indesiderata e dai
>    virus!
> 
> References
> 
>    1. http://datatag.web.cern.ch/datatag/WP3/sctp/tests.htm
>    2. http://www.openss7.org/sctp_perf.html
>    3. http://it.rd.yahoo.com/mail/taglines/*http://it.mail.yahoo.com

> _______________________________________________
> tsvwg mailing list
> tsvwg <at> ietf.org
> https://www1.ietf.org/mailman/listinfo/tsvwg

--

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
bidulock <at> openss7.org    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

Re: sctp performance

Brian/Salvatore:

Just a couple of comments...

Brian F. G. Bidulock wrote:
> Salvatore,
> 
> Performance tests on a clean network do not exercise SCTP's strong
> points.  SCTP was not really intended to outperform TCP on a clean
> network with a single interface: it was intended to outperform TCP on a
> real network with noise, failures and reconfiguration.  If you look at
> even single stream tests under noise SCTP rapidly outperforms TCP in
> dirty networks.
> 
> The SOCK_SEQPACKET test deviation from theoretical is probably more of a
> software glitch in bundling, Nagle'ing and Minshall'ing in the OpenSS7
> stack more than a problem with the protocol (that was noted in the
> results).

I would tend to agree with Brian here.. SOCK_SEQPACKET is really
an implementation shade.. In KAME other than controlling a
few minor things all the same code runs... so it sould make
very little difference....

> 
> The SOCK_STREAM tests on a single interface did outperform TCP, however,
> note that the tests saturated the network and both are running very
> close to theoretical maximums.
> 
> The interesting thing about the OpenSS7 tests were the multiple
> interface tests on SOCK_STREAM, that showed that max throughput was a
> multiple of the number of available interfaces.  Once TCP saturates
> a single interface it has nowhere else to go.  SCTP can load multiple
> interfaces to saturation.
> 
> These were throughput tests.  Latency tests show SCTP has comparable
> results to TCP (with slight improvements) on clean networks, and
> SCTP exhibits significant reduced latencies to TCP under even as little
> as 5% noise.  On distrupted networks (50% interface failure on a 1 or
> 2 second duty cycle) TCP stops and SCTP continues.
> 
> There are a lot of good papers out there testing more real environments
> as well as NS-2 simulations that demonstrate the benefits of SCTP over
> TCP for its original intended application, telephony signalling.  Search
> on "SCTP Performance" and you can read for the better part of a day.
> 

A couple of other things:

a) The CERN tests were done on an OLD OLD OLD version of lk-sctp which
    was NOT performance tuned at all.. I am not even sure if they have
    completed a pass yet of tuning.. I know its on their to-do list.

b) IPERF ... which is used on both sites.. to me.. is NOT a very good
    benchmark. SCTP sends "messages" from one end to the other. TCP sends
    a byte stream from one end to the other. You then take and use IPERF
    to test the number of bytes streaming from one end to the other.
    This puts SCTP at a dis-advantage. Especially considering that MOST
    ALL applications send messages NOT streams of bytes. So if you added
    some additional overhead to TCP.. say framing messages at the app
    layer.. this would slow TCP down I would wager.

    I have always wanted to develop a variable message sender that could
    send a sequence of variable messages and get a reading of messages
    per second for various sizes....

c) CRC32c is extremely expensive (in overall terms of CPU use) on the
    TCP side they generally have an off-load engine they can use to
    not even do the c-sum... so in such a case when you begin to
    get CPU limited SCTP will lag behind TCP... just on the checksum
    basis alone. Bottom line is once you get CRC32c offload SCTP will
    be able to do a bit better :-D

Regards

R

> --brian
> 
> On Thu, 07 Oct 2004, Salvatore Loreto wrote:
> 
> 
>>   Hi all,
>>
>>
>>
>>   reading the experiments on SCTP performance
>>
>>
>>
>>   LKSCTP
>>
>>   [1]http://datatag.web.cern.ch/datatag/WP3/sctp/tests.htm
>>
>>   OPENSS7
>>   [2]http://www.openss7.org/sctp_perf.html
>>
>>
>>
>>   If  we  ignore  the  gap  underline between  TCP  and  SCTP in LKSCTP,
>>   becose due   to   implementation   problem   ( and  I  think  now  the
>>   situation is  better,  isn't  it? does someone have up to date results
>>   about?)
>>
>>   and use just the result underline on OPENSSS7
>>
>>
>>
>>   I've some questions/doubts and I'd like know you opinion about:
>>
>>
>>
>>   1)  the  test  SCTP with 1 interface shows a substanzial equivalent to
>>   TCP.
>>
>>   Is  it  possible SCTP with 1 interface is TCP, or if you want if there
>>   is    not HoL block there are not difference between two protocols due
>>   to the same congestion algorithm.
>>
>>
>>
>>   2)  SCTP  with  2  interface has a better performance just for message
>>   size greater than 64-byte.
>>
>>   But  SCTP was born for Signaling transport, and signaling messages are
>>   yes lager than 64-bye but not very large.
>>
>>
>>
>>   when in the experiment is used SOCK_SEQPACKET
>>
>>   3)  the  real  start up with 1 or 2 interfaces are both worse than the
>>   Theoretical  one.  The  strange  is the reale one with 2 interfaces is
>>   worse than the reale one with just one interface.
>>
>>   4)  similar  considerations  ( done above ) could be done also for the
>>   other phases . So SOCK_SEQPACKET seems not so good.
>>
>>
>>
>>
>>
>>   regards
>>
>>   sal
>>
>>
>>
>>
>>     _________________________________________________________________
>>
>>   [3]Yahoo!  Mail  ti  protegge  gratis  dalla  posta indesiderata e dai
>>   virus!
>>
>>References
>>
>>   1. http://datatag.web.cern.ch/datatag/WP3/sctp/tests.htm
>>   2. http://www.openss7.org/sctp_perf.html
>>   3. http://it.rd.yahoo.com/mail/taglines/*http://it.mail.yahoo.com
> 
> 
>>_______________________________________________
>>tsvwg mailing list
>>tsvwg <at> ietf.org
>>https://www1.ietf.org/mailman/listinfo/tsvwg
> 
> 
> 

--

-- 
Randall Stewart
ITD
803-345-0369 <or> 815-342-5222

Re: sctp performance

_______________________________________________
tsvwg mailing list
tsvwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/tsvwg

Re: sctp performance

Hi,

Just adding another pointer -

Sourabh Ladha had some (throughput) results on the KAME stack on FreeBSD
4.6 (TCP NewReno):
http://www.cis.udel.edu/~amer/PEL/poc/pdf/IPCCC2004-ladha.FTP.over.SCTP.pdf

regards,
jana

On Thu, 7 Oct 2004, Pawel Hadam wrote:

> (I am sorry for my previous, signed e-mail)
>
> Hi
>
>  > a) The CERN tests were done on an OLD OLD OLD version of lk-sctp which
>  >    was NOT performance tuned at all.. I am not even sure if they have
>  >    completed a pass yet of tuning.. I know its on their to-do list.
>
>
> I was working with linux kernel 2.6.7 with lkSCTP 1.0.1 and I obtained
> much better results.
>
>  >    I have always wanted to develop a variable message sender that could
>  >    send a sequence of variable messages and get a reading of messages
>  >    per second for various sizes....
>
>
> Sounds like my own testing apps. I wrote a sender and receiver that
> accept some customization parameters (message size, buffers).
>
> I wrote a raport about my measurements and experiments with lkSCTP on
> 1Gb network, but:
>
> 1) It is now being read by my supervisors.
> 2) It is in french :) - I really am sorry, it was written for a french
> project.
>
> Once the raport finished, I will, I hope, try to publish something in
> this subject.
>
> If anybody wants the french version, I may send him the PDF (there are a
> lot of pictures :))). Any comments are always welcome.
>
> Regards
> Pawel
>

---------------------------------------------------------------
Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
Protocol Engineering Lab  --   CIS   --  University Of Delaware
---------------------------------------------------------------

Re: sctp performance

Janardhan,

Interesting: at 0% probability of loss, single homed KAME SCTP performed
poorer than TCP.  It looks like SCTP implementors still have something to
learn from TCP implementors in some cases.  SCTP wasn't really aimed
exclusively at greater than 3% message loss networks...  Signalling networks
normally have far better than 1ppm BER.

--brian

On Thu, 07 Oct 2004, Janardhan Iyengar wrote:

> Hi,
> 
> Just adding another pointer -
> 
> Sourabh Ladha had some (throughput) results on the KAME stack on FreeBSD
> 4.6 (TCP NewReno):
> http://www.cis.udel.edu/~amer/PEL/poc/pdf/IPCCC2004-ladha.FTP.over.SCTP.pdf
> 
> regards,
> jana
> 
> On Thu, 7 Oct 2004, Pawel Hadam wrote:
> 
> > (I am sorry for my previous, signed e-mail)
> >
> > Hi
> >
> >  > a) The CERN tests were done on an OLD OLD OLD version of lk-sctp which
> >  >    was NOT performance tuned at all.. I am not even sure if they have
> >  >    completed a pass yet of tuning.. I know its on their to-do list.
> >
> >
> > I was working with linux kernel 2.6.7 with lkSCTP 1.0.1 and I obtained
> > much better results.
> >
> >  >    I have always wanted to develop a variable message sender that could
> >  >    send a sequence of variable messages and get a reading of messages
> >  >    per second for various sizes....
> >
> >
> > Sounds like my own testing apps. I wrote a sender and receiver that
> > accept some customization parameters (message size, buffers).
> >
> > I wrote a raport about my measurements and experiments with lkSCTP on
> > 1Gb network, but:
> >
> > 1) It is now being read by my supervisors.
> > 2) It is in french :) - I really am sorry, it was written for a french
> > project.
> >
> > Once the raport finished, I will, I hope, try to publish something in
> > this subject.
> >
> > If anybody wants the french version, I may send him the PDF (there are a
> > lot of pictures :))). Any comments are always welcome.
> >
> > Regards
> > Pawel
> >
> 
> ---------------------------------------------------------------
> Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
> Protocol Engineering Lab  --   CIS   --  University Of Delaware
> ---------------------------------------------------------------

--

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
bidulock <at> openss7.org    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

Re: sctp performance

Brian F. G. Bidulock wrote:
> Janardhan,
> 
> Interesting: at 0% probability of loss, single homed KAME SCTP performed
> poorer than TCP.  It looks like SCTP implementors still have something to
> learn from TCP implementors in some cases.  SCTP wasn't really aimed
> exclusively at greater than 3% message loss networks...  Signalling networks
> normally have far better than 1ppm BER.

Hmm..

The results are interesting.. but I think your 3% figure is a
bit off..

Thats only one of the many graphs present...

Some considerations of these results:

a) Netbsd does NOT apply maxburst in TCP. So this will
    definetly influence small transfers... and if you
    note the larger the transfer the less the difference
    between the graphs.. in fact the 1Meg transfer in the
    later pages looks like possibly SCTP in all cases did
    better instead of in 2 out of 3 cases as it did in all others.

b) Note that at ALL times if you use all of sctp .. multi-streaming
    etc. it does better.. it is only one of the 3 graphs that is slower
    in small transfers.

c) 2002 was about the time I was performance optimizing KAME.. which
    actually needs to have a second pass... so I am not really sure
    which version Sourabh was actually using...

d) Considering the very very little difference between TCP and SCTP
    even in these cases .. thats pretty darn good.. especially when
    you consider that the netbsd stack as been tweaked and tuned for
    at least 10 years... and KAME was about 1 - 1.5 years when Sourabh
    did this work... all in all I think the results quite positive :-D

Note to Salvatore: Yes KAME does have H-S TCP in it.. its a configurable
option when you compile it. And no I have no papers on any results since
my focus as been in other places then that :-0

R

 >
> 
> --brian
> 
> On Thu, 07 Oct 2004, Janardhan Iyengar wrote:
> 
> 
>>Hi,
>>
>>Just adding another pointer -
>>
>>Sourabh Ladha had some (throughput) results on the KAME stack on FreeBSD
>>4.6 (TCP NewReno):
>>http://www.cis.udel.edu/~amer/PEL/poc/pdf/IPCCC2004-ladha.FTP.over.SCTP.pdf
>>
>>regards,
>>jana
>>
>>On Thu, 7 Oct 2004, Pawel Hadam wrote:
>>
>>
>>>(I am sorry for my previous, signed e-mail)
>>>
>>>Hi
>>>
>>> > a) The CERN tests were done on an OLD OLD OLD version of lk-sctp which
>>> >    was NOT performance tuned at all.. I am not even sure if they have
>>> >    completed a pass yet of tuning.. I know its on their to-do list.
>>>
>>>
>>>I was working with linux kernel 2.6.7 with lkSCTP 1.0.1 and I obtained
>>>much better results.
>>>
>>> >    I have always wanted to develop a variable message sender that could
>>> >    send a sequence of variable messages and get a reading of messages
>>> >    per second for various sizes....
>>>
>>>
>>>Sounds like my own testing apps. I wrote a sender and receiver that
>>>accept some customization parameters (message size, buffers).
>>>
>>>I wrote a raport about my measurements and experiments with lkSCTP on
>>>1Gb network, but:
>>>
>>>1) It is now being read by my supervisors.
>>>2) It is in french :) - I really am sorry, it was written for a french
>>>project.
>>>
>>>Once the raport finished, I will, I hope, try to publish something in
>>>this subject.
>>>
>>>If anybody wants the french version, I may send him the PDF (there are a
>>>lot of pictures :))). Any comments are always welcome.
>>>
>>>Regards
>>>Pawel
>>>
>>
>>---------------------------------------------------------------
>>Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
>>Protocol Engineering Lab  --   CIS   --  University Of Delaware
>>---------------------------------------------------------------
> 
> 

--

-- 
Randall Stewart
ITD
803-345-0369 <or> 815-342-5222

Re: sctp performance

Hi

> a) The CERN tests were done on an OLD OLD OLD version of lk-sctp which
>    was NOT performance tuned at all.. I am not even sure if they have
>    completed a pass yet of tuning.. I know its on their to-do list.

I was working with linux kernel 2.6.7 with lkSCTP 1.0.1 and I obtained 
much better results.

>    I have always wanted to develop a variable message sender that could
>    send a sequence of variable messages and get a reading of messages
>    per second for various sizes....

Sounds like my own testing apps. I wrote a sender and receiver that 
accept some customization parameters (message size, buffers).

I wrote a raport about my measurements and experiments with lkSCTP on 
1Gb network, but:

1) It is now being read by my supervisors.
2) It is in french :) - I really am sorry, it was written for a french 
project.

Once the raport finished, I will, I hope, try to publish something in 
this subject.

If anybody wants the french version, I may send him the PDF (there are a 
lot of pictures :))). Any comments are always welcome.

Regards
Pawel

_______________________________________________
tsvwg mailing list
tsvwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/tsvwg

Re: sctp performance

(I am sorry for my previous, signed e-mail)

Hi

 > a) The CERN tests were done on an OLD OLD OLD version of lk-sctp which
 >    was NOT performance tuned at all.. I am not even sure if they have
 >    completed a pass yet of tuning.. I know its on their to-do list.

I was working with linux kernel 2.6.7 with lkSCTP 1.0.1 and I obtained 
much better results.

 >    I have always wanted to develop a variable message sender that could
 >    send a sequence of variable messages and get a reading of messages
 >    per second for various sizes....

Sounds like my own testing apps. I wrote a sender and receiver that 
accept some customization parameters (message size, buffers).

I wrote a raport about my measurements and experiments with lkSCTP on 
1Gb network, but:

1) It is now being read by my supervisors.
2) It is in french :) - I really am sorry, it was written for a french 
project.

Once the raport finished, I will, I hope, try to publish something in 
this subject.

If anybody wants the french version, I may send him the PDF (there are a 
lot of pictures :))). Any comments are always welcome.

Regards
Pawel

Re: sctp performance

Hi

> OPENSS7
> http://www.openss7.org/sctp_perf.html

I am interested in a case with 2 interfaces.
How did they do the load-balancing between 2 interfaces?
It was on SCTP or application (iperf ???) level ???

Does anybody know?

Pawel

_______________________________________________
tsvwg mailing list
tsvwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/tsvwg