headers
sporetsky | 2 May 2005 18:47

RE: IGP Convergence

Thanks Diego and others at Ixia!  I will incorporate these changes into what is intended to be the final version of these drafts.  I have previously received comments from a couple of reviewers at Spirent, so leading test equipment vendors have provided input into this work item.
 
BMWG-ers,  If there are any further comments then please raise them now.
 
Thanks,
Scott
-----Original Message-----
From: Diego Dugatkin [mailto:diego <at> ixiacom.com]
Sent: Friday, April 29, 2005 8:23 PM
To: bmwg <at> ietf.org; sporetsky <at> quarrytech.com
Cc: Al Morton; Kevin Dubray
Subject: IGP Convergence

Please find the following feedback on the IGP Convergence drafts:

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-05.txt

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-05.txt

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-05.txt

 

Overall, we find these drafts solid and have only a few small comments:

 

1.                Methodology doc, page 6: it was a bit confusing if they were talking about reporting the timer values of the DUT or the emulated devices.  One can assume it's the DUT but a word or two could make this clearer.

 

2.                Methodology doc, page 6-8: in the test procedure for 4.1.1, steps 4 and 7 mention SONET specifically.  Unless these test cases are only for SONET they should make this a more generic statement for removing the layer-1 connection (e.g. disconnect an Ethernet cable).  The same issue appears in the other test cases.

 

3.                Methodology doc, page 10:  test case 4.4 is missing step 5.

 

(I have also sent other comments directly to Scott, discussing precision of the convergence time measurements based on the proposed methodology, and a potential modification to that.)

 

 

Regards,

 

Dr. Diego Dugatkin

Principal Technologist

Ixia

26601 West Agoura Road

Calabasas, CA91302 - U.S.A.

www.ixiacom.com

Tel. direct:  +1 (818) 444-3124

 

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 0 comments |
headers
Kevin Dubray | 5 May 2005 21:59
Favicon

Which is correct?

Hello BMWG folk:

I pulled the following post from public a NSP support list.  While
it could generate lots of discussion points in a variety of
dimensions :-), I think I'd like to use the datum to reinforce
a single point.

We, as purveyors of fine benchmarking methodology, should
aspire to derive those methodologies that promote consistent
measurement across multiple measurement implementations.  If
we don't, then the notion of "benchmark" becomes diminished.

Of course, a single data point doesn't constitute a trend, but
presuming a uniform test environment, I thought it an
interesting illustration.

-Kevin

-------------------------------------------------------------
To: juniper-nsp at puck.nether.net
Subject: [j-nsp] Latency FastEthernet & Testing with Anritsu, Agilent,
SmartBits

Hello everybody! I tested M20 (RFC 2544).
The results:
(Juniper M20, 4 Port FastEthernet, Average Latency, in microseconds)

128 Byte
------------------------------------------------------
NetStat   26 mcs
Anritsu MD1230B  20 mcs
Agilent N2X   48 (!!!) mcs (Min=32, Max=71)
Spirent SmartBits 600B  N/A (Min=26, Max=51)
------------------------------------------------------

1024 Byte
------------------------------------------------------
NetStat   100 mcs
Anritsu MD1230B  28 mcs
Agilent N2X   177 (!!!!) mcs (Min=106, Max=293)
Spirent SmartBits 600B N/A (Min=99, Max=289)
------------------------------------------------------

Which are correct?
Permalink | Reply | 1 comment |
headers
Jim McQuaid | 5 May 2005 22:15

RE: Which is correct?

One  underlying issue, of course, is a) do the suppliers of these
measurement tools claim to be RFC2544 compliant and b) are they?

BMWG has no control over these issues, more or less, but I think that
they're the right questions.

That said, when you get down to deltas in the dozens of microsecond range
you're in a different zone.

cheers,

Jim

Jim McQuaid
Product Manager  |  NetIQ Corporation

Direct 919.767.0296  |  Fax 919.767.0201  |  www.netiq.com
4825 Creekstone Drive  |  Suite 400  |  Durham, NC, 27703

-----Original Message-----
From: bmwg-bounces <at> ietf.org [mailto:bmwg-bounces <at> ietf.org] On Behalf Of
Kevin Dubray
Sent: Thursday, May 05, 2005 4:00 PM
To: bmwg <at> ietf.org
Subject: [bmwg] Which is correct?

Hello BMWG folk:

I pulled the following post from public a NSP support list.  While
it could generate lots of discussion points in a variety of
dimensions :-), I think I'd like to use the datum to reinforce
a single point.

We, as purveyors of fine benchmarking methodology, should
aspire to derive those methodologies that promote consistent
measurement across multiple measurement implementations.  If
we don't, then the notion of "benchmark" becomes diminished.

Of course, a single data point doesn't constitute a trend, but
presuming a uniform test environment, I thought it an
interesting illustration.

-Kevin

-------------------------------------------------------------
To: juniper-nsp at puck.nether.net
Subject: [j-nsp] Latency FastEthernet & Testing with Anritsu, Agilent,
SmartBits

Hello everybody! I tested M20 (RFC 2544).
The results:
(Juniper M20, 4 Port FastEthernet, Average Latency, in microseconds)

128 Byte
------------------------------------------------------
NetStat   26 mcs
Anritsu MD1230B  20 mcs
Agilent N2X   48 (!!!) mcs (Min=32, Max=71)
Spirent SmartBits 600B  N/A (Min=26, Max=51)
------------------------------------------------------

1024 Byte
------------------------------------------------------
NetStat   100 mcs
Anritsu MD1230B  28 mcs
Agilent N2X   177 (!!!!) mcs (Min=106, Max=293)
Spirent SmartBits 600B N/A (Min=99, Max=289)
------------------------------------------------------

Which are correct?

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 0 comments |
headers
Brough Davis | 5 May 2005 23:14
Picon

RE: Which is correct?

I'm curious how many tests were run by each vendor and if the averages were
taken (or just the min).  This is frustrating from the standpoint that
existing benchmark tools are not consistent with the statistical methods
even though they are all performing 2544 tests (which I'm pretty sure they
all are).  Notice the Netstat and Anritsu values are similar to the min
values for the other two systems.  

Because these tests are all in the micro-seconds realm are there any
reasonable number of tests and averages that should be performed...as well
as acceptable standard deviations?  My apologies if these are already
spelled out in the RFC.

Brough 

-----Original Message-----
From: bmwg-bounces <at> ietf.org [mailto:bmwg-bounces <at> ietf.org] On Behalf Of
Kevin Dubray
Sent: Thursday, May 05, 2005 4:00 PM
To: bmwg <at> ietf.org
Subject: [bmwg] Which is correct?

Hello BMWG folk:

I pulled the following post from public a NSP support list.  While it could
generate lots of discussion points in a variety of dimensions :-), I think
I'd like to use the datum to reinforce a single point.

We, as purveyors of fine benchmarking methodology, should aspire to derive
those methodologies that promote consistent measurement across multiple
measurement implementations.  If we don't, then the notion of "benchmark"
becomes diminished.

Of course, a single data point doesn't constitute a trend, but presuming a
uniform test environment, I thought it an interesting illustration.

-Kevin

-------------------------------------------------------------
To: juniper-nsp at puck.nether.net
Subject: [j-nsp] Latency FastEthernet & Testing with Anritsu, Agilent,
SmartBits

Hello everybody! I tested M20 (RFC 2544).
The results:
(Juniper M20, 4 Port FastEthernet, Average Latency, in microseconds)

128 Byte
------------------------------------------------------
NetStat   26 mcs
Anritsu MD1230B  20 mcs
Agilent N2X   48 (!!!) mcs (Min=32, Max=71)
Spirent SmartBits 600B  N/A (Min=26, Max=51)
------------------------------------------------------

1024 Byte
------------------------------------------------------
NetStat   100 mcs
Anritsu MD1230B  28 mcs
Agilent N2X   177 (!!!!) mcs (Min=106, Max=293)
Spirent SmartBits 600B N/A (Min=99, Max=289)
------------------------------------------------------

Which are correct?

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 0 comments |
headers
John Dawson | 5 May 2005 22:33
Favicon

RE: Which is correct?

About 7 years ago, I believe it was at IETF-42, I had presented to the BMWG a white-paper about Latency measurements and an effect, that at the time, I called transmit synchronization. 

Pretty much the white-paper said that using 2544 as the basis for creating traffic flows, you are going to get nothing but junk measurements when it comes to latency.

Since then there has been no substantial body of work relative to Latency measurements.

In my estimation, RFC2544 isn't going to cut it when it comes to latency.  You would be better off throwing the entire Methodology out the window and starting from scratch.  The best approach, that I have been able to dream up, is that you actually want every port on your test system using a different gap setting -- based upon prime number values -- such that ports can never synchronize to one another and generate artificial delays that don't really exist.

No surprise on my part that 3 different vendors yield different latency results.  The more vendors you add, the more different results you can expect.

The entire thought process behind latency has to change.  But then, that was well received 8 years ago, so I doubt it will be well received now.  I think I still have my original research should someone care to review this topic again.

John D.

-----Original Message-----
From: bmwg-bounces <at> ietf.org [mailto:bmwg-bounces <at> ietf.org] On Behalf Of Jim McQuaid
Sent: Thursday, May 05, 2005 1:16 PM
To: 'Kevin Dubray'; 'bmwg <at> ietf.org'
Subject: RE: [bmwg] Which is correct?


One  underlying issue, of course, is a) do the suppliers of these
measurement tools claim to be RFC2544 compliant and b) are they?

BMWG has no control over these issues, more or less, but I think that
they're the right questions.

That said, when you get down to deltas in the dozens of microsecond range
you're in a different zone.

cheers,

Jim




Jim McQuaid
Product Manager  |  NetIQ Corporation

Direct 919.767.0296  |  Fax 919.767.0201  |  www.netiq.com
4825 Creekstone Drive  |  Suite 400  |  Durham, NC, 27703




-----Original Message-----
From: bmwg-bounces <at> ietf.org [mailto:bmwg-bounces <at> ietf.org] On Behalf Of
Kevin Dubray
Sent: Thursday, May 05, 2005 4:00 PM
To: bmwg <at> ietf.org
Subject: [bmwg] Which is correct?


Hello BMWG folk:

I pulled the following post from public a NSP support list.  While
it could generate lots of discussion points in a variety of
dimensions :-), I think I'd like to use the datum to reinforce
a single point.

We, as purveyors of fine benchmarking methodology, should
aspire to derive those methodologies that promote consistent
measurement across multiple measurement implementations.  If
we don't, then the notion of "benchmark" becomes diminished.

Of course, a single data point doesn't constitute a trend, but
presuming a uniform test environment, I thought it an
interesting illustration.


-Kevin

-------------------------------------------------------------
To: juniper-nsp at puck.nether.net
Subject: [j-nsp] Latency FastEthernet & Testing with Anritsu, Agilent,
SmartBits

Hello everybody! I tested M20 (RFC 2544).
The results:
(Juniper M20, 4 Port FastEthernet, Average Latency, in microseconds)

128 Byte
------------------------------------------------------
NetStat   26 mcs
Anritsu MD1230B  20 mcs
Agilent N2X   48 (!!!) mcs (Min=32, Max=71)
Spirent SmartBits 600B  N/A (Min=26, Max=51)
------------------------------------------------------

1024 Byte
------------------------------------------------------
NetStat   100 mcs
Anritsu MD1230B  28 mcs
Agilent N2X   177 (!!!!) mcs (Min=106, Max=293)
Spirent SmartBits 600B N/A (Min=99, Max=289)
------------------------------------------------------

Which are correct?

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg 

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 1 comment |
headers
David Newman | 6 May 2005 00:00

RE: Which is correct?

On Thu, 5 May 2005, John Dawson wrote:

> In my estimation, RFC2544 isn't going to cut it when it comes to latency.
> You would be better off throwing the entire Methodology out the window and
> starting from scratch.

We already have headed well down that road. For example, 2544 s calls for 
a backbone traffic pattern in which all transmitters simultaneously target 
one receive port.

How many testers actually do that? In my experience, it is far more common 
to measure "RFC 2544 throughput" using RFC 2889 mesh patterns, which 
deliberately avoid presenting two or more packets to the same destination 
at the same instant.

Of course, the latency measurements from these two patterns will be very 
different.

John's ideas about synchronization are worth reconsidering. I'd also raise 
two more concerns:

1. How do we ensure that the test instrument, cabling, and other factors 
aren't themselves contributing to the latency measurement? I have seen 
this referred to as "intrinsic latency." I don't think it should be 
considered instrinsic; this is extra stuff that needs to be characterized 
and compensated for. It should NOT be charged to the DUT/SUT.

2. This has been brought up before, but is it appropriate to measure 
latency only at the throughput oload, as suggested in RFC 2544?

Reduced oload is certainly one way for vendors to claim "better" results. 
Whether it's appropriate to do so is a whole other question.

dn
Permalink | Reply | 0 comments |
headers
Dawit Birhanu (dawit | 6 May 2005 02:18
Picon
Favicon

RE: Which is correct?


Measuring latency at the throughput oload is actually the main reasons
why you see so many measurement discrepancies. There is always some
deviation in data rate (100 PPM or 10 PPM, depending on the spec). If
you measure latency at line rate, which is the throughput for most
routers/switches especially at large packet size, the slight deviation
in data rate translates into significant latency variations. You can
even get different latency measurement results from the same test
equipment vendor because of this problem.  

-dawit

> -----Original Message-----
> From: bmwg-bounces <at> ietf.org [mailto:bmwg-bounces <at> ietf.org] On 
> Behalf Of David Newman
> Sent: Thursday, May 05, 2005 3:00 PM
> To: bmwg <at> ietf.org
> Subject: RE: [bmwg] Which is correct?
> 
> On Thu, 5 May 2005, John Dawson wrote:
> 
> > In my estimation, RFC2544 isn't going to cut it when it 
> comes to latency.
> > You would be better off throwing the entire Methodology out 
> the window 
> > and starting from scratch.
> 
> We already have headed well down that road. For example, 2544 
> s calls for a backbone traffic pattern in which all 
> transmitters simultaneously target one receive port.
> 
> How many testers actually do that? In my experience, it is 
> far more common to measure "RFC 2544 throughput" using RFC 
> 2889 mesh patterns, which deliberately avoid presenting two 
> or more packets to the same destination at the same instant.
> 
> Of course, the latency measurements from these two patterns 
> will be very different.
> 
> John's ideas about synchronization are worth reconsidering. 
> I'd also raise two more concerns:
> 
> 1. How do we ensure that the test instrument, cabling, and 
> other factors aren't themselves contributing to the latency 
> measurement? I have seen this referred to as "intrinsic 
> latency." I don't think it should be considered instrinsic; 
> this is extra stuff that needs to be characterized and 
> compensated for. It should NOT be charged to the DUT/SUT.
> 
> 2. This has been brought up before, but is it appropriate to 
> measure latency only at the throughput oload, as suggested in 
> RFC 2544?
> 
> Reduced oload is certainly one way for vendors to claim 
> "better" results. 
> Whether it's appropriate to do so is a whole other question.
> 
> dn
> 
> _______________________________________________
> bmwg mailing list
> bmwg <at> ietf.org
> https://www1.ietf.org/mailman/listinfo/bmwg
>
Permalink | Reply | 3 comments |
headers
sporetsky | 28 May 2005 13:49

RE: IGP Convergence

Diego and BMWG-ers,
 
Excellent comments below. 

1.                Methodology doc, page 6: it was a bit confusing if they were talking about reporting the timer values of the DUT or the emulated devices.  One can assume it's the DUT but a word or two could make this clearer.

 

2.                Methodology doc, page 6-8: in the test procedure for 4.1.1, steps 4 and 7 mention SONET specifically.  Unless these test cases are only for SONET they should make this a more generic statement for removing the layer-1 connection (e.g. disconnect an Ethernet cable).  The same issue appears in the other test cases.

 

3.                Methodology doc, page 10:  test case 4.4 is missing step 5.

 
In addition,  the following notes were made in the IETF-62 meeting minutes:
 
(methodology)

- There is an IESG mandate to handle IPv6 on par with IPv4

- There should be a test case with a smaller number of routes (to vet config)

- Multiple ingress interfaces must share the Forwarding Capacity,

otherwise congestion will result

- Establish the Maximum Offered Load with a test up-front to set the baseline

To address these I propose the following changes

I. Add two references:

      [5] Bradner, S., "Benchmarking Terminology for Network Interconnection
            Devices", RFC 1242, IETF, July 1991.

      [6] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
            Network Interconnect Devices", RFC 2544, IETF, March 1999.
  

II. Modify paragraphs 3.2.6, 3.2.7, and 3.3 to read as follows:

  3.2.6 Interface Types
   All test cases in this methodology document may be executed with
   any interface type.  All interfaces MUST be the same media and
   Throughout [5,6] for each test case.  Media and protocols MUST
   be configured for minimum failure detection delay to minimize
   the contribution to the measured Convergence time.  For example,
   configure SONET with  minimum carrier-loss-delay or Bi-directional
   Forwarding Detection (BFD) [7].

   3.2.7 Offered Load
   The offered Load MUST be the Throughput of the device as defined
   in [5] and benchmarked in [6] at a fixed packet size.  The packet
   size is selectable and MUST be recorded.  The Throughput MUST be
   measured at the Preferred Egress Interface and the Next-Best
   Egress Interface.  The duration of offered load MUST be greater
   than the convergence time.  The destination addresses for the
   offered load MUST be distributed such that all routes are matched. 
   This enables Full Convergence [2] to be observed.

  3.3 Reporting Format
   For each test case, it is recommended that the following reporting
   format be completed:
 
        Parameter                                 Units
        ---------                                 -----
    IGP                                       (ISIS or OSPF)
    Interface Type                            (GigE, POS, ATM, etc.)
    Packet Size offered to DUT                bytes
    IGP Routes advertised to DUT              number of IGP routes
    Packet Sampling Interval on Tester        seconds or milliseconds
    IGP Timer Values configured on DUT
         SONET Failure Indication Delay    seconds or milliseconds
            IGP Hello Timer                   seconds or milliseconds
            IGP Dead-Interval                 seconds or milliseconds
            LSA Generation Delay              seconds or milliseconds
            LSA Flood Packet Pacing           seconds or milliseconds
            LSA Retransmission Packet Pacing  seconds or milliseconds
            SPF Delay                         seconds or milliseconds
        Benchmarks
         Rate-Derived Convergence Time     seconds or milliseconds
         Loss-Derived Convergence Time     seconds or milliseconds
         Restoration Convergence Time      seconds or milliseconds
 

III. In all procedures paragraph 2 changes to:

 2. Send offered load at measured Throughput with fixed packet size
           to destinations matching all IGP routes from Tester to DUT on
           Ingress Interface [2].

instead of:

 2. Send traffic at maximum forwarding rate to destinations
           matching all IGP routes from Tester to DUT on Ingress Interface [2].

IV. In all procedures use of 'SONET' replace with 'link'

V. Introduction includes statement: "These methodologies apply to IPv4 and IPv6 traffic
   as well as IPv4 and IPv6 IGPs."

Please send me any comments to these proposed changes.

Scott

-----Original Message-----
From: Diego Dugatkin [mailto:diego <at> ixiacom.com]
Sent: Friday, April 29, 2005 8:23 PM
To: bmwg <at> ietf.org; sporetsky <at> quarrytech.com
Cc: Al Morton; Kevin Dubray
Subject: IGP Convergence

Please find the following feedback on the IGP Convergence drafts:

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-05.txt

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-05.txt

http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-app-05.txt

 

Overall, we find these drafts solid and have only a few small comments:

 

1.                Methodology doc, page 6: it was a bit confusing if they were talking about reporting the timer values of the DUT or the emulated devices.  One can assume it's the DUT but a word or two could make this clearer.

 

2.                Methodology doc, page 6-8: in the test procedure for 4.1.1, steps 4 and 7 mention SONET specifically.  Unless these test cases are only for SONET they should make this a more generic statement for removing the layer-1 connection (e.g. disconnect an Ethernet cable).  The same issue appears in the other test cases.

 

3.                Methodology doc, page 10:  test case 4.4 is missing step 5.

 

(I have also sent other comments directly to Scott, discussing precision of the convergence time measurements based on the proposed methodology, and a potential modification to that.)

 

 

Regards,

 

Dr. Diego Dugatkin

Principal Technologist

Ixia

26601 West Agoura Road

Calabasas, CA91302 - U.S.A.

www.ixiacom.com

Tel. direct:  +1 (818) 444-3124

 

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 1 comment |

Gmane