sudeep g ggg | 29 Jan 15:03 2016

REG:draft-prajac-bmwg-pmtest-00.txt

Respected All,

I could not able to upload the draft.So I am attaching it by mail.

Regards,
Sudhin

Get your own FREE website, FREE domain & FREE mobile app with Company email.  
Know More >
Network Working Group                              Praveen Ananthasankaran 
Internet Draft                                       Sudhin Jacob
Intended Status: Experimental                        Juniper Networks  
                                                     January 25,2016
													

                                                     


       			draft-prajac-bmwg-pmtest-00.txt                    



Status of This Memo

This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.Internet-Drafts are working documents 
of the Internet Engineering Task Force (IETF), its areas, and its working
groups.Note that other groups may also distribute working documents as
Internet-Drafts.Internet-Drafts are draft documents valid for a maximum
of six months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."The list 
of current Internet-Drafts can be accessed at 
http://www.ietf.org/ietf/1id-abstracts.txt 
The list of Internet-Draft Shadow Directories can be accessed 
at http://www.ietf.org/shadow.html

This Internet-Draft will expire on July 22, 2016.



Copyright Notice

Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved.This document is subject
to BCP 78 and the IETF Trust’s Legal Provisions Relating to IETF
Documents (http://trustee.ietf.org/license-info) in effect on the
date of publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document.























Table of Contents

1.Introduction....................................................3

2.Test Setup......................................................4

  2.1.Test Topology...............................................4
  
  2.2.Network.....................................................4
  
3.Test Procedure..................................................5

4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 12

5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . .15

6. Security Considerations . . . . . . . . . . . . . . . . . . . .15

7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 15

7.1. Normative References . . . . . . . . . . . . . . . . . . . ..15





































1.Introduction

Performance monitoring explained in ITU Y1731  for measuring the
loss and delay over different layer 2 services.This document
defines the methodologies for benchmarking performance loss and
delay over the layer 2 point to point services running on a router.
Hence,the authors have taken the approach of considering PM as a black
box, defining the methodology to benchmark the PM feature using various
testing methodologies














1.1 Terminologies

PM - Performance monitoring

In-profile - CIR termed as green packets.

Out-profile - EIR Yellow/Amber packet.

LMM - Loss Measurement Message

LMR- Loss Measurement Reply

DMM - Delay Measurement Message

DMR - Delay MEasurement Reply

P Router - Provider Router.

PE Router - Provider Edge Router

CE Router - customer Edge  Router

DUT - Device under Test



Frame Lossfar-end = |TxFCb[tc] – TxFCb[tp]| – |RxFCb[tc] – RxFCb[tp]|

Frame Lossnear-end = |TxFCf[tc] – TxFCf[tp]| – |RxFCl[tc] – RxFCl[tp]|

Frame Delay = (RxTimeb – TxTimeStampf) – (TxTimeStampb – RxTimeStampf)










2.1 Test Topology


	 | Traffic Generator
+----------+
|          |
|  PE2     |
|          |
+----------+
    |
    |
+----------+
|          |
|  Core    |
|  router  |
+----------+
   |                     
   |                     
+----------+       
|          |       
|   DUT    |       
|    PE1   |       
+----------+       
     |                  
     |--- Traffic Generator




2.2 Network

The network consists of 3 routers and 2 traffic generator ports.DUT
is acting as one of  PE to CE. The core router is acting as P router.
There is layer 2(point to point) services running from PE1 to PE2.
On the top of that performance monitoring loss and delay measurements
are running.PE1 is acting as DUT.






3.Test Procedure

The test defined to measure the performance of DUT near end and far end
loss, the packets transferred over a period of time. Measuring single class
Colored, colorless loss measurement, multiclass colored and colorless
loss measurement. Average delay, best case and worst case delay.




3.1  Basic Functional testing

3.1.1 Check the near end and far end loss for single class colorless
loss measurement in DUT.

Objective

Check the near end and far end loss over layer 2 point to point service 
running in DUT.The loss will be measured on single class colorless that
means all the in profile and out profile.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 single class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service in
DUT,some in In-profile and some in out profile. Then send this traffic for 
1 min and then stop.

Measurement

The loss measurement must show in DUT 60,000 pps as near end and far end
CIR.There should not be any loss shown in the output.



3.1.2 Check the near end and far end loss for single class colorless
loss measurement by dropping the LMR packet.

Objective

Check the near end and far end loss over layer 2 point to point service
in a DUT.The loss will be measured on single class colorless that means
all the in profile and out profile packets must be counted.

Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 single class loss measurement over this service in DUT. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service,
some in In-profile and some in out profile. Then send this traffic for 
1 min and then stop.During the traffic flow drop 2 or 3 LMR packet. 

Measurement

The loss measurement must show 60,000 pps in DUT as near end and far end
CIR.There should not be any loss shown in the output.



3.1.3 Check the near end and far end loss for single class colorless
loss measurement over a period of time in DUT.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on single class colorless that means all
the in profile and out profile packets must be counted.

Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 single class loss measurement over this service in DUT. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service,
some in In-profile and some in out profile. Then send this traffic for 
1 hr and then stop.

Measurement

The loss measurement must show 36,00,000 pps in DUT as near end and 
far end CIR.There should not be any loss shown in the output.



3.1.4 Check the near end and far end loss for single class colorless
loss measurement by dropping packets.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on single class colorless that means
all the in profile and out profile packets must be counted.

Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 single class loss measurement over this service in DUT. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service,
some in In-profile and some in out profile. Then send this traffic for 
1 min during that time frame drop 100 packets from the core router in
both direction.

Measurement

The loss measurement must show near end and far end loss 100 in DUT.



3.1.5 Check the near end and far end loss for multi class colorless
loss measurement in DUT.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on multi class colorless that means all
the in profile and out profile packets must be counted.

Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 multi class loss measurement over this service in DUT. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service
in different classes.some in In-profile and some in out profile.
Then send this traffic for 1 min and then stop.

Measurement

The loss measurement must show 60,000 pps as near end and far end CIR in DUT
for each class configured in the router.There should not be any loss shown
in the output.



3.1.6 Check the near end and far end loss for multi class colorless
loss measurement by dropping the LMR packet.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on multi class colorless that means all the 
in profile and out profile packets must be counted.

Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 multi class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service
in different classes.Some in In-profile and some in out profile.
Then send this traffic for 1 min and then stop.During the traffic
flow drop 2 or 3 LMR packet. 

Measurement

The loss measurement must show 60,000 pps as near end and far end CIR
in each class configured in DUT.There should not be any loss shown
in the output.




3.1.7 Check the near end and far end loss for multi class colorless
loss measurement over a period of time in DUT.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on multi class colorless that means all
the in profile and out profile packets must be counted.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 multi class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service
in different classes.Some in In-profile and some in out profile.
Then send this traffic for 1 hr and then stop.

Measurement

The loss measurement must show 36,00,000 pps as near end and far end CIR
in each class configured in DUT.There should not be any loss shown in
the output.



3.1.8 Check the near end and far end loss for multi class colorless
loss measurement by dropping packets.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT The loss will be measured on multi class colorless that means all
the in profile and out profile packets must be counted.

Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 multi class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service
in different classes.Some in In-profile and some in out profile.
Then send this traffic for 1 min during that time frame drop 100
packets from the core router in both direction.



Measurement

The loss measurement must show near end and far end loss in DUT,
100 packets for each class



3.1.9 Check the near end and far end loss for single class colored
loss measurement in DUT.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on single class colored that means all
the in profile packets must be counted.Out profile packets must not.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 single class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service,
for example 500 pps as out profile and 500 pps as green from the 
traffic generator. Then send this traffic for 1 min and then stop.


Measurement

The loss measurement must measure only in profile packet.There should
not be any traffic drop.The Loss measurement output taken in DUT will
show as 30,000 pps as near end CIR and far end CIR.


3.1.10 Check the near end and far end loss for single class colored
loss measurement by dropping the LMR packet.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on single class colored that means all
the in profile packets must be counted.Out profile packets must not.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 single class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 500 pps
with in profile and 500 pps with out profile from both traffic 
generator towards this service.Then send this traffic for 
1 min and then stop.During the traffic flow drop 2 or 3 LMR packet. 

Measurement

The loss measurement must measure only in profile packet.There should
not be any traffic drop.Only pps count will show in DUT near end far
end CIR as 30,000 pps. The packets which are out of profile will not
be counted.




3.1.11 Check the near end and far end loss for single class colored
loss measurement over a period of time in DUT.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on single class colored that means all
the in profile packets must be counted.Out profile must not.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 single class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 500 pps
as in profile and 500 pps as out profile from both traffic generator
towards this service,some in In-profile and some in out profile.
Then send this traffic for 1 hr and then stop.


Measurement

The loss measurement must count only in profile packet. The output taken
taken in DUT will show 18,00,000 pps as near end and far end CIR.




3.1.12 Check the near end and far end loss for single class colored
loss measurement by dropping packets.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on single class colored that means all
the in profile packets must be counted.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 single class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with color for in profile from both traffic generator towards this service,
Then send this traffic for 1 min during that time frame drop 100 packets
which are in profile from the core router in both direction.



Measurement

The loss measurement must show near end and far end loss 100 in DUT.




3.1.13 Check the near end and far end loss for multi class colored
loss measurement in DUT.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on multi class colored that means all
the in profile and out profile packets must be counted.



Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 multi class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 500 pps
with in profile and 500 pps with out profile from both traffic generator
towards this service in each class.Then send this traffic for 
1 min and then stop.



Measurement

The loss measurement must show only in profile packets as near end and far
end CIR for each class configured in the router.There should not be any loss
shown in the output.The output must show  30,000 pps in each class
taken in DUT.




3.1.14 Check the near end and far end loss for multi class colored
loss measurement by dropping the LMR packet.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on multi class colored that means all
the in profile packets must be counted.Out profile packets are not counted.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 multi class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 500 pps
as in profile and 500 pps as out profile from both traffic generator
towards this service in different classes.
Then send this traffic for 1 min and then stop.During the traffic
flow drop 2 or 3 LMR packet. 

Measurement

The loss measurement must show only in profile packets as near end and far
end CIR for each class configured in the router.There should not be any loss
shown in the output.The output must show 30,000 pps in each class
taken in DUT.





3.1.15 Check the near end and far end loss for multi class colored
loss measurement over a period of time in DUT.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT The loss will be measured on multi class colored that means all 
the in profile and out profile packets must be counted.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 multi class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 500 pps
as in profile and 500 pps as out profile from both traffic generator
towards this service in different classes.
Then send this traffic for 1 hr and then stop.


Measurement

The loss measurement must show only in profile packets as near end and far
end CIR for each class configured in the router.There should not be any loss
shown in the output.The output must show less 18,00,000 pps as near end and 
far end CIR in each class taken in DUT.




3.1.16 Check the near end and far end loss for multi class colored
loss measurement in DUT by dropping packets.

Objective

Check the near end and far end loss over layer 2 point to point service
in DUT.The loss will be measured on multi class colored that means all
the in profile packets must be counted.Out profile packets are not.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
Y1731 multi class loss measurement over this service. Configure a 
profile to mark the packet in profile and out of profile. Send 1000 pps
with different color from both traffic generator towards this service
in different classes.Some in In-profile and some in out profile.
Then send this traffic for 1 min during that time frame drop 100 in profile
packets from the core router in both direction.



Measurement

The loss measurement must show near end and far end loss 100 in
each class taken in DUT.





3.2.17 Check the delay in point to point service and point to
multi point service in DUT


Objective

Check the two way delay measurement in point to point service and
point to multi point service using Y1731 Delay measurement in DUT.


Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
point to multi point service between PE1 and PE2. configure Y1731 delay
measurement over these service. Take the delay measurement(2 way) over
the service running in DUT for a period of time with out traffic.
This can be taken at 5 mins interval over a period of two hours.

Measurement

The two way delay values taken from DUT should not show drastic variation.
it must be consistent over a period of 2hrs.




3.2.18 Check the delay in point to point service and point to
multi point service running in DUT by dropping a couple of DMR
packet.


Objective

Check the two way delay measurement in point to point service and point
to multi point service using Y1731 Delay measurement in DUT.

Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
point to multi point service between PE1 and PE2. configure Y1731 delay
measurement over these service. Take the delay measurement(2 way) over
the service for a period of time with out traffic in DUT.This can be taken
at 5 mins interval over a period of two hours. During this time use 
firewall or impairment device to drop DMR 2 to 3 packets.

Measurement

The two way delay values taken in DUT should not show drastic variation.
it must be consistent over a period of 2hrs.




3.2.19 Check the delay in point to point service and point to
multi point service with traffic up to max bandwidth provisioned
for the services in DUT.


Objective

Check the two way delay measurement in point to point service and 
point to multi point service in DUT using Y1731 Delay measurement
with traffic

Procedure

Configure layer 2 point to point service between PE1 and PE2.Configure 
point to multi point service between PE1 and PE2. configure Y1731 delay
measurement over these service running in DUT the two way delay outputs
can be taken at 5 mins interval over a period of two hours with traffic
send with max bandwidth of the interface.

Measurement

The two way delay values should not show drastic variation.it must be
consistent over a period of 2hrs and measure the difference of delay 
with traffic and without traffic.









3.3 Realiblity


3.3.1 To Check the PM statistics are stored during routing engine
switch over in DUT..

Objective

Send 1,00,000 frames from CE to DUT from traffic generator with different
SA and DA.Send 1,00,000 frames from traffic generator to R1 with different
SA and DA so that 2,00,000 mac address will be learned in DUT. There is
a bi directional traffic flow with 1,00,000 pps in each direction.
Then do a routing engine failover in DUT.

Procedure

Configure EVPN EVI in R1,MHPE2,DUT.All 4 routers except CE are running
mpls,bgp,RR is acting as route reflector to R1,MHPE2 and DUT.Once the
bgp comes up check the DUT evpn table.For MH PE ESI must be configured
per IFD/Interface.Using RT(traffic generator) to send the traffic to
the routers.

Measurement


There should not be any traffic loss, the loss and delay measurement taken
in DUT after switch over,the counters of loss and delay should not reset.










4. Acknowledgements

We would like to thank Bhuvaneswaran Vengainathan of Veryx Technologies 
and Al Morton of (AT&T) for their support and encouragements.



5.IANA Considerations

No IANA Action is requested at this time.


6.Security Considerations

There is no additional consideration from RFC6192.

7. References

7.1. Normative References

[ITU-Y1731] OAM functions and mechanisms for Ethernet based networks



Authors’ Addresses


Praveen Ananthasankaran
Juniper Networks
1133 Innovation Way
Sunnyvale, California 94089 USA
Email: panantha <at> juniper.net


Sudhin Jacob
Juniper Networks
1133 Innovation Way
Sunnyvale, California 94089 USA
Email: sjacob <at> juniper.net

				
_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg
sudeep g ggg | 21 Jan 05:02 2016

REG: IETF 95 last date for draft presentation

Hi All,

Could you please let me know when is the last date for draft submission. May I know when is the last date for presentation request in IETF 95. Apologies, if the information is already shared.

Regards,
Sudhin

Get your own FREE website, FREE domain & FREE mobile app with Company email.  
Know More >
_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg
internet-drafts | 7 Jan 23:40 2016
Picon

I-D Action: draft-ietf-bmwg-dcbench-terminology-03.txt


A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Benchmarking Methodology Working Group of the IETF.

        Title           : Data Center Benchmarking Terminology
        Authors         : Lucien Avramov
                          Jacob Rapp
	Filename        : draft-ietf-bmwg-dcbench-terminology-03.txt
	Pages           : 15
	Date            : 2016-01-07

Abstract:
The purpose of this informational document is to establish definitions,
discussion and measurement techniques for data center benchmarking.
Also, it is to introduce new terminologies applicable to data center
performance evaluations. The purpose of this document is not to define
the test methodology, but rather establish the important concepts when
one is interested in benchmarking network switches and routers in the
data center.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-dcbench-terminology/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-03

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-bmwg-dcbench-terminology-03

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/
internet-drafts | 6 Jan 02:12 2016
Picon

I-D Action: draft-ietf-bmwg-ipv6-nd-01.txt


A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Benchmarking Methodology Working Group of the IETF.

        Title           : Benchmarking IPv6 Neighbor Cache Behavior
        Authors         : Bill Cerveny
                          Ron Bonica
	Filename        : draft-ietf-bmwg-ipv6-nd-01.txt
	Pages           : 10
	Date            : 2016-01-05

Abstract:
   This document is a benchmarking instantiation of RFC 6583:
   "Operational Neighbor Discovery Problems" [RFC6583].  It describes a
   general testing procedure and measurements that can be performed to
   evaluate how the problems described in RFC 6583 may impact the
   functionality or performance of intermediate nodes.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-ipv6-nd/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-bmwg-ipv6-nd-01

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ipv6-nd-01

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/
Picon

bmwg - New Meeting Session Request for IETF 95


A new meeting session request has just been submitted by Sarah Banks, a Chair of the bmwg working group.

---------------------------------------------------------
Working Group Name: Benchmarking Methodology
Area Name: Operations and Management Area
Session Requester: Sarah Banks

Number of Sessions: 1
Length of Session(s):  2.5 Hours
Number of Attendees: 30
Conflicts to Avoid: 
 First Priority: aqm ippm lmap ntp idr tsvwg tictoc sfc sdnrg nfvrg
 Second Priority: irtfopen xrblock tsvarea

Special Requests:
  Much better attendance and participation/recruiting when session NOT on Friday
---------------------------------------------------------
Sarah Banks | 8 Dec 17:53 2015
Picon

WGLC for draft-ietf-bmwg-virtual-net

Hello,
	In Yokohama there was an action for me to ask for WGLC on this draft. Here goes!

	There were 6 people in the Yokohama room who'd read the draft. There's been consistent feedback and
support for this draft, and I'd like to ask the WG to send this draft to WGLC. Please reply with your support
or dissent in a timely manner.

Kind regards,
Sarah
Sarah Banks | 8 Dec 17:50 2015
Picon

Minutes from IETF94

Hello BMWG!
I'm a little tardy about the minutes this meeting - I apologize! They are posted here:

Please give me a gentle kick via email if I've messed something up.

I know we say this a lot, but a huge thanks to Marius and Ramki for the minutes. I attended remotely, and while it was flawless, it's not the same as in the room, and I hadn't realized how much I'd missed until I read through their minutes. We're very lucky to have your help! Thanks!

Cheers
Sarah

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg
Sugumaran, Varthamanan | 30 Nov 04:55 2015

WG ADOPTION: draft-bhuvan-bmwg-of-controller-benchmarking (terms and meth)

Hi,

I support the draft / This work will be helpful for SDN community.

 

Thanks

Varthaman

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg
Marius Georgescu | 12 Nov 16:19 2015
Picon

Re: Mean vs Median

>my suggestion would be characterizing the latency by providing several
percentiles. I'd suggest using 10, 20, ... 90, 99, (and maybe 99.9 if we
define the required number of samples as large enough to capture long
tails). This is still a small number of data points that can easily be
included in a test report. But it is effectively a quantized CDF and
therefore captures the whole distribution including its shape reasonably
well.

[MG]  Thanks for sharing your paper. I am not sure if this solution would
make a good comparison base. As I see it, the test report has to be
synthetic enough to allow easy comparison. Reporting a single number might
not be enough, but reporting 10 numbers is too much imo.  
Marius Georgescu | 12 Nov 09:30 2015
Picon

Re: Mean vs Median

Hello Stenio,

> On Nov 12, 2015, at 03:46, Stenio Fernandes <sflf <at> cin.ufpe.br> wrote:
> 
> Very interesting results, Paul. I'd like to have a look at the results in the journal paper.
> 
> In the past, statisticians struggled with a shortage of samples to do their stuff. This is not the case for
the computer networking people as we are constantly flooded with samples. 
> 
> The discussions so far have led me to conclude that in the context here, there is no need to make any
assumptions on the sample set. Any specific measure of centrality or dispersion might not be precise
enough in some cases. As stated by others, mean/median would work for well-behaved (e.g., normally
distributed) data, but would not work for multi-modal or heavy-tailed ones. Recall that heavy-tailed
distributions are usually characterized by the shape and location parameters instead of mean and
variance. Regarding the number of samples, it is really tough to characterize heavy-tailed or
multi-modal distributions with a few samples, even using advanced algorithms for maximum-likelihood
estimation. 

As stated in the email to Paul, I don’t think increasing the sample size would be a problem, as long as the
(minimum) test time is within reasonable limits. I agree that the stability of the dataset is important.
However, a more stable dataset would still need to be summarized/reported. If not average (arithmetic
mean), median … how can we meaningfully and consistently express a test result ? 

From the Al and Paul’s examples/replies as well as other discussions I had in IETF94 about this, a
“fast&hard” solution could be the Median + a measure of variance.

Marius
_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg
Marius Georgescu | 12 Nov 09:01 2015
Picon

Re: Mean vs Median

Hi Paul,

> On Nov 11, 2015, at 23:11, Paul Emmerich <emmericp <at> net.in.tum.de> wrote:
> 
> Hi,
> 
> On 10.11.15 02:18, Marius Georgescu wrote:
> > Can you give us more context (test setup; physical/virtualized tester/DUT; one
tester/sender_receiver tester ... ) on these measurements?
> 
> Example 1: forwarding UDP packets with a userspace application at 20% of the maximum possible load
between two ports (unidirectional, MoonGen as packet generator externally)
> 
> Median latency: 42 us, 99th perc: 45 us, 99.9th perc: 71 us
> 
> I then moved the same setup into a KVM VM connected via Open vSwitch and a VirtIO vNIC and the latency
distribution changed (at 20% of the maximum load of the new setup). The traffic still came from an external host.
> 
> Median latency: 205 us, 99th perc: 341 us, 99.9th perc: 3030 us
> Example 2: see my other email from earlier today. The long tail there isn't quite as long because it uses a
"proper" packet forwarder instead of a userspace application. But the only reason it isn't as bad there is
because there was no other load on the system.
> 

Thanks for sharing the results. They are indeed quite relevant. I would like to see the rest of the draft, if
possible. 

> Example 3: interesting things happen once other stuff runs on the same hypervisor. Note that this is a
typical scenario for fancy NFV setups where you just deploy your VM whereever. The other VMs compete for
resources and this can lead to excessive delays. There is an interesting paper by Whiteaker et al. which
looks at latencies with Xen and Linux-VServer virtualization [1].
> 
> 
> > I think we should keep practicality in mind here. If we follow RFC2544.latency measurement, the frame
stream has to be 2 min long. 2000 min ~ 33h  of testing for just one test sounds unreasonable to me. I would
agree to have a lower bound for the sample size as RFC2544 actually recommends (n > 20).
> 
> Is there a good reason why this can't be changed to something more modern for a new standard? For example, I
usually aim for 1000 timestamped packets per second in my tests which works just fine.

I don’t think it would be a problem to have n>1000, as long as the test time would be 120s (or something more
reasonable than 33h).  

> 
> 
> Paul
> 
> [1] http://ccr.sigcomm.org/online/files/p39-v41n1f2-whiteakerPS.pdf

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg

Gmane