Al Morton | 1 Aug 17:22 2011
Picon

[IPFIX] WGLC: draft-ietf-bmwg-ipflow-meth-03

BMWG,
CC: IPFIX WG,

This message begins the third WG Last call on the draft:

IP Flow Information Accounting and Export Benchmarking Methodology

A URL for this draft is:
http://tools.ietf.org/html/draft-ietf-bmwg-ipflow-meth-03

The Last Call will end on August 31, 2011.

We have discussed this draft in the working group for
over three years and made many improvements.
We have also benefited from review by folks from IPFIX WG.

I now ask everyone to consider items where they commented earlier,
and make sure that the resolutions are satisfactory (and thanks to
those who did this during the second WGLC).

Please weigh-in on whether or not this Internet-Draft
should be given to the Area Directors and IESG for consideration and
publication as an Informational RFC.  Send your comments
to this list and/or acmorton <at> att.com.

Al
bmwg chair

_______________________________________________
IPFIX mailing list
(Continue reading)

Barry Constantine | 2 Aug 13:25 2011

Re: RFC2544 Statement

Hi Al,
 
I reread the draft closely and see that your points are well stated.
 
I have no further comments at this time.
 
Thanks,
Barry
 
From: Al Morton [acmorton <at> att.com]
Sent: Friday, July 29, 2011 8:07 PM
To: Barry Constantine; bmwg <at> ietf.org
Subject: Re: [bmwg] RFC2544 Statement

Hi Barry,

I think you've pointed-out two of the problems
with industry references to RFC 2544 in your message below:

Few, if any, vendors say that their tests are "derived from"
RFC 2544, because then they would clearly state what parts of
the procedures they changed, what new metrics they defined, etc.

Also, few if any service providers use the exact RFC 2544 metrics
in their SLAs. Various metrics like Committed Info Rate and Delay Variation
need to be measured and validated, but these metrics do not appear in
RFC 2544.

Speaking for my co-authors, I think we would prefer that network
providers apply appropriate standards for real-world service activation and
trouble-shooting where they exist (ITU-T Rec Y.1564 is one example),
and develop new standards where they don't. Standards Development Orgs
should include RFC 2544 as an informative reference if they believe
those methods can be used as a starting point for live
network test procedures and trouble-shooting.

Al
(as co-author/participant)

At 06:06 PM 7/28/2011, Barry Constantine wrote:
Content-Language: en-US
Content-Type: multipart/alternative;
         boundary="_000_94DEE80C63F7D34F9DC9FE69E39436BE3A0BCB7BB3MILEXCH1dsjds_"

Hi All,
 
I think this is a good effort with the right intentions.
 
There are two (2) scenarios that network providers use RFC2544 (or derivatives) today:
 
1. "Turn-up" testing which is conducted on a business class, production network before end-customer traffic is commissioned.  This is generally considered the verification of the end-customer SLA.
 
2. Troubleshooting.  This is performed after a end-customer has been commissioned and the SLA is disputed (poor application performance, etc.).  The network provider coordinates a time to swtich the end-customer to a back-up link and then re-conducts RFC2544 type testing on the production link.
 
I think this statement will be taken more credibly by network providers if it is acknowledged that these types of scenarios exist today and that caution should be used when performing RFC2544 (especially in case #2, troubleshooting).
 
Al, I think you mentioned there was a section related to exceptions (?), if these points are agreeable, then I think this section should be moved closer to the front so that a reader will grasp the overall message.
 
My two cents,
Barry
_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg
Al Morton | 2 Aug 15:28 2011
Picon

Draft IETF-81 Minutes

are here http://www.ietf.org/proceedings/81/minutes/bmwg.html

T H A N K S to Sarah Banks for taking notes in Quebec!

Also, to Chris I. for expertly handling jabber interactions.

Please review the minutes/session consensus points and send
comments before Aug 24, 2011.

regards,
Al
bmwg chair
Curtis Villamizar | 2 Aug 17:21 2011

Re: RFC2544 Statement


In message <94DEE80C63F7D34F9DC9FE69E39436BE3A0BCB7BB3 <at> MILEXCH1.ds.jdsu.net>
Barry Constantine writes:
>  
> Hi All,
>  
> I think this is a good effort with the right intentions.
>  
> There are two (2) scenarios that network providers use RFC2544 (or
> derivatives) today:
>  
> 1. "Turn-up" testing which is conducted on a business class,
>    production network before end-customer traffic is commissioned.
>    This is generally considered the verification of the end-customer
>    SLA.
>  
> 2. Troubleshooting.  This is performed after a end-customer has been
>    commissioned and the SLA is disputed (poor application performance,
>    etc.).  The network provider coordinates a time to swtich the
>    end-customer to a back-up link and then re-conducts RFC2544 type
>    testing on the production link.
>  
> I think this statement will be taken more credibly by network
> providers if it is acknowledged that these types of scenarios exist
> today and that caution should be used when performing RFC2544
> (especially in case #2, troubleshooting).
>  
> Al, I think you mentioned there was a section related to exceptions
> (?), if these points are agreeable, then I think this section should
> be moved closer to the front so that a reader will grasp the overall
> message.
>  
> My two cents,
> Barry

Barry,

The intent of RFC2544 is as a methodology for "benchmarking", not for
SLA verification and trouble shooting.  This should be very clear
after reading the first paragraph of the introduction.

  1. Introduction

   Vendors often engage in "specsmanship" in an attempt to give their
   products a better position in the marketplace.  This often involves
   "smoke & mirrors" to confuse the potential users of the products.

This is for evaluating equipment in the lab before you make a huge
purchase of equipment for a large network deployment.

Any provider that is going to spend many millios of dollars on a
deployment will hire staff specifically for testing, use the best
available off-the-shelf testing availalbe, and do way more testing
than what is described in RFC2544 before buying anything (often
borrowing the equipment used in an initial evaluation from the vendor,
then only buying from first round qualifiers for extended testing).

Of course, for small purchases far less testing rigor is applied.

BMWG has morphed over time, but originally the focus was on testing
for provider networks in the high growth years of the Internet.  When
RFC 2544 was written there was still way too much emphasis in BMWG on
repeatable results and way too little emphasis on exposing fatal flaws
in routers before they were deployed in real networks.

Curtis

BTW- The worst product ever to benchmark well in limited benchmarks
was the Cisco 7000.  In a real network it was terrible because it
dropped packets after any route change.  A network of Cisco 7000 could
easily be made to go unstable when a lot of route change was occurring
(which resulted in a positive feedback and at times massive
instabilities in networks built with these routers).  Lessons were
learned and the 7500, CEF, DCEF, GSR, etc followed and other vendors
emerged, some which learned, others which didn't.
Kevin Dubray | 2 Aug 18:53 2011
Picon

Re: RFC2544 Statement

See, Al, that training as a used car salesman comes in handy! :-)

For me, one of the key tenets in this I-D (and the WG moving forward)  is the notion of an ITE.  Are folks happy with the description of ITE in section 3?

-Kevin




On 8/2/2011 7:25 AM, Barry Constantine wrote:
<!--P { MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px } -->
Hi Al,
 
I reread the draft closely and see that your points are well stated.
 
I have no further comments at this time.
 
Thanks,
Barry
 
From: Al Morton [acmorton <at> att.com]
Sent: Friday, July 29, 2011 8:07 PM
To: Barry Constantine; bmwg <at> ietf.org
Subject: Re: [bmwg] RFC2544 Statement
...
Speaking for my co-authors, I think we would prefer that network
providers apply appropriate standards for real-world service activation and
trouble-shooting where they exist (ITU-T Rec Y.1564 is one example),
and develop new standards where they don't. Standards Development Orgs
should include RFC 2544 as an informative reference if they believe
those methods can be used as a starting point for live
network test procedures and trouble-shooting.

Al
(as co-author/participant)


_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www.ietf.org/mailman/listinfo/bmwg
Al Morton | 2 Aug 20:19 2011
Picon

Re: RFC2544 Statement

At 12:53 PM 8/2/2011, Kevin Dubray wrote:
>For me, one of the key tenets in this I-D (and the WG moving 
>forward)  is the notion of an ITE.  Are folks happy with the 
>description of ITE in section 3?
>
>-Kevin

Let's find-out, Kevin.
For those who are about to read the draft,
ITE = Isolated Test Environment.
Curtis Villamizar | 2 Aug 22:22 2011

Re: RFC2544 Statement


In message <201108021818.p72IIonp001499 <at> alpd052.aldc.att.com>
Al Morton writes:
>  
> At 12:53 PM 8/2/2011, Kevin Dubray wrote:
> >For me, one of the key tenets in this I-D (and the WG moving 
> >forward)  is the notion of an ITE.  Are folks happy with the 
> >description of ITE in section 3?
> >
> >-Kevin
>  
> Let's find-out, Kevin.
> For those who are about to read the draft,
> ITE = Isolated Test Environment.

Al,

I assume that the draft is draft-chairs-bmwg-2544-as-00.txt

And yes.  RFC 2544 is intended for ITE only and should not be used in
any other way.

BTW- Serious damage has been done by testing very high end routers
where the lab got accidentally connected to the real network and
misrouted and the lab test equipment, not using just net 10 or other
private addresses proceeded to launch an effective denial of service
attack on a number of networks.  I have seen this happen at both a
network service provider and at a high end equipment vendor.

I read the draft.  Its really short.  If the misuse has occurred, then
you have my support for making this a WG doc and moving it forward
promptly.  No salesmanship or arm twisting required.

If you want you can mention OAM work as well as IPPM.  MPLS-TP loss
measurement (LM) OAM comes to mind for validation of SLA metrics
without adding traffic.  It is a highly accurate passive verification
tool with almost zero overhead.  There is other work going on for
measuring delay and jitter.  Just a suggestion.  Feel free to ignore.

Curtis
Al Morton | 2 Aug 23:34 2011
Picon

Re: RFC2544 Statement

At 04:22 PM 8/2/2011, Curtis Villamizar wrote:
>...
>I read the draft.  Its really short.  If the misuse has occurred, then
>you have my support for making this a WG doc and moving it forward
>promptly.  No salesmanship or arm twisting required.

Good, thanks.

>If you want you can mention OAM work as well as IPPM.  MPLS-TP loss
>measurement (LM) OAM comes to mind for validation of SLA metrics
>without adding traffic.  It is a highly accurate passive verification
>tool with almost zero overhead.  There is other work going on for
>measuring delay and jitter.  Just a suggestion.  Feel free to ignore.

I'll take a look at the TP Loss (again) soon.
All passive measurement methods are not created equal...

Al
Curtis Villamizar | 3 Aug 05:20 2011

Re: RFC2544 Statement


In message <201108022134.p72LYDDv032349 <at> alpd052.aldc.att.com>
Al Morton writes:
>  
> >If you want you can mention OAM work as well as IPPM.  MPLS-TP loss
> >measurement (LM) OAM comes to mind for validation of SLA metrics
> >without adding traffic.  It is a highly accurate passive verification
> >tool with almost zero overhead.  There is other work going on for
> >measuring delay and jitter.  Just a suggestion.  Feel free to ignore.
>  
> I'll take a look at the TP Loss (again) soon.
> All passive measurement methods are not created equal...
>  
> Al

Al,

If you understand PPP LQM, then you'll understand LM OAM.

Curtis
Al Morton | 4 Aug 17:40 2011
Picon

BMWG Supplemental page back on-line

BMWG,

I just learned from an old friend that our supplemental page
was temporarily disabled.

http://home.comcast.net/~acmacm/BMWG/

is reachable again. Apologies to those who tried and got 404.

In other news, I've just learned that both Routing ADs are
ready to clear their DISCUSSes on the IGP Dataplane drafts,
we just need a revised meth draft with 2 small changes...

...and the people prepared to rejoice,
Al
bmwg chair

Gmane