WG Action: RECHARTER: Benchmarking Methodology (bmwg)

The Benchmarking Methodology (bmwg) working group in the Operations and
Management Area of the IETF has been rechartered.  For additional
information, please contact the Area Directors or the working group
Chairs.

Benchmarking Methodology (bmwg)
---------------------------------------------------
Current Status: Active Working Group

Chair:
  Al Morton <acmorton <at> att.com>

Operations and Management Area Directors:
  Ronald Bonica <rbonica <at> juniper.net>
  Dan Romascanu <dromasca <at> avaya.com>

Operations and Management Area Advisor:
  Ronald Bonica <rbonica <at> juniper.net>

Mailing Lists:
  Address:	bmwg <at> ietf.org
  To Subscribe:	bmwg-request <at> ietf.org
  Archive:	http://www.ietf.org/mail-archive/web/bmwg/

Description of Working Group:

The Benchmarking Methodology Working Group (BMWG) will continue to 
produce a series of recommendations concerning the key performance 
characteristics of internetworking technologies, or benchmarks for 
network devices, systems, and services. Taking a view of networking 

Re: WG Action: RECHARTER: Benchmarking Methodology (bmwg)

Hi,

There are two typos in our new Charter text,
please see edits below.  Ron will approve the
Milestone change, if necessary.

thanks, and sorry for the trouble,
Al
chair, bmwg

At 05:58 PM 11/1/2010, IESG Secretary wrote:
>The Benchmarking Methodology (bmwg) working group in the Operations and
>Management Area of the IETF has been rechartered.  For additional
>information, please contact the Area Directors or the working group
>Chairs.
>
>Benchmarking Methodology (bmwg)
>---------------------------------------------------
>Current Status: Active Working Group
>
>Chair:
>   Al Morton <acmorton <at> att.com>
>
>...stuff deleted...
>
>* Data Center Bridging Devices:
>Some key concepts from BMWG's past work are not meaningful when testing
>switches that implement new IEEE specifications in the area of data
>center bridging. For example, throughput as defined in RFC 1242 cannot
>be measured when testing devices that implement three new IEEE

Re: WGLC: draft-ietf-bmwg-reset-02

At 07:39 AM 10/11/2010, Al Morton wrote:
>This message begins the Second WG Last call on the draft:
>
>         Title           : Device Reset Characterization
>         Author(s)       : R. Asati, et al.
>         Filename        : draft-ietf-bmwg-reset-02.txt

I have the following comments as this last call closes.
I believe these are entirely editorial suggestions.

There are several instances where the Time-stamp method
of recovery time measurement was not fully incorporated
into the text, but I consider these predominantly
editorial changes as well.

Al (as a participant)

OLD
1. Introduction

    An operational forwarding device (or one of its components) may need
    to be re-started for a variety of reasons, an event that we call a
    "reset" in this draft. Since there may be an interruption in the
    forwarding operation during a reset, it is useful to know how long a
    device takes to resume the forwarding operation. In other words, it
    is desired to know how long the recovery time after the reset is.

Suggest:
            ...In other words, it
    is desired to know the duration of the recovery time following the reset.

Re: WGLC: draft-ietf-bmwg-reset-02

Al, 

Thanks for your review and feedback. Your proposed changes look fine to
me and will improve the readability.

Cheers,
Rajiv

> -----Original Message-----
> From: bmwg-bounces <at> ietf.org [mailto:bmwg-bounces <at> ietf.org] On Behalf
Of Al Morton
> Sent: Sunday, November 07, 2010 5:00 PM
> To: bmwg <at> ietf.org
> Subject: Re: [bmwg] WGLC: draft-ietf-bmwg-reset-02
> 
> At 07:39 AM 10/11/2010, Al Morton wrote:
> >This message begins the Second WG Last call on the draft:
> >
> >         Title           : Device Reset Characterization
> >         Author(s)       : R. Asati, et al.
> >         Filename        : draft-ietf-bmwg-reset-02.txt
> 
> I have the following comments as this last call closes.
> I believe these are entirely editorial suggestions.
> 
> There are several instances where the Time-stamp method
> of recovery time measurement was not fully incorporated
> into the text, but I consider these predominantly
> editorial changes as well.
> 

I-D Action:draft-ietf-bmwg-igp-dataplane-conv-term-22.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Benchmarking Methodology Working Group of the IETF.

	Title           : Terminology for Benchmarking Link-State IGP Data Plane Route Convergence
	Author(s)       : S. Poretsky, et al.
	Filename        : draft-ietf-bmwg-igp-dataplane-conv-term-22.txt
	Pages           : 29
	Date            : 2010-11-08

This document describes the terminology for benchmarking Interior
Gateway Protocol (IGP) Route Convergence.  The terminology is to be
used for benchmarking IGP convergence time through externally
observable (black box) data plane measurements.  The terminology can
be applied to any link-state IGP, such as ISIS and OSPF.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-term-22.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org

I-D Action:draft-ietf-bmwg-igp-dataplane-conv-meth-22.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Benchmarking Methodology Working Group of the IETF.

	Title           : Benchmarking Methodology for Link-State IGP Data Plane Route Convergence
	Author(s)       : S. Poretsky, et al.
	Filename        : draft-ietf-bmwg-igp-dataplane-conv-meth-22.txt
	Pages           : 39
	Date            : 2010-11-08

This document describes the methodology for benchmarking Link-State
Interior Gateway Protocol (IGP) Route Convergence.  The methodology
is to be used for benchmarking IGP convergence time through
externally observable (black box) data plane measurements.  The
methodology can be applied to any link-state IGP, such as ISIS and
OSPF.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-igp-dataplane-conv-meth-22.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

_______________________________________________
bmwg mailing list

Re: WGLC: draft-ietf-bmwg-reset-02

Al, WG,

Many thanks, all of your comments as well as all WGLC comments have been
incorporated into the just posted draft-ietf-bmwg-reset-03 revision,
which is much improved !

Thanks,

-- Carlos.

On 11/7/2010 4:59 PM, Al Morton wrote:
> At 07:39 AM 10/11/2010, Al Morton wrote:
>> This message begins the Second WG Last call on the draft:
>>
>>         Title           : Device Reset Characterization
>>         Author(s)       : R. Asati, et al.
>>         Filename        : draft-ietf-bmwg-reset-02.txt
> 
> I have the following comments as this last call closes.
> I believe these are entirely editorial suggestions.
> 
> There are several instances where the Time-stamp method
> of recovery time measurement was not fully incorporated
> into the text, but I consider these predominantly
> editorial changes as well.
> 
> Al (as a participant)
> 
> OLD
> 1. Introduction

I-D Action:draft-ietf-bmwg-reset-03.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Benchmarking Methodology Working Group of the IETF.

	Title           : Device Reset Characterization
	Author(s)       : R. Asati, et al.
	Filename        : draft-ietf-bmwg-reset-03.txt
	Pages           : 18
	Date            : 2010-11-09

An operational forwarding device may need to be re-started
(automatically or manually) for a variety of reasons, an event that
we call a "reset" in this document. Since there may be an
interruption in the forwarding operation during a reset, it is
useful to know how long a device takes to resume the forwarding
operation.

This document specifies a methodology for characterizing reset (and
recovery time) during benchmarking of forwarding devices, and
provides clarity and consistency in reset test procedures beyond
what's specified in RFC2544. It therefore updates RFC2544.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-reset-03.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

Re: Draft BMWG Agenda Posted for IETF-79

I'd like to propose a small revision to our draft
agenda.  Fred Baker has prepared a benchmarking proposal
and has asked for a few minutes at the beginning of our
session to discuss it, face2face with those present:
http://tools.ietf.org/html/draft-baker-bmwg-testing-eyeball-happiness-00

I've exchanged some messages with Fred and suggested
a few changes that would make this work more consistent with
our charter, such as removing the Pass/Fail threshold.
We can discuss these points on the list and at the session
on Thursday afternoon, Beijing time.

More background on this topic is available in the
Wing and Yourtchenko draft Fred references:
http://tools.ietf.org/html/draft-wing-v6ops-happy-eyeballs-ipv6-01
and the Sept 2010 edition of IPJ.

So, we'll take a few minutes for this at the start of
our session, and Fred can attend another session as soon
as we are done.

Al
bmwg chair

At 08:02 AM 10/27/2010, Al Morton wrote:
>BMWG,
>
>The draft agenda is posted:
>http://www.ietf.org/proceedings/79/agenda/bmwg.txt
>

Re: Draft BMWG Agenda Posted for IETF-79

Interesting draft. I've seen this behavior but hadn't thought about
testing it.

In lieu of a pass/fail metric it might make sense to define
response-time goalposts, such as (warning: strawman example) the
interval between Alice's first DNS request and Bob's SYN/ACK.

Also, it probably would be helpful to clarify the procedure for testing
non-determinism in DNS servers (how many trials is adequate?).

Same thing for clearing Alice's resolver between iterations. I
understand the concept of trying with and without Bob's info in Alice's
resolver, but I'm less clear on the statement that "The fact that Alice
was able to contain Bob at a given address should not preclude Alice
trying other addresses on subsequent attempts."

Different implementations do different things, but in general I think
most prefer to use info from their own resolvers if it exists over
running another DNS query (and some do this even when they're not
supposed to). A simple requirement that resolvers should/should not be
cleared before a given test would address this.

dn

On 11/9/10 5:06 PM, Al Morton wrote:
> I'd like to propose a small revision to our draft
> agenda.  Fred Baker has prepared a benchmarking proposal
> and has asked for a few minutes at the beginning of our
> session to discuss it, face2face with those present:
> http://tools.ietf.org/html/draft-baker-bmwg-testing-eyeball-happiness-00