headers
Howard C. Berkowitz | 7 Apr 2004 01:43

Re: Is the BMWG a proper home for this I-D?

>Supporters of the BMWG:
>
>A [long] while back, Russ White approached the BMWG with an 
>individually submitted I-D
>that presented considerations when measuring network convergence:
>
>http://www.ietf.org/internet-drafts/draft-white-network-benchmark-00.txt
>
>The draft seeks to highlight: ...considerations that testers should 
>be aware of when
>attempting to measure network convergence using various methods. 
>(Adapted from the I-D's
>abstract.)
>
>Not being a pure terminology or methodological statement, or 
>constraining itself to
>laboratory evaluations, the topic might not be a perfect fit as BMWG 
>output.  On the
>other hand, the community might benefit from memo on this topic.
>
>Please offer your comments in two dimensions: a) does the topic 
>merit future effort, and
>b) is the BMWG the best home for this effort (as opposed to another 
>WG or progressing the
>work as an Individually submitted I-D)?  (You might have to read the 
>WG Charter to determine
>part B. :-)

I'm not quite sure that I'm ready to answer in the specific 
dimensions, but I can make some comments. I'll add a dimension that I
(Continue reading)

Permalink | Reply | 12 comments |
headers
Jim McQuaid | 7 Apr 2004 12:56

RE: Is the BMWG a proper home for this I-D?

<sorry for the blank reply>

At the time the IPPM was "spun off" of the BMWG, the general understanding
was that BMWG would look at "things" more or less in isolation, although
that concept is somewhat elastic.  IPPM would look at end to end network
kinds of performance issues.

It seems clear to me that convergence is a performance characteristic of
complete networks and primarily an end to end or at least, core element to
core element characteristic.

My thought would be that this is interesting and useful work but not
necessarily for BMWG.

Jim McQuaid

-----Original Message-----
From: Howard C. Berkowitz [mailto:hcb <at> gettcomm.com]
Sent: Tuesday, April 06, 2004 7:43 PM
To: bmwg <at> ietf.org
Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

>Supporters of the BMWG:
>
>A [long] while back, Russ White approached the BMWG with an 
>individually submitted I-D
>that presented considerations when measuring network convergence:
>
>http://www.ietf.org/internet-drafts/draft-white-network-benchmark-00.txt
>
(Continue reading)

Permalink | Reply | 10 comments |
headers
Jim McQuaid | 7 Apr 2004 12:56

RE: Is the BMWG a proper home for this I-D?


-----Original Message-----
From: Howard C. Berkowitz [mailto:hcb <at> gettcomm.com]
Sent: Tuesday, April 06, 2004 7:43 PM
To: bmwg <at> ietf.org
Subject: Re: [bmwg] Is the BMWG a proper home for this I-D?

>Supporters of the BMWG:
>
>A [long] while back, Russ White approached the BMWG with an 
>individually submitted I-D
>that presented considerations when measuring network convergence:
>
>http://www.ietf.org/internet-drafts/draft-white-network-benchmark-00.txt
>
>The draft seeks to highlight: ...considerations that testers should 
>be aware of when
>attempting to measure network convergence using various methods. 
>(Adapted from the I-D's
>abstract.)
>
>Not being a pure terminology or methodological statement, or 
>constraining itself to
>laboratory evaluations, the topic might not be a perfect fit as BMWG 
>output.  On the
>other hand, the community might benefit from memo on this topic.
>
>Please offer your comments in two dimensions: a) does the topic 
>merit future effort, and
>b) is the BMWG the best home for this effort (as opposed to another
(Continue reading)

Permalink | Reply | 0 comments |
headers
Howard C. Berkowitz | 7 Apr 2004 15:28

RE: Is the BMWG a proper home for this I-D?

At 10:56 AM +0000 4/7/04, Jim McQuaid wrote:
><sorry for the blank reply>
>
>At the time the IPPM was "spun off" of the BMWG, the general understanding
>was that BMWG would look at "things" more or less in isolation, although
>that concept is somewhat elastic.  IPPM would look at end to end network
>kinds of performance issues.
>
>It seems clear to me that convergence is a performance characteristic of
>complete networks and primarily an end to end or at least, core element to
>core element characteristic.
>
>My thought would be that this is interesting and useful work but not
>necessarily for BMWG.
>
>Jim McQuaid

I don't completely disagree with you.  I do feel that single-platform 
performance definition and benchmarking is definitely within the 
scope of BMWG, and some of my comments relating to clarifying test 
methods (e.g., black box vs. white box vs. passive analysis) 
definitely can be applied to one box.  BMWG has always looked at 
forwarding plane performance of single boxes, and I really don't 
think we are straying far to look at there control plane performance.

You are correct that IPPM, as well as some of the internet 
scalability issues both in the IETF and IRTF, are probably better 
places for end-to-end convergence.

At least in the case of BGP, however, there is a meaningful
(Continue reading)

Permalink | Reply | 8 comments |
headers
Robert Holley | 7 Apr 2004 15:36
Picon
Favicon

RE: Is the BMWG a proper home for this I-D?


I think the concept of "things in isolation" can be
broadened to encompass things not necessarily contained
on a single element.

For example, routing protocol convergence would impact
multiple components, 1) single node reaching steady state
2) all routing nodes in AS reaching steady state.

However, routing protocol convergence may be one
of many factors that would make up the
"overall end-to-end" performance picture.

A complex network may consist of many things that
can be initially considered in isolation, but each
having subtle relationships, which could be
"reconstituted" into an end-to-end picture.

For example, a network carrying MPLS tunnels,
Multicast, some VPN technologies etc. and
in addition running a routing protocol.

At first each one of the technologies could be
studied independently and "benchmarked". Then the
overall description of the network performance would
be a combination of each of the independent benchmarks.

So, it may be that the benchmarking group could develop a
benchmark description for each of the various routing protocols,
then some group could develop benchmarks for the other technologies
(Continue reading)

Permalink | Reply | 0 comments |
headers
David Newman | 7 Apr 2004 18:19

RE: Is the BMWG a proper home for this I-D?


On Wed, 7 Apr 2004, Howard C. Berkowitz wrote:

> I don't completely disagree with you.  I do feel that single-platform
> performance definition and benchmarking is definitely within the
> scope of BMWG, and some of my comments relating to clarifying test
> methods (e.g., black box vs. white box vs. passive analysis)
> definitely can be applied to one box.

What about 2547bis et al, where one physical element contains multiple
logical elements? In effect a "one box only" rule would let slide virtual
technologies like MPLS VPNs (which IMO shouldn't be getting any slack...)
but not more basic stuff like straight routing.

Rather than getting hung up on one vs. not one box, it might be more
constructive to draw a distinction between measurements made in isolation
and those made end-to-end on live networks (for some definition of
"live").

The bmwg has always worked to define the measurement of devices or systems
(multiple devices) in a controlled environment. This is the "bench" in
benchmarking and while it's conceptually 180 degrees the opposite of live
network testing it doesn't necessarily rule out multiple-box tests either.

dn
Permalink | Reply | 6 comments |
headers
Robert Holley | 7 Apr 2004 20:04
Picon
Favicon

RE: Is the BMWG a proper home for this I-D?


To a certain extent Howard's example of multiple components
of a single routing protocol re-enforces my earlier concept,
by diving down in a hierachy.

He referred to route reflectors, intra-as etc, as all being
sub-components of a problem (focusing on routing protocols).

And the next level UP in the hierarchy would be routing
protocols themselves as a part of a bigger picture
(as I eluded to earlier).

As far as the mentioned rule of thumb.... on "synthetic lab" vs
"field measurement";

I'm not familar with your history on this criteria, but I would
assume that if standard interfaces (SNMP) are used to define the "model",
then it could be evalutated in the lab with generators and monitoring tools,
and also deployed in the field (at least the monitoring part).

Of course there will most likely be the issues of MIB development to
support the requirements for the newly defined benchmark interfaces.

I assume MIB development is another group?

regards
Bob

> -----Original Message-----
> From: bmwg-admin <at> ietf.org [mailto:bmwg-admin <at> ietf.org]On Behalf Of
(Continue reading)

Permalink | Reply | 0 comments |
headers
Kevin Dubray | 7 Apr 2004 21:17
Favicon

Re: Is the BMWG a proper home for this I-D?


David Newman wrote:

> Rather than getting hung up on one vs. not one box, it might be more
> constructive to draw a distinction between measurements made in isolation
> and those made end-to-end on live networks (for some definition of
> "live").
> 
> The bmwg has always worked to define the measurement of devices or systems
> (multiple devices) in a controlled environment. This is the "bench" in
> benchmarking and while it's conceptually 180 degrees the opposite of live
> network testing it doesn't necessarily rule out multiple-box tests either.

Hear, hear.

This is reinforced by language in the charter:

"To better distinguish the BMWG from other measurement initiatives
in the IETF, the scope of the BMWG is limited to technology
characterization using simulated stimuli in a laboratory environment.
Said differently, the BMWG does not attempt to produce benchmarks for
live, operational networks."

No mention of number of nodes. :-)

-Kevin
Permalink | Reply | 0 comments |
headers
Howard C. Berkowitz | 7 Apr 2004 23:23

RE: Is the BMWG a proper home for this I-D?

At 9:19 AM -0700 4/7/04, David Newman wrote:
>On Wed, 7 Apr 2004, Howard C. Berkowitz wrote:
>
>>  I don't completely disagree with you.  I do feel that single-platform
>>  performance definition and benchmarking is definitely within the
>>  scope of BMWG, and some of my comments relating to clarifying test
>>  methods (e.g., black box vs. white box vs. passive analysis)
>>  definitely can be applied to one box.
>
>What about 2547bis et al, where one physical element contains multiple
>logical elements? In effect a "one box only" rule would let slide virtual
>technologies like MPLS VPNs (which IMO shouldn't be getting any slack...)
>but not more basic stuff like straight routing.

Let me clarify. Single box is clearly in scope, but I have absolutely 
nothing against multiple box as long as those boxes can be bounded 
and the bounded system tested with isolated loads. Are we in violent 
agreement?

What I was trying to exclude was "Internet-wide" scope and 
uncontrolled, non-repeatable loads.

>
>Rather than getting hung up on one vs. not one box, it might be more
>constructive to draw a distinction between measurements made in isolation
>and those made end-to-end on live networks (for some definition of
>"live").

That's valid.  I'd also ask Russ et al if this distinction takes 
passive traffic monitoring out of the scope here, and brings us back
(Continue reading)

Permalink | Reply | 4 comments |
headers
Jim McQuaid | 8 Apr 2004 00:09

RE: Is the BMWG a proper home for this I-D?ch

You make an excellent point (can it be tested with created traffic or does
it require live network traffic in some form).  That's the "bench" in
benchmark; it can be done with lab equipment.

I would agree with that distinction.  On the other hand, if there is
enthusiasm to do this work within BMWG, tthat may be more important than
legalisms, however sound.

Jim McQuaid

-----Original Message-----
From: Howard C. Berkowitz [mailto:hcb <at> gettcomm.com]
Sent: Wednesday, April 07, 2004 9:29 AM
To: bmwg <at> ietf.org
Subject: RE: [bmwg] Is the BMWG a proper home for this I-D?

At 10:56 AM +0000 4/7/04, Jim McQuaid wrote:
><sorry for the blank reply>
>
>At the time the IPPM was "spun off" of the BMWG, the general understanding
>was that BMWG would look at "things" more or less in isolation, although
>that concept is somewhat elastic.  IPPM would look at end to end network
>kinds of performance issues.
>
>It seems clear to me that convergence is a performance characteristic of
>complete networks and primarily an end to end or at least, core element to
>core element characteristic.
>
>My thought would be that this is interesting and useful work but not
>necessarily for BMWG.
(Continue reading)

Permalink | Reply | 1 comment |

Gmane