headers
Al Morton | 1 Nov 2006 15:43
Picon
Favicon

Reminder on Slides for IETF-67

At 07:32 AM 10/11/2006, Al Morton wrote:
>Also (after the draft session agenda is published):
>
>      All presentation materials submitted by NOVEMBER 1.
>      *One* revision to the presentation allowed, >24hours before session.
Permalink | Reply | 0 comments |
headers
Poretsky, Scott | 3 Nov 2006 23:13

RE: Comments on SIP term-02 and meth-00 drafts


-----Original Message-----
From: Daryl Malas [mailto:daryl <at> level3.net] 
Sent: Tuesday, October 31, 2006 5:12 PM
To: Al Morton
Cc: bmwg <at> ietf.org
Subject: Re: [bmwg] Comments on SIP term-02 and meth-00 drafts

I just read through these two drafts, and I agree with Al's assessment
on the last two metrics: Session Setup Delay and Session Teardown Delay.
These overlap with the metrics in the draft-malas-perfor...-05.txt.
IMO, they seem very "end-to-end" in nature, as opposed to benchmarking a
single DUT/SUT.

Or...are you benchmarking them in the following manner:
Session Setup Delay: From the time a UAC (DUT) receives an INVITE to the
time it responds with a 18x message

Session Teardown Delay: From the time the DUT/SUT receives a BYE to the
time it is able to respond with a 200

--Daryl

SCOTT> Yes we are benchmarking in that manner.  End-to-end session
setup/teardown delay is very dependent on that metric for the individual
DUT.
Permalink | Reply | 0 comments |
headers
Al Morton | 4 Nov 2006 05:34
Picon
Favicon

Remote Participation

I had some requests for the jabber instructions,
you can find them here:
https://datatracker.ietf.org/public/meeting_materials.cgi?meeting_num=67

It looks like the audio streaming info is under construction here:
http://videolab.uoregon.edu/events/ietf/

Al
Permalink | Reply | 0 comments |
headers
Al Morton | 4 Nov 2006 14:26
Picon
Favicon

Correction: Remote Participation

In the light of day, I saw that the first link was
incorrect. "Copy" must have missed.

I had some requests for the jabber instructions,
you can find them here:
http://www.ietf.org/meetings/text_conf.html

It looks like the audio streaming info is under construction here:
http://videolab.uoregon.edu/events/ietf/

Al
Permalink | Reply | 0 comments |
headers
Poretsky, Scott | 5 Nov 2006 14:55

RE: [Sipping] draft-rosenberg-sipping-overload-reqs recovery



I think it is worth mentioning that there is work in the Benchmarking Methodology Working Group (BMWG) to perform this type of SIP device benchmarking.  This can be found at:

http://tools.ietf.org/id/draft-poretsky-sip-bench-term-02.txt

and

http://www.ietf.org/internet-drafts/draft-poretsky-bmwg-sip-bench-meth-00.txt

-----Original Message-----
From: Cullen Jennings [mailto:fluffy <at> cisco.com]
Sent: Fri 11/3/2006 9:23 PM
To: sipping; Jonathan Rosenberg
Subject: [Sipping] draft-rosenberg-sipping-overload-reqs recovery


A possible additional requirement....

Imagine a system (perhaps a single proxy) that could do 100cps. It is 
in steady state doing 80cps with very few retransmission. Then for 5 
minutes the incoming requests goes to 500cps then drops back to an 
incoming call rate of 80cps. The question is, how long before the 
system gets back to the state where it if is successfully processing 
all the 80cps?

I have seen systems that never recover - that is bad. I think one of 
the design goals is that it is at least possible to build to systems 
that recover back to steady state relatively quickly after an 
overload impulse.



_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use sip-implementors <at> cs.columbia.edu for questions on current sip
Use sip <at> ietf.org for new developments of core SIP

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 0 comments |
headers
Arun Gandhi | 6 Nov 2006 20:45
Favicon

RE: New Version of Protection Benchmarking (terminology/Methodology)

Dear Authors,

 

This is a neat work. However, I do have some comments/suggestions/clarifications on the first reviewed draft.

 

1) Draft- BMWG Protection Terminology

 

a) Section 1 (Introduction): Such technologies include High Availability

  (HA) stateful failover. Virtual Router Redundancy Protocol (VRRP)

                    ^^^^^

Arun> this looks like a typo. Need comma between the two.

 

b) Section 1 (Introduction) last paragraph (first 5-6 lines):

 

   Arun> IMHO, we should add the definition of primary & backup paths as

   Arun> done nicely for the others key items.

 

   Arun> Here’s my suggestion for first couple lines…

 

   Arun> Protection Switching consists of a minimum of two Protection-

   Arun> Switching Nodes with a Primary Path and a Backup Path. A primary

   Arun> path may be defined as a path that is currently being used to

   Arun> forward traffic while a Backup Path may be defined as a path

   Arun> that is a standby for the primary path and has the same or

   Arun> different resource constraints such as, bandwidth, latency (see

   Arun> Sections 3.1.4 and 3.1.6). A Failover Event occurs along the

   Arun> Primary Path.

 

c) Section 3.1.1:

 

   Arun> appears twice in the document that needs to be corrected (likely

   Arun> an oversight).

 

d) Under section 3.1.1 (Path) -> “See Also”:

 

   Arun> [please correct my understanding] the topics provided doesn’t

   Arun> seem to have any relevance here.

 

e) Section 3.2.1: Definition of “Protection Switching System”:

 

   > Definition:

   > A SUT that is capable of Failover from a Primary to a Backup Path.

                   ^^^^^^^^^^^^^^^^^^^    

   Arun> in my understanding, this should be “capable of switchover” and

   Arun> not “Failover”. But reading the draft in entirety this seems apt

   Arun> unless there are other suggestions.

 

f) Section 3.2.4 -> Discussion (last line):

 

> A Backup Path providing Path Protection MUST have the same ingress

> node as the Primary Path.

 

   Arun> should not the backup path have the same ingress AND the egress

   Arun> nodes as the primary path?

 

g) Section 3.3:

 

   Arun> Failover Time, Failure Recovery, and Reversion Time should also

   Arun> be defined under this Section 3.3 (Failure).

 

h) Section 3.3.4:

 

   Arun> please complete acronym FRR (for completeness).

 

i) Sections 3.5.1 and 3.5.2:

 

   Arun> Definitions for 3.5.1 and 3.5.2 are ditto. The authors might want

   Arun> to reword to show the distinction between the two.

 

Lastly, IMHO, “Failover” in the draft must be replaced with “Switchover” as it is also consistent with the MPLS terminology of PSL (Path Switching LSR) and PML (Path Merge LSR).

 

Hope this helps.

 

Regards,

Arun Gandhi.

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 0 comments |
headers
Arun Gandhi | 7 Nov 2006 17:21
Favicon

RE: WG Last Call on Accelerated Stress Benchmarking Drafts

Hi Scott,

>> The BMWG makes it a point to not benchmark 'services', so I recommend
>> we stick with use of the word 'protocols' and so not replace it with
>> 'services'. 

> I respectfully disagree as based on this particular work item (document's
> motivation), I'm still of the opinion that we are not characterizing
> different protocols together but in fact different services with protocols
> per se overlaid on a SUT. Inevitably (and as nicely explained), a router
> in an operational network is configured with multiple protocols (ospf,bgp,
> mpls, multicast, etc.) and services (such as vpns, vpls, policies, etc.)
> based on the role it is playing in that network. Also, there are other
> features/services such as graceful-restart (GRS), non-stop routing, ISSU,
> and so on that may be turned on as well.

Also, here's what I just found on the BMWG charter (section 'Description of
Working Group')...

"The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services 
                                                ^^^^^^^^^^^^^^^^^^^ 
that are built from these technologies.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results."

Regards,
Arun Gandhi.

-----Original Message-----
From: Arun Gandhi [mailto:gandhi <at> juniper.net] 
Sent: Tuesday, November 07, 2006 10:20 AM
To: 'Poretsky, Scott'; 'Al Morton'; bmwg <at> ietf.org; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

Hi Scott,

> The terms 'black-box' and 'white-box' have been frequently used in the
> BMWG over the past 10 years. I would be happy to use this work item as
> the place to define them. The focus of this work is black-box 
> benchmarking, consistent with the BMWG charter. With this particular 
> work item the working group agreed there would be value in observing
> White-Box benchmarks so it was preferred to include that in an
> appendix.

I totally agree.

> In addition, I do not agree with your suggestion to use "number" as a
> unit. Specifically you recommended to replace unit "sessions" with unit
> "number".  When answering the question "How many sessions?" could you
> answer 10 numbers or 10 sessions.  No change will be made.

A little correction... i was actually thinking "number of sessions". Sorry
for the confusion. Is this agreeable?

> The BMWG makes it a point to not benchmark 'services', so I recommend
> we stick with use of the word 'protocols' and so not replace it with
> 'services'. 

I respectfully disagree as based on this particular work item (document's
motivation), I'm still of the opinion that we are not characterizing
different protocols together but in fact different services with protocols
per se overlaid on a SUT. Inevitably (and as nicely explained), a router in
an operational network is configured with multiple protocols (ospf, bgp,
mpls, multicast, etc.) and services (such as vpns, vpls, policies, etc.)
based on the role it is playing in that network. Also, there are other
features/services such as graceful-restart (GRS), non-stop routing, ISSU,
and so on that may be turned on as well.

Also, IMHO, under Section 3.3 we should add "Number of ifls/ifds" and
"Number of policies" in anyone of the four buckets (where it seems most
fit).

Lastly, just a thought/question...Under Section 3.3.3, is it possible to
actually explicitly outline a threshold based on the system/SUT resources in
use after which a box can be deemed as "Unstable" (e.g. a router in an
operational network using 85% of the system resources will/should be
considered unstable)? 

Thanks,
Arun Gandhi.

-----Original Message-----
From: Poretsky, Scott [mailto:sporetsky <at> reefpoint.com]
Sent: Friday, October 20, 2006 12:46 PM
To: Arun Gandhi; Al Morton; bmwg <at> ietf.org; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

Hi Arun,

I apologize that this response is so late.  

The BMWG makes it a point to not benchmark 'services', so I recommend we
stick with use of the word 'protocols' and so not replace it with
'services'.  The terms 'black-box' and 'white-box' have been frequently
used in the BMWG over the past 10 years.  I would be happy to use this
work item as the place to define them.  The focus of this work is
black-box benchmarking, consistent with the BMWG charter.  With this
particular work item the working group agreed there would be value in
observing White-Box benchmarks so it was preferred to include that in an
appendix.  At this time there is no plan to make further changes on
these topics unless others in the working group want to continue the
discussion.

In addition, I do not agree with your suggestion to use "number" as a
unit.  Specifically you recommended to replace unit "sessions" with unit
"number".  When answering the question "How many sessions?" could you
answer 10 numbers or 10 sessions.  No change will be made.

Also, there are 3 sentence changes you requested below for which I do
not see the value of those changes.  Maybe you could provide more
information.

Thanks for your thorough review of the documents.

Scott

-----Original Message-----
From: Arun Gandhi [mailto:gandhi <at> juniper.net] 
Sent: Tuesday, August 29, 2006 11:55 AM
To: 'Al Morton'; bmwg <at> ietf.org; Poretsky, Scott; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts
Importance: High

Hello Authors,

I am little late in responding wrt the open dates but I suppose my
feedback will get evaluated. I will also follow up with the feedback on
methodology document shortly.

<-- Section 1 (Introduction): 

In general I believe this draft should not just address multiple
Protocols operational simultaneously but "different services" 
being overlaid in a fully operational box.

> Routers in an operational network are simultaneously configured with
> multiple protocols and security policies while forwarding traffic and
> being managed.

Arun> IMHO, replace "protocols" with "services" as in ISP world (which
Arun> is the motivation for this draft) there may be different service
Arun> sets such as l3vpn, l2vpns, VPLS, etc. along with routing and 
Arun> signaling protocols configured.

> It is helpful to accelerate these network operational conditions so 
> that the router under test can be benchmarked with faster test 
> duration.

Arun> This needs revision/clarification.

> Multiple benchmarks are made for each Benchmark Plane during each 
> Phase.

Arun> Benchmarks are defined for each plane to be characterized in all 
Arun> phases.

> These benchmarks White Box benchmarks are provided in Appendix 1 for
> additional DUT behavior measurements.

Arun> Two points:
Arun> 1) This line needs revision.
Arun> 2) A new term "White" box is added without being explained. Also, 
Arun> the impression I get reading this draft is that it's a black box
Arun> test. Also, since the motivation is the benchmarking of a router
Arun> in ISP domain then for sure it is a black box testing. Regardless
Arun> we should monitor/characterize other system parameters such as 
Arun> "memory usage", "CPU utilization", "install/bring up times", and 
Arun> so on.

<-- Section 3.2.1 (Control Plane)

> The Control Plane defines the Configuration, Startup Conditions, and
> Instability Conditions of the control protocols. Control Plane 
> protocols may include routing protocols, multicast protocols, and 
> MPLS protocols.     

Arun> IMHO, this statement should be revised to be "The Control Plane 
Arun> defines the Configuration, Startup Conditions, and Instability
Arun> Conditions of the control plane protocols, which may include 
Arun> routing protocols, multicast protocols, and MPLS protocols.  

<-- Section 3.3.2.3 (Control Plane)

> Measurement units: sessions

Arun> IMHO, this should be replaced with "number" (as "sessions" in not
Arun> a measuring unit) 

Thanks,
Arun

-----Original Message-----
From: Al Morton [mailto:acmorton <at> att.com] 
Sent: Monday, July 17, 2006 9:31 AM
To: bmwg <at> ietf.org
Subject: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

BMWG:

A WG Last Call period for the Internet-Drafts on

Terminology for Accelerated Stress Benchmarking Methodology Guidelines
for Accelerated Stress Benchmarking will be open from 17 July 2006
through 28 August 2006 (6 weeks).

URLs for these Internet-Drafts are:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-09.tx
t
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.tx
t

These versions incorporate comments resulting from implementation.
The authors believe that all comments from the working group have been
addressed, dating back to June 2003 when version 00 of the terminology
draft was published.

These drafts are entering the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
At this time, we are seeking volunteers to complete the BMWG Last Call
Review Template in the message of 4/27/04 titled, "Refinements to the
BMWG Last Call Process".
Volunteers should contact acmorton <at> att.com A copy of the template is
available here:
http://home.comcast.net/~acmacm/IDcheck/LastCallTemplate.txt

Whether or not you decide to volunteer for the directed review, please
*read* and express your opinion on whether or not this Internet-Draft
should be given to the Area Directors for consideration as an
Informational RFC.  Send your comments to this list or acmorton <at> att.com 

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 1 comment |
headers
McQuaid, Jim | 7 Nov 2006 17:49
Favicon

RE: WG Last Call on Accelerated Stress Benchmarking Drafts

I think that the distinction between "devices,"  "protocols," and
"services" becomes impossible and / or metaphysical at some point.  IP
routing and switching are behaviors of network elements that may the
service of a network possible.  Perhaps the real issue is the layer at
which a service is provided.

cheers,

Jim McQuaid

NetQoS

-----Original Message-----
From: Arun Gandhi [mailto:gandhi <at> juniper.net] 
Sent: Tuesday, November 07, 2006 11:22 AM
To: 'Poretsky, Scott'; 'Al Morton'; bmwg <at> ietf.org; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Hi Scott,

>> The BMWG makes it a point to not benchmark 'services', so I recommend
>> we stick with use of the word 'protocols' and so not replace it with
>> 'services'. 

> I respectfully disagree as based on this particular work item
(document's
> motivation), I'm still of the opinion that we are not characterizing
> different protocols together but in fact different services with
protocols
> per se overlaid on a SUT. Inevitably (and as nicely explained), a
router
> in an operational network is configured with multiple protocols
(ospf,bgp,
> mpls, multicast, etc.) and services (such as vpns, vpls, policies,
etc.)
> based on the role it is playing in that network. Also, there are other
> features/services such as graceful-restart (GRS), non-stop routing,
ISSU,
> and so on that may be turned on as well.

Also, here's what I just found on the BMWG charter (section 'Description
of
Working Group')...

"The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services 
                                                ^^^^^^^^^^^^^^^^^^^ 
that are built from these technologies.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results."

Regards,
Arun Gandhi.

-----Original Message-----
From: Arun Gandhi [mailto:gandhi <at> juniper.net] 
Sent: Tuesday, November 07, 2006 10:20 AM
To: 'Poretsky, Scott'; 'Al Morton'; bmwg <at> ietf.org; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Hi Scott,

> The terms 'black-box' and 'white-box' have been frequently used in the
> BMWG over the past 10 years. I would be happy to use this work item as
> the place to define them. The focus of this work is black-box 
> benchmarking, consistent with the BMWG charter. With this particular 
> work item the working group agreed there would be value in observing
> White-Box benchmarks so it was preferred to include that in an
> appendix.

I totally agree.

> In addition, I do not agree with your suggestion to use "number" as a
> unit. Specifically you recommended to replace unit "sessions" with
unit
> "number".  When answering the question "How many sessions?" could you
> answer 10 numbers or 10 sessions.  No change will be made.

A little correction... i was actually thinking "number of sessions".
Sorry
for the confusion. Is this agreeable?

> The BMWG makes it a point to not benchmark 'services', so I recommend
> we stick with use of the word 'protocols' and so not replace it with
> 'services'. 

I respectfully disagree as based on this particular work item
(document's
motivation), I'm still of the opinion that we are not characterizing
different protocols together but in fact different services with
protocols
per se overlaid on a SUT. Inevitably (and as nicely explained), a router
in
an operational network is configured with multiple protocols (ospf, bgp,
mpls, multicast, etc.) and services (such as vpns, vpls, policies, etc.)
based on the role it is playing in that network. Also, there are other
features/services such as graceful-restart (GRS), non-stop routing,
ISSU,
and so on that may be turned on as well.

Also, IMHO, under Section 3.3 we should add "Number of ifls/ifds" and
"Number of policies" in anyone of the four buckets (where it seems most
fit).

Lastly, just a thought/question...Under Section 3.3.3, is it possible to
actually explicitly outline a threshold based on the system/SUT
resources in
use after which a box can be deemed as "Unstable" (e.g. a router in an
operational network using 85% of the system resources will/should be
considered unstable)? 

Thanks,
Arun Gandhi.

-----Original Message-----
From: Poretsky, Scott [mailto:sporetsky <at> reefpoint.com]
Sent: Friday, October 20, 2006 12:46 PM
To: Arun Gandhi; Al Morton; bmwg <at> ietf.org; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Hi Arun,

I apologize that this response is so late.  

The BMWG makes it a point to not benchmark 'services', so I recommend we
stick with use of the word 'protocols' and so not replace it with
'services'.  The terms 'black-box' and 'white-box' have been frequently
used in the BMWG over the past 10 years.  I would be happy to use this
work item as the place to define them.  The focus of this work is
black-box benchmarking, consistent with the BMWG charter.  With this
particular work item the working group agreed there would be value in
observing White-Box benchmarks so it was preferred to include that in an
appendix.  At this time there is no plan to make further changes on
these topics unless others in the working group want to continue the
discussion.

In addition, I do not agree with your suggestion to use "number" as a
unit.  Specifically you recommended to replace unit "sessions" with unit
"number".  When answering the question "How many sessions?" could you
answer 10 numbers or 10 sessions.  No change will be made.

Also, there are 3 sentence changes you requested below for which I do
not see the value of those changes.  Maybe you could provide more
information.

Thanks for your thorough review of the documents.

Scott

-----Original Message-----
From: Arun Gandhi [mailto:gandhi <at> juniper.net] 
Sent: Tuesday, August 29, 2006 11:55 AM
To: 'Al Morton'; bmwg <at> ietf.org; Poretsky, Scott; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts
Importance: High

Hello Authors,

I am little late in responding wrt the open dates but I suppose my
feedback will get evaluated. I will also follow up with the feedback on
methodology document shortly.

<-- Section 1 (Introduction): 

In general I believe this draft should not just address multiple
Protocols operational simultaneously but "different services" 
being overlaid in a fully operational box.

> Routers in an operational network are simultaneously configured with
> multiple protocols and security policies while forwarding traffic and
> being managed.

Arun> IMHO, replace "protocols" with "services" as in ISP world (which
Arun> is the motivation for this draft) there may be different service
Arun> sets such as l3vpn, l2vpns, VPLS, etc. along with routing and 
Arun> signaling protocols configured.

> It is helpful to accelerate these network operational conditions so 
> that the router under test can be benchmarked with faster test 
> duration.

Arun> This needs revision/clarification.

> Multiple benchmarks are made for each Benchmark Plane during each 
> Phase.

Arun> Benchmarks are defined for each plane to be characterized in all 
Arun> phases.

> These benchmarks White Box benchmarks are provided in Appendix 1 for
> additional DUT behavior measurements.

Arun> Two points:
Arun> 1) This line needs revision.
Arun> 2) A new term "White" box is added without being explained. Also, 
Arun> the impression I get reading this draft is that it's a black box
Arun> test. Also, since the motivation is the benchmarking of a router
Arun> in ISP domain then for sure it is a black box testing. Regardless
Arun> we should monitor/characterize other system parameters such as 
Arun> "memory usage", "CPU utilization", "install/bring up times", and 
Arun> so on.

<-- Section 3.2.1 (Control Plane)

> The Control Plane defines the Configuration, Startup Conditions, and
> Instability Conditions of the control protocols. Control Plane 
> protocols may include routing protocols, multicast protocols, and 
> MPLS protocols.     

Arun> IMHO, this statement should be revised to be "The Control Plane 
Arun> defines the Configuration, Startup Conditions, and Instability
Arun> Conditions of the control plane protocols, which may include 
Arun> routing protocols, multicast protocols, and MPLS protocols.  

<-- Section 3.3.2.3 (Control Plane)

> Measurement units: sessions

Arun> IMHO, this should be replaced with "number" (as "sessions" in not
Arun> a measuring unit) 

Thanks,
Arun

-----Original Message-----
From: Al Morton [mailto:acmorton <at> att.com] 
Sent: Monday, July 17, 2006 9:31 AM
To: bmwg <at> ietf.org
Subject: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

BMWG:

A WG Last Call period for the Internet-Drafts on

Terminology for Accelerated Stress Benchmarking Methodology Guidelines
for Accelerated Stress Benchmarking will be open from 17 July 2006
through 28 August 2006 (6 weeks).

URLs for these Internet-Drafts are:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-09.tx
t
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.tx
t

These versions incorporate comments resulting from implementation.
The authors believe that all comments from the working group have been
addressed, dating back to June 2003 when version 00 of the terminology
draft was published.

These drafts are entering the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
At this time, we are seeking volunteers to complete the BMWG Last Call
Review Template in the message of 4/27/04 titled, "Refinements to the
BMWG Last Call Process".
Volunteers should contact acmorton <at> att.com A copy of the template is
available here:
http://home.comcast.net/~acmacm/IDcheck/LastCallTemplate.txt

Whether or not you decide to volunteer for the directed review, please
*read* and express your opinion on whether or not this Internet-Draft
should be given to the Area Directors for consideration as an
Informational RFC.  Send your comments to this list or acmorton <at> att.com 

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 0 comments |
headers
Arun Gandhi | 7 Nov 2006 16:20
Favicon

RE: WG Last Call on Accelerated Stress Benchmarking Drafts

Hi Scott,

> The terms 'black-box' and 'white-box' have been frequently used in the
> BMWG over the past 10 years. I would be happy to use this work item as
> the place to define them. The focus of this work is black-box 
> benchmarking, consistent with the BMWG charter. With this particular 
> work item the working group agreed there would be value in observing
> White-Box benchmarks so it was preferred to include that in an
> appendix.

I totally agree.

> In addition, I do not agree with your suggestion to use "number" as a
> unit. Specifically you recommended to replace unit "sessions" with unit
> "number".  When answering the question "How many sessions?" could you
> answer 10 numbers or 10 sessions.  No change will be made.

A little correction... i was actually thinking "number of sessions". Sorry
for the confusion. Is this agreeable?

> The BMWG makes it a point to not benchmark 'services', so I recommend
> we stick with use of the word 'protocols' and so not replace it with
> 'services'. 

I respectfully disagree as based on this particular work item (document's
motivation), I'm still of the opinion that we are not characterizing
different protocols together but in fact different services with protocols
per se overlaid on a SUT. Inevitably (and as nicely explained), a router in
an operational network is configured with multiple protocols (ospf, bgp,
mpls, multicast, etc.) and services (such as vpns, vpls, policies, etc.)
based on the role it is playing in that network. Also, there are other
features/services such as graceful-restart (GRS), non-stop routing, ISSU,
and so on that may be turned on as well.

Also, IMHO, under Section 3.3 we should add "Number of ifls/ifds" and
"Number of policies" in anyone of the four buckets (where it seems most
fit).

Lastly, just a thought/question...Under Section 3.3.3, is it possible to
actually explicitly outline a threshold based on the system/SUT resources in
use after which a box can be deemed as "Unstable" (e.g. a router in an
operational network using 85% of the system resources will/should be
considered unstable)? 

Thanks,
Arun Gandhi.

-----Original Message-----
From: Poretsky, Scott [mailto:sporetsky <at> reefpoint.com]
Sent: Friday, October 20, 2006 12:46 PM
To: Arun Gandhi; Al Morton; bmwg <at> ietf.org; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

Hi Arun,

I apologize that this response is so late.  

The BMWG makes it a point to not benchmark 'services', so I recommend we
stick with use of the word 'protocols' and so not replace it with
'services'.  The terms 'black-box' and 'white-box' have been frequently
used in the BMWG over the past 10 years.  I would be happy to use this
work item as the place to define them.  The focus of this work is
black-box benchmarking, consistent with the BMWG charter.  With this
particular work item the working group agreed there would be value in
observing White-Box benchmarks so it was preferred to include that in an
appendix.  At this time there is no plan to make further changes on
these topics unless others in the working group want to continue the
discussion.

In addition, I do not agree with your suggestion to use "number" as a
unit.  Specifically you recommended to replace unit "sessions" with unit
"number".  When answering the question "How many sessions?" could you
answer 10 numbers or 10 sessions.  No change will be made.

Also, there are 3 sentence changes you requested below for which I do
not see the value of those changes.  Maybe you could provide more
information.

Thanks for your thorough review of the documents.

Scott

-----Original Message-----
From: Arun Gandhi [mailto:gandhi <at> juniper.net] 
Sent: Tuesday, August 29, 2006 11:55 AM
To: 'Al Morton'; bmwg <at> ietf.org; Poretsky, Scott; shankar.rao <at> qwest.com
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts
Importance: High

Hello Authors,

I am little late in responding wrt the open dates but I suppose my
feedback will get evaluated. I will also follow up with the feedback on
methodology document shortly.

<-- Section 1 (Introduction): 

In general I believe this draft should not just address multiple
Protocols operational simultaneously but "different services" 
being overlaid in a fully operational box.

> Routers in an operational network are simultaneously configured with
> multiple protocols and security policies while forwarding traffic and
> being managed.

Arun> IMHO, replace "protocols" with "services" as in ISP world (which
Arun> is the motivation for this draft) there may be different service
Arun> sets such as l3vpn, l2vpns, VPLS, etc. along with routing and 
Arun> signaling protocols configured.

> It is helpful to accelerate these network operational conditions so 
> that the router under test can be benchmarked with faster test 
> duration.

Arun> This needs revision/clarification.

> Multiple benchmarks are made for each Benchmark Plane during each 
> Phase.

Arun> Benchmarks are defined for each plane to be characterized in all 
Arun> phases.

> These benchmarks White Box benchmarks are provided in Appendix 1 for
> additional DUT behavior measurements.

Arun> Two points:
Arun> 1) This line needs revision.
Arun> 2) A new term "White" box is added without being explained. Also, 
Arun> the impression I get reading this draft is that it's a black box
Arun> test. Also, since the motivation is the benchmarking of a router
Arun> in ISP domain then for sure it is a black box testing. Regardless
Arun> we should monitor/characterize other system parameters such as 
Arun> "memory usage", "CPU utilization", "install/bring up times", and 
Arun> so on.

<-- Section 3.2.1 (Control Plane)

> The Control Plane defines the Configuration, Startup Conditions, and
> Instability Conditions of the control protocols. Control Plane 
> protocols may include routing protocols, multicast protocols, and 
> MPLS protocols.     

Arun> IMHO, this statement should be revised to be "The Control Plane 
Arun> defines the Configuration, Startup Conditions, and Instability
Arun> Conditions of the control plane protocols, which may include 
Arun> routing protocols, multicast protocols, and MPLS protocols.  

<-- Section 3.3.2.3 (Control Plane)

> Measurement units: sessions

Arun> IMHO, this should be replaced with "number" (as "sessions" in not
Arun> a measuring unit) 

Thanks,
Arun

-----Original Message-----
From: Al Morton [mailto:acmorton <at> att.com] 
Sent: Monday, July 17, 2006 9:31 AM
To: bmwg <at> ietf.org
Subject: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

BMWG:

A WG Last Call period for the Internet-Drafts on

Terminology for Accelerated Stress Benchmarking Methodology Guidelines
for Accelerated Stress Benchmarking will be open from 17 July 2006
through 28 August 2006 (6 weeks).

URLs for these Internet-Drafts are:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-09.tx
t
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.tx
t

These versions incorporate comments resulting from implementation.
The authors believe that all comments from the working group have been
addressed, dating back to June 2003 when version 00 of the terminology
draft was published.

These drafts are entering the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
At this time, we are seeking volunteers to complete the BMWG Last Call
Review Template in the message of 4/27/04 titled, "Refinements to the
BMWG Last Call Process".
Volunteers should contact acmorton <at> att.com A copy of the template is
available here:
http://home.comcast.net/~acmacm/IDcheck/LastCallTemplate.txt

Whether or not you decide to volunteer for the directed review, please
*read* and express your opinion on whether or not this Internet-Draft
should be given to the Area Directors for consideration as an
Informational RFC.  Send your comments to this list or acmorton <at> att.com 

_______________________________________________
bmwg mailing list
bmwg <at> ietf.org
https://www1.ietf.org/mailman/listinfo/bmwg
Permalink | Reply | 2 comments |
headers
Aamer Akhter (aakhter | 7 Nov 2006 21:56
Picon
Favicon

Re: I-D ACTION:draft-sdry-bmwg-mvpnscale-00.txt


BMWG folks,

This is a great effort and results from the document would be very
helpful for deployment assessment etc before fully going into MVPN. Many
SPs traditionally have not offered mcast services or may even not have
their core mcast enabled. I believe the sluggishness is in part due to
lack of results such as what the mvpnscale draft would provide. I fully
support this work for the BMWG.

I've already done an initial review, but will provide additional
feedback as time permits.

"Silvija Andrijic Dry (sdry)" <sdry <at> cisco.com> wrote in message
news:<B5B7DAAD94DFDB41821DC302D96A4CE7026748A1 <at> xmb-rtp-215.amer.cisco.co
m>...
BMWG members,

Please note the new draft addressing MVPN scalability benchmarking. This
is 
a draft in support of a new BMWG work proposal and authors would
appreciate feedback and readiness to discuss this topic in San Diego.

Btw, draft will be presented at both BMWG and MBONED meetings in San
Diego.

Thx,

Silvija

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

	Title		: Multicast VPN Scalability Benchmarking
	Author(s)	: S. Dry, et al.
	Filename	: draft-sdry-bmwg-mvpnscale-00.txt
	Pages		: 43
	Date		: 2006-10-12
	
    Multicast VPN (MVPN) is a service deployed by VPN service providers
    to enable their customers to use IP multicast applications over
VPNs.
    With the increased popularity the scalability of deploying such a
    service is becoming of a great interest. This document defines
    standard metric and test methodology for characterizing and
comparing
    control plane MVPN scalability of Provider Edge (PE) devices that
    implement MVPN service.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-sdry-bmwg-mvpnscale-00.txt

To remove yourself from the I-D Announcement list, send a message to
i-d-announce-request <at> ietf.org with the word unsubscribe in the body of
the message.
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the
username "anonymous" and a password of your e-mail address. After
logging in, type "cd internet-drafts" and then "get
draft-sdry-bmwg-mvpnscale-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html or
ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	mailserv <at> ietf.org.
In the body type:
	"FILE /internet-drafts/draft-sdry-bmwg-mvpnscale-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail
readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

Content-Type: text/plain
Content-ID: <2006-10-12145922.I-D <at> ietf.org>

ENCODING mime
FILE /internet-drafts/draft-sdry-bmwg-mvpnscale-00.txt

<ftp://ftp.ietf.org/internet-drafts/draft-sdry-bmwg-mvpnscale-00.txt>
_______________________________________________
I-D-Announce mailing list
I-D-Announce <at> ietf.org
https://www1.ietf.org/mailman/listinfo/i-d-announce
----------

--

-- 
Aamer Akhter / aa <at> cisco.com
Ent & Commercial Systems, cisco Systems
Permalink | Reply | 1 comment |

Gmane