A. Jean Mahoney | 26 May 22:57 2016

Assignments for the 2016-06-02 Telechat

Hi all,

The following reviewers have assignments:

Reviewer          LC end    Draft
-------------------------------------------------------------------------
Christer Holmberg 16-05-12  draft-ietf-dime-e2e-sec-req-04 **

Elwyn Davies      16-05-16  draft-ietf-manet-rfc6779bis-05 **

Francis Dupont    16-05-23  draft-ietf-clue-data-model-schema-13

Fernando Gont     16-05-26  draft-hardie-rfc6455-iana-clarification-02

Joel Halpern      16-05-13  draft-sheffer-rfc6982bis-01 *

Meral Shirazipour 16-05-31  draft-ietf-geojson-02

Pete Resnick      16-06-01  draft-ietf-mile-rfc5070-bis-16

Roni Even         16-05-17  draft-leiba-cotton-iana-5226bis-12 *

* Earlier draft reviewed
** Already reviewed

I have made the assignments in the review tool:
http://art.tools.ietf.org/tools/art/genart/

And the assignments are captured in the spreadsheets:
http://wiki.tools.ietf.org/dav/genart/gen-art.html
(Continue reading)

A. Jean Mahoney | 26 May 22:53 2016

A *new* batch of IETF LC reviews - 2016-05-26

Hi all,

The following reviewers have assignments:

Reviewer        LC end      Draft
---------------------------------------------------------------------
Dan Romascanu   2016-06-06  draft-ietf-rtgwg-bgp-routing-large-dc-10 *

Matt Miller     2016-06-01  draft-ietf-avtext-splicing-notification **

Peter Yee       2016-06-03  draft-ietf-dmarc-interoperability-15

Ralph Droms     2016-06-07  draft-ietf-idr-ix-bgp-route-server-10

Ron Bonica      2016-06-08
                       draft-ietf-ccamp-additional-signal-type-g709v3-04

Roni Even       2016-06-02  draft-ietf-sidr-rfc6485bis-05 **

Russ Housley    2016-06-22  draft-ietf-forces-interfelfb-04

*  Already reviewed
** 2nd LC

I have made the assignments in the review tool:
http://art.tools.ietf.org/tools/art/genart/

And the assignments are captured in the spreadsheets:
http://wiki.tools.ietf.org/dav/genart/gen-art.html
http://wiki.tools.ietf.org/dav/genart/gen-art-by-reviewer.html
(Continue reading)

Matt Miller (mamille2 | 25 May 02:29 2016
Picon
Gravatar

Review of draft-ietf-mmusic-msid-13

I am the assigned Gen-ART reviewer for this draft. The General Area Review
Team (Gen-ART) reviews all IETF documents being processed by the IESG for
the IETF Chair.  Please treat these comments just like any other last call
comments.

For more information, please see the FAQ at

<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Document: draft-ietf-mmusic-msid-13
Reviewer: Matthew A. Miller
Review Date: 2016-05-24
IETF LC End Date: 2016-05-24
IESG Telechat date: N/A

Summary:

This document is ready for publication as a Proposed Standard once
some minor editorial nits are address.

Major issues:  NONE

Minor issues:  NONE

Nits/editorial comments:

* idnits is complaining that something smells of code that isn't bounded.
However, I think it is clear that the "suspicious text" is the detailed
SDP example, and is already clearly noted as such.

(Continue reading)

Elwyn Davies | 25 May 01:23 2016
Picon

Gen-art telechat review of draft-ietf-manet-rfc6779bis-06

I am the assigned Gen-ART reviewer for this draft. The General Area Review Team (Gen-ART) reviews all IETF documents being processed by the IESG for the IETF Chair. Please wait for direction from your document shepherd or AD before posting a new version of the draft. For more information, please see the FAQ at <http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>. Document: draft-ietf-manet-rfc6779bis-06.txt Reviewer: Elwyn Davies Review Date: 2016/05/25 IETF LC End Date: 2016/05/16 IESG Telechat date: 2016/06/02 Summary: Ready. Thanks for tweaking the editorials mentioned in my last call review.
_______________________________________________
Gen-art mailing list
Gen-art <at> ietf.org
https://www.ietf.org/mailman/listinfo/gen-art
Roni Even | 24 May 09:11 2016
Picon

Gen-ART LC review of draft-leiba-cotton-iana-5226bis

 

Sorry for the late feedback

 

I am the assigned Gen-ART reviewer for this draft. For background on Gen-ART, please see the FAQ at <http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Please resolve these comments along with any other Last Call comments you may receive.

Document:  draft-leiba-cotton-iana-5226bis-12

Reviewer: Roni Even

Review Date:2016–5-24

IETF LC End Date: 2016–5-17

IESG Telechat date:

 

Summary: This draft is ready for publication as a BCP  RFC.

 

 

Major issues:

 

Minor issues:

 

I am wondering about the lack of normative language for example in section 4.11

   “When reviewing a document that asks IANA to create a new registry or change a registration policy to any policy more stringent than Expert Review or Specification Required, the IESG should ask for justification to ensure that more relaxed policies have been considered and that the strict policy is the right one.”

Is the “should” normative here?

 Nits/editorial comments:

 

 

_______________________________________________
Gen-art mailing list
Gen-art <at> ietf.org
https://www.ietf.org/mailman/listinfo/gen-art
A. Jean Mahoney | 19 May 23:54 2016

A *new* batch of IETF LC reviews - 2016-05-19

Hi all,

The following reviewers have assignments:

Reviewer          LC end      Draft
---------------------------------------------------------------------
Meral Shirazipour 2016-05-31  draft-ietf-geojson-03 *

Paul Kyzivat      2016-05-31  draft-ietf-tsvwg-rfc5405bis-13

Pete Resnick      2016-06-01  draft-ietf-mile-rfc5070-bis-21 *

* On the 6/2 telechat

I have made the assignments in the review tool:
http://art.tools.ietf.org/tools/art/genart/

And the assignments are captured in the spreadsheets:
http://wiki.tools.ietf.org/dav/genart/gen-art.html
http://wiki.tools.ietf.org/dav/genart/gen-art-by-reviewer.html

The updated template is included below.

Thanks,

Jean

-------------------------------------------------------------------

I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed
by the IESG for the IETF Chair.  Please treat these comments just
like any other last call comments.

For more information, please see the FAQ at

<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Document:
Reviewer:
Review Date:
IETF LC End Date:
IESG Telechat date: (if known)

Summary:

Major issues:

Minor issues:

Nits/editorial comments:
Ronald Bonica | 19 May 16:22 2016
Picon

Gen-ART review for draft-ietf-aqm-pie

I am the assigned Gen-ART reviewer for this draft. For background on Gen-ART, please see the FAQ at <http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>

Document:                                      draft-ietf-aqm-pie
Reviewer:                                        Ron Bonica
Review Date:                                  2016-05-19
IETF LC End Date:                          2016-05-04
IETF Telechat Date:                      2016-05-19

Summary:          This document is ready for publication.

It appears to be a reasonable experiment with adaptive queuing. (Full disclosure: I am not an expert in this area.)

Major Issues: None

Minor Issues: None

Editorial Issues: None
Kuhn Nicolas | 18 May 17:55 2016
Picon

Re: Gen-ART review of draft-ietf-aqm-eval-guidelines-11

Dear Ralph, 

For your information, we have been working on a -12 version, which diffs with the v11 are attached to this email.
We can push a new ID after the telechat tomorrow if you think it is necessary.
On top of many small things, we have adapted the normative throughout the whole document (but , kept it for
the conclusion section, since someone who want to characterize an AQM may want to be sure on what scenarios
are more important. The normative language induces some weighs on the different scenarios. I hope you re
fine with it.)

Kind regards,

Nicolas

-----Message d'origine-----
De : Ralph Droms (rdroms) [mailto:rdroms <at> cisco.com] 
Envoyé : jeudi 28 avril 2016 21:23
À : gen-art <at> ietf.org
Cc : draft-ietf-aqm-eval-guidelines.all <at> tools.ietf.org
Objet : Gen-ART review of draft-ietf-aqm-eval-guidelines-11

I am the assigned Gen-ART reviewer for this draft. The General Area Review Team (Gen-ART) reviews all IETF
documents being processed by the IESG for the IETF Chair.  Please treat these comments just like any other
last call comments.

For more information, please see the FAQ at

<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Document: draft-ietf-aqm-eval-guidelines-11
Reviewer: Ralph Droms
Review Date: 2016-04-28
IETF LC End Date: 2016-05-04
IESG Telechat date: 2016-05-19

Summary: This draft is on the right track but has open issues, described in the review.

In general, I think the document could be read, implemented and used to generate useful characterizations
of AQM schemes.  However, the motivations for some of the measurements and scenarios seems weak to me,
which might compromise the weight given to the conclusions drawn from the guidelines.

Major issues:

None.  However, the list of minor issues and nits, taken together, could be considered a major issue to be
resolved before publication.

Minor issues:

I often react to the use of RFC 2119 language in an Informational document by asking is that language really
necessary?  I'll ask the question here: in the context of this Informational document, which appears to be
entirely advisory in providing guidelines, what does the use of RFC 2119 "requirements language" add to
the meaning of the document.

Figure 1 is not clear to me.  Where are the physical links and interfaces?  Are there multiple physical
senders and receivers or are "senders A" instantiated on a single host (does it make a difference)?  Are
there static-sized buffers for each interface or do all the buffers share one memory space?

In section 3.1, is there a need to say something about the relative capacities of the various links and the
rates at which the various flows generate traffic?

I would have trouble following the guidelines set out in section 4.3.1.  I can understand the need for
consideration of the tunable control parameters when comparing different AQM schemes.  However, I don't
know what "comparable" means for control parameters that are likely quite different between AQM
schemes.  I also think one would want to compare optimal control settings for the different schemes, to
compare best-case performance.  Or, for AQM schemes whose performance is highly dependent on
operational conditions, one might want to compare settings that are sub-optimal for any particular test
condition but that give better performance over a wide range of conditions.

Section 4.4 seems to give advice to the AQM designer rather than describe guidelines for
characterization.  Section 4.4 should either be rewritten to give guidelines for structuring
measurements to account for varying packet sizes or the section should be elided.

In section 4.5, what is the motivation for giving the advice about ECN to AQM designers?  I can understand
that ECN will have affect the impact of AQM, but for this document I think the section should focus on
measurement guidlines that account for that impact.

The specific topology in section 10 does not seem well-motivated to me.  Why is router R with no AQM included
in the topology?  The choice of measurements is similarly not well-motivated.  Why would it not be of
interest to run all the tests described earlier in the document?

Nits/editorial comments:

There are several instances of the word "advice" which should be replaced with "advise"; e.g., in section 2.3.

Last sentence of the abstract: I don't get the meaning of "precautionary characterizations of AQM
schemes".  I recommend that the phrase be reworded

Section 1, first paragraph: The last sentence doesn't follow the rest of the paragraph and I recommend that
it be elided.

Section 1, third paragraph: This text is redundant with the text in the Glossary section:

   When speaking of a specific queue in this
   document, "buffer occupancy" refers to the amount of data (measured
   in bytes or packets) that are in the queue, and the "maximum buffer
   size" refers to the maximum buffer occupancy.

Section 1, third paragraph:

OLD:

   In real
   implementations of switches, a global memory is often shared between
   the available devices, and thus, the maximum buffer size may vary
   over the time.

NEW:

   In switches and routers, a global memory space is often shared
   between the available interfaces, and thus, the maximum buffer size
   for any given interface may vary over the time.

Section 1, fifth paragraph, last sentence: Is this document just concerned with "deployability" or more
generally with "applicability, performance and deployability"?

Section 1.1, first paragraph: Would it be helpful to qualify "goodput" as "goodput in individual flows",
to contrast with "goodput at a router"?  If "goodput" is well-known in this community to be "flow goodput",
no change is needed.

Section 1.1, second paragraph: What is "BDP", as in "BDP-sized buffer"?

Section 1.1, third paragraph, first sentence: "environment" is unclear; perhaps "deployment scenario"
or "use case"?  I'm not sure what was meant.

Section 1.3: for completeness, a definition of "latency" would be useful.

Section 2, first paragraph: I'm going to sound pedantic here, but it seems to me this section is not just
intended "to better quantify (1) the reduction of latency, (2) maximization of goodput and (3) the
trade-off between these two" but also to define other performance metrics that get to the goal from the
abstract of describing "various criteria for performing precautionary characterizations of AQM schemes"

Section 2, last paragraph: What is "application-limited traffic"?  Later, what is "non
application-limited" traffic?

Section 2.2: s/of/and/

Section 2.5, first paragraph: s/induces/requires/  ???

Section 2.6, second paragraph is just not clear to me.  In particular, what is the antecedent of "they":

   The introduction of an AQM scheme would impact these
   metrics (end-to-end latency and jitter) and therefore they should be
   considered in the end-to-end evaluation of performance."

Section 5.1: is this new text correct?

OLD:

   The transmission of the non application-limited flow must start
   before the transmission of the application-limited flow and only
   after the steady state has been reached by non application-limited
   flow.

NEW:

   The transmission of the non application-limited flow must start
   first and the transmission of the application-limited flow starts
   after the non application-limited flow has reached steady state.

Throughout section 5, the text like "the graph described in Section 2.7 could be generated" seems redundant.

Section 6.1, second paragraph: I don't understand the last sentence.  What is the "type of fairness" that
the AQM scheme might affect?

Section 6.2, second paragraph: The words up to the first period don't constitute a sentence.

In section 7.2, what is "IW10"?  Why are the particular traffic flow chosen in figure 2?  s/could/should/ in
the last paragraph?

< draft-ietf-aqm-eval-guidelines-11.txt   draft-ietf-aqm-eval-guidelines-12.txt >skipping to change at page 6, line 12 ¶ skipping to change at page 6, line 22 ¶skipping to change at page 7, line 19 ¶ skipping to change at page 7, line 42 ¶skipping to change at page 7, line 42 ¶ skipping to change at page 8, line 17 ¶skipping to change at page 9, line 44 ¶ skipping to change at page 10, line 20 ¶skipping to change at page 11, line 42 ¶ skipping to change at page 12, line 39 ¶skipping to change at page 17, line 33 ¶ skipping to change at page 18, line 27 ¶skipping to change at page 19, line 16 ¶ skipping to change at page 20, line 10 ¶skipping to change at page 21, line 22 ¶ skipping to change at page 22, line 16 ¶skipping to change at page 22, line 8 ¶ skipping to change at page 22, line 50 ¶skipping to change at page 23, line 9 ¶ skipping to change at page 23, line 47 ¶skipping to change at page 24, line 48 ¶ skipping to change at page 25, line 39 ¶skipping to change at page 25, line 44 ¶ skipping to change at page 26, line 36 ¶skipping to change at page 26, line 26 ¶ skipping to change at page 27, line 20 ¶skipping to change at page 28, line 5 ¶ skipping to change at page 29, line 5 ¶skipping to change at page 28, line 50 ¶ skipping to change at page 29, line 50 ¶skipping to change at page 31, line 21 ¶ skipping to change at page 32, line 21 ¶skipping to change at page 33, line 31 ¶ skipping to change at page 33, line 31 ¶ End of changes. 126 change blocks. 331 lines changed or deleted 347 lines changed or added
Internet Engineering Task Force N. Kuhn, Ed. Internet Engineering Task Force N. Kuhn, Ed.
Internet-Draft CNES, Telecom Bretagne Internet-Draft CNES, Telecom Bretagne
Intended status: Informational P. Natarajan, Ed. Intended status: Informational P. Natarajan, Ed.
Expires: August 4, 2016 Cisco Systems Expires: November 19, 2016 Cisco Systems
N. Khademi, Ed. N. Khademi, Ed.
University of Oslo University of Oslo
D. Ros D. Ros
Simula Research Laboratory AS Simula Research Laboratory AS
February 2016 May 18, 2016
AQM Characterization Guidelines AQM Characterization Guidelines
draft-ietf-aqm-eval-guidelines-11 draft-ietf-aqm-eval-guidelines-12
Abstract Abstract
Unmanaged large buffers in today's networks have given rise to a slew Unmanaged large buffers in today's networks have given rise to a slew
of performance issues. These performance issues can be addressed by of performance issues. These performance issues can be addressed by
some form of Active Queue Management (AQM) mechanism, optionally in some form of Active Queue Management (AQM) mechanism, optionally in
combination with a packet scheduling scheme such as fair queuing. combination with a packet scheduling scheme such as fair queuing.
This document describes various criteria for performing precautionary This document describes various criteria for performing
characterizations of AQM schemes. characterizations of AQM schemes.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 4, 2016. This Internet-Draft will expire on November 19, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Goals of this document . . . . . . . . . . . . . . . . . 5 1.1. Reducing the latency and maximizing the goodput . . . . . 5
1.2. Requirements Language . . . . . . . . . . . . . . . . . . 6 1.2. Goals of this document . . . . . . . . . . . . . . . . . 5
1.3. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 6
2. End-to-end metrics . . . . . . . . . . . . . . . . . . . . . 6 1.4. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 6
2. End-to-end metrics . . . . . . . . . . . . . . . . . . . . . 7
2.1. Flow completion time . . . . . . . . . . . . . . . . . . 7 2.1. Flow completion time . . . . . . . . . . . . . . . . . . 7
2.2. Flow start up time . . . . . . . . . . . . . . . . . . . 7 2.2. Flow start up time . . . . . . . . . . . . . . . . . . . 8
2.3. Packet loss . . . . . . . . . . . . . . . . . . . . . . . 7 2.3. Packet loss . . . . . . . . . . . . . . . . . . . . . . . 8
2.4. Packet loss synchronization . . . . . . . . . . . . . . . 8 2.4. Packet loss synchronization . . . . . . . . . . . . . . . 9
2.5. Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.5. Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.6. Latency and jitter . . . . . . . . . . . . . . . . . . . 9 2.6. Latency and jitter . . . . . . . . . . . . . . . . . . . 10
2.7. Discussion on the trade-off between latency and goodput . 10 2.7. Discussion on the trade-off between latency and goodput . 10
3. Generic setup for evaluations . . . . . . . . . . . . . . . . 10 3. Generic setup for evaluations . . . . . . . . . . . . . . . . 11
3.1. Topology and notations . . . . . . . . . . . . . . . . . 11 3.1. Topology and notations . . . . . . . . . . . . . . . . . 11
3.2. Buffer size . . . . . . . . . . . . . . . . . . . . . . . 12 3.2. Buffer size . . . . . . . . . . . . . . . . . . . . . . . 13
3.3. Congestion controls . . . . . . . . . . . . . . . . . . . 12 3.3. Congestion controls . . . . . . . . . . . . . . . . . . . 13
4. Methodology, Metrics, AQM Comparisons, Packet Sizes, 4. Methodology, Metrics, AQM Comparisons, Packet Sizes,
Scheduling and ECN . . . . . . . . . . . . . . . . . . . . . 13 Scheduling and ECN . . . . . . . . . . . . . . . . . . . . . 14
4.1. Methodology . . . . . . . . . . . . . . . . . . . . . . . 13 4.1. Methodology . . . . . . . . . . . . . . . . . . . . . . . 14
4.2. Comments on metrics measurement . . . . . . . . . . . . . 13 4.2. Comments on metrics measurement . . . . . . . . . . . . . 14
4.3. Comparing AQM schemes . . . . . . . . . . . . . . . . . . 14 4.3. Comparing AQM schemes . . . . . . . . . . . . . . . . . . 15
4.3.1. Performance comparison . . . . . . . . . . . . . . . 14 4.3.1. Performance comparison . . . . . . . . . . . . . . . 15
4.3.2. Deployment comparison . . . . . . . . . . . . . . . . 15 4.3.2. Deployment comparison . . . . . . . . . . . . . . . . 16
4.4. Packet sizes and congestion notification . . . . . . . . 15 4.4. Packet sizes and congestion notification . . . . . . . . 16
4.5. Interaction with ECN . . . . . . . . . . . . . . . . . . 15 4.5. Interaction with ECN . . . . . . . . . . . . . . . . . . 16
4.6. Interaction with Scheduling . . . . . . . . . . . . . . . 16 4.6. Interaction with Scheduling . . . . . . . . . . . . . . . 17
5. Transport Protocols . . . . . . . . . . . . . . . . . . . . . 16 5. Transport Protocols . . . . . . . . . . . . . . . . . . . . . 17
5.1. TCP-friendly sender . . . . . . . . . . . . . . . . . . . 17 5.1. TCP-friendly sender . . . . . . . . . . . . . . . . . . . 18
5.1.1. TCP-friendly sender with the same initial congestion 5.1.1. TCP-friendly sender with the same initial congestion
window . . . . . . . . . . . . . . . . . . . . . . . 17 window . . . . . . . . . . . . . . . . . . . . . . . 18
5.1.2. TCP-friendly sender with different initial congestion 5.1.2. TCP-friendly sender with different initial congestion
windows . . . . . . . . . . . . . . . . . . . . . . . 17 windows . . . . . . . . . . . . . . . . . . . . . . . 18
5.2. Aggressive transport sender . . . . . . . . . . . . . . . 18 5.2. Aggressive transport sender . . . . . . . . . . . . . . . 19
5.3. Unresponsive transport sender . . . . . . . . . . . . . . 18 5.3. Unresponsive transport sender . . . . . . . . . . . . . . 19
5.4. Less-than Best Effort transport sender . . . . . . . . . 19 5.4. Less-than Best Effort transport sender . . . . . . . . . 20
6. Round Trip Time Fairness . . . . . . . . . . . . . . . . . . 19 6. Round Trip Time Fairness . . . . . . . . . . . . . . . . . . 20
6.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 19 6.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 20
6.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 20 6.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 21
6.3. Metrics to evaluate the RTT fairness . . . . . . . . . . 20 6.3. Metrics to evaluate the RTT fairness . . . . . . . . . . 21
7. Burst Absorption . . . . . . . . . . . . . . . . . . . . . . 21
7. Burst Absorption . . . . . . . . . . . . . . . . . . . . . . 20 7.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 21
7.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 20 7.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 22
7.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 21 8. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 23
8. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 22 8.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 23
8.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 22
8.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 23 8.2. Recommended tests . . . . . . . . . . . . . . . . . . . . 23
8.2.1. Definition of the congestion Level . . . . . . . . . 23 8.2.1. Definition of the congestion Level . . . . . . . . . 24
8.2.2. Mild congestion . . . . . . . . . . . . . . . . . . . 23 8.2.2. Mild congestion . . . . . . . . . . . . . . . . . . . 24
8.2.3. Medium congestion . . . . . . . . . . . . . . . . . . 23 8.2.3. Medium congestion . . . . . . . . . . . . . . . . . . 24
8.2.4. Heavy congestion . . . . . . . . . . . . . . . . . . 24 8.2.4. Heavy congestion . . . . . . . . . . . . . . . . . . 24
8.2.5. Varying the congestion level . . . . . . . . . . . . 24 8.2.5. Varying the congestion level . . . . . . . . . . . . 25
8.2.6. Varying available capacity . . . . . . . . . . . . . 24 8.2.6. Varying available capacity . . . . . . . . . . . . . 25
8.3. Parameter sensitivity and stability analysis . . . . . . 25 8.3. Parameter sensitivity and stability analysis . . . . . . 26
9. Various Traffic Profiles . . . . . . . . . . . . . . . . . . 26 9. Various Traffic Profiles . . . . . . . . . . . . . . . . . . 26
9.1. Traffic mix . . . . . . . . . . . . . . . . . . . . . . . 26 9.1. Traffic mix . . . . . . . . . . . . . . . . . . . . . . . 27
9.2. Bi-directional traffic . . . . . . . . . . . . . . . . . 26 9.2. Bi-directional traffic . . . . . . . . . . . . . . . . . 27
10. Multi-AQM Scenario . . . . . . . . . . . . . . . . . . . . . 27 10. Multi-AQM Scenario . . . . . . . . . . . . . . . . . . . . . 27
10.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 27 10.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 28
10.2. Details on the evaluation scenario . . . . . . . . . . . 27 10.2. Details on the evaluation scenario . . . . . . . . . . . 28
11. Implementation cost . . . . . . . . . . . . . . . . . . . . . 27 11. Implementation cost . . . . . . . . . . . . . . . . . . . . . 28
11.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 27 11.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 28
11.2. Recommended discussion . . . . . . . . . . . . . . . . . 28 11.2. Recommended discussion . . . . . . . . . . . . . . . . . 29
12. Operator Control and Auto-tuning . . . . . . . . . . . . . . 28 12. Operator Control and Auto-tuning . . . . . . . . . . . . . . 29
12.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 28 12.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . 29
12.2. Recommended discussion . . . . . . . . . . . . . . . . . 29 12.2. Recommended discussion . . . . . . . . . . . . . . . . . 30
13. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 29 13. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 30
14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 31
15. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 30 15. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 31
16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32
17. Security Considerations . . . . . . . . . . . . . . . . . . . 31 17. Security Considerations . . . . . . . . . . . . . . . . . . . 32
18. References . . . . . . . . . . . . . . . . . . . . . . . . . 31 18. References . . . . . . . . . . . . . . . . . . . . . . . . . 32
18.1. Normative References . . . . . . . . . . . . . . . . . . 31 18.1. Normative References . . . . . . . . . . . . . . . . . . 32
18.2. Informative References . . . . . . . . . . . . . . . . . 33 18.2. Informative References . . . . . . . . . . . . . . . . . 33
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 34 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35
1. Introduction 1. Introduction
Active Queue Management (AQM) [RFC7567] addresses the concerns Active Queue Management (AQM) [RFC7567] addresses the concerns
arising from using unnecessarily large and unmanaged buffers to arising from using unnecessarily large and unmanaged buffers to
improve network and application performance. Several AQM algorithms improve network and application performance. Several AQM algorithms
have been proposed in the past years, most notably Random Early have been proposed in the past years, most notably Random Early
Detection (RED), BLUE, and Proportional Integral controller (PI), and Detection (RED), BLUE, and Proportional Integral controller (PI), and
more recently CoDel [NICH2012] and PIE [PAN2013]. In general, these more recently CoDel [I-D.ietf-aqm-codel] and PIE [I-D.ietf-aqm-pie]
algorithms actively interact with the Transmission Control Protocol In general, these algorithms actively interact with the Transmission
(TCP) and any other transport protocol that deploys a congestion Control Protocol (TCP) and any other transport protocol that deploys
control scheme to manage the amount of data they keep in the network. a congestion control scheme to manage the amount of data they keep in
The available buffer space in the routers and switches should be the network. The available buffer space in the routers and switches
large enough to accommodate the short-term buffering requirements. should be large enough to accommodate the short-term buffering
requirements. AQM schemes aim at reducing buffer occupancy, and
AQM schemes aim at reducing buffer occupancy, and therefore the end- therefore the end-to-end delay. Some of these algorithms, notably
to-end delay. Some of these algorithms, notably RED, have also been RED, have also been widely implemented in some network devices.
widely implemented in some network devices. However, the potential However, the potential benefits of the RED scheme have not been
benefits of the RED scheme have not been realized since RED is realized since RED is reported to be usually turned off.
reported to be usually turned off. The main reason of this
reluctance to use RED in today's deployments comes from its
sensitivity to the operating conditions in the network and the
difficulty of tuning its parameters.
A buffer is a physical volume of memory in which a queue or set of A buffer is a physical volume of memory in which a queue or set of
queues are stored. When speaking of a specific queue in this queues are stored. When speaking of a specific queue in this
document, "buffer occupancy" refers to the amount of data (measured document, "buffer occupancy" refers to the amount of data (measured
in bytes or packets) that are in the queue, and the "maximum buffer in bytes or packets) that are in the queue, and the "maximum buffer
size" refers to the maximum buffer occupancy. In real size" refers to the maximum buffer occupancy. In switches and
implementations of switches, a global memory is often shared between routers, a global memory space is often shared between the available
the available devices, and thus, the maximum buffer size may vary interfaces, and thus, the maximum buffer size for any given interface
over the time. may vary over the time.
Bufferbloat [BB2011] is the consequence of deploying large unmanaged Bufferbloat [BB2011] is the consequence of deploying large unmanaged
buffers on the Internet -- the buffering has often been measured to buffers on the Internet -- the buffering has often been measured to
be ten times or hundred times larger than needed. Large buffer sizes be ten times or hundred times larger than needed. Large buffer sizes
in combination with TCP and/or unresponsive flows increases end-to- in combination with TCP and/or unresponsive flows increases end-to-
end delay. This results in poor performance for latency-sensitive end delay. This results in poor performance for latency-sensitive
applications such as real-time multimedia (e.g., voice, video, applications such as real-time multimedia (e.g., voice, video,
gaming, etc). The degree to which this affects modern networking gaming, etc). The degree to which this affects modern networking
equipment, especially consumer-grade equipment's, produces problems equipment, especially consumer-grade equipment's, produces problems
even with commonly used web services. Active queue management is even with commonly used web services. Active queue management is
thus essential to control queuing delay and decrease network latency. thus essential to control queuing delay and decrease network latency.
The Active Queue Management and Packet Scheduling Working Group (AQM The Active Queue Management and Packet Scheduling Working Group (AQM
WG) was chartered to address the problems with large unmanaged WG) was chartered to address the problems with large unmanaged
buffers in the Internet. Specifically, the AQM WG is tasked with buffers in the Internet. Specifically, the AQM WG is tasked with
standardizing AQM schemes that not only address concerns with such standardizing AQM schemes that not only address concerns with such
buffers, but also are robust under a wide variety of operating buffers, but also are robust under a wide variety of operating
conditions. This document provides characterization guidelines that conditions. This document provides characterization guidelines that
can be used to assess the deployability of an AQM, whether it is can be used to assess the applicability, performance and
candidate for standardization at IETF or not. deployability of an AQM, whether it is candidate for standardization
at IETF or not.
[RFC7567] separately describes the AQM algorithm implemented in a AQM algorithm implemented in a router can be separated from the
router from the scheduling of packets sent by the router. The rest scheduling of packets sent out by the router [RFC7567]. The rest of
of this memo refers to the AQM as a dropping/marking policy as a this memo refers to the AQM as a dropping/marking policy as a
separate feature to any interface scheduling scheme. This document separate feature to any interface scheduling scheme. This document
may be complemented with another one on guidelines for assessing may be complemented with another one on guidelines for assessing
combination of packet scheduling and AQM. We note that such a combination of packet scheduling and AQM. We note that such a
document will inherit all the guidelines from this document plus any document will inherit all the guidelines from this document plus any
additional scenarios relevant for packet scheduling such as flow additional scenarios relevant for packet scheduling such as flow
starvation evaluation or impact of the number of hash buckets. starvation evaluation or impact of the number of hash buckets.
1.1. Goals of this document 1.1. Reducing the latency and maximizing the goodput
The trade-off between reducing the latency and maximizing the goodput The trade-off between reducing the latency and maximizing the goodput
is intrinsically linked to each AQM scheme and is key to evaluating is intrinsically linked to each AQM scheme and is key to evaluating
its performance. Whenever possible, solutions ought to aim at both its performance. To ensure the safety deployment of an AQM, its
maximizing goodput and minimizing latency. Moreover, to ensure the behaviour should be assessed in a variety of scenarios. Whenever
safety deployment of an AQM, its behaviour should be assessed in a possible, solutions ought to aim at both maximizing goodput and
variety of scenarios. minimizing latency.
1.2. Goals of this document
This document recommends a generic list of scenarios against which an This document recommends a generic list of scenarios against which an
AQM proposal should be evaluated, considering both potential AQM proposal should be evaluated, considering both potential
performance gain and safety of deployment. The guidelines help to performance gain and safety of deployment. The guidelines help to
quantify performance of AQM schemes in terms of latency reduction, quantify performance of AQM schemes in terms of latency reduction,
goodput maximization and the trade-off between these two. The goodput maximization and the trade-off between these two. The
document presents central aspects of an AQM algorithm that should be document presents central aspects of an AQM algorithm that should be
considered whatever the context, such as burst absorption capacity, considered whatever the context, such as burst absorption capacity,
RTT fairness or resilience to fluctuating network conditions. The RTT fairness or resilience to fluctuating network conditions. The
guidelines also discuss methods to understand the various aspects guidelines also discuss methods to understand the various aspects
associated with safely deploying and operating the AQM scheme. Thus, associated with safely deploying and operating the AQM scheme. Thus,
one of the key objectives behind formulating the guidelines is to one of the key objectives behind formulating the guidelines is to
help ascertain whether a specific AQM is not only better than drop- help ascertain whether a specific AQM is not only better than drop-
tail (i.e. without AQM and with a BDP-sized buffer) but also safe to tail (i.e. without AQM and with a BDP-sized buffer) but also safe to
deploy: the guidelines can be used to compare several AQM proposals deploy: the guidelines can be used to compare several AQM proposals
with each other, and should be used to compare a proposal with drop- with each other, but should be used to compare a proposal with drop-
tail. tail.
This memo details generic characterization scenarios against which
any AQM proposal should be evaluated, irrespective of whether or not
an AQM is standardized by the IETF. This documents recommends the
relevant scenarios and metrics to be considered. The document
presents central aspects of an AQM algorithm that should be
considered whatever the context, such as burst absorption capacity,
RTT fairness or resilience to fluctuating network conditions.
These guidelines do not define and are not bound to a particular These guidelines do not define and are not bound to a particular
environment or evaluation toolset. Instead the guidelines can be deployment scenario or evaluation toolset. Instead the guidelines
used to assert the potential gain of introducing an AQM for the can be used to assert the potential gain of introducing an AQM for
particular environment, which is of interest to the testers. These the particular environment, which is of interest to the testers.
guidelines do not cover every possible aspect of a particular These guidelines do not cover every possible aspect of a particular
algorithm. These guidelines do not present context-dependent algorithm. These guidelines do not present context-dependent
scenarios (such as 802.11 WLANs, data-centers or rural broadband scenarios (such as 802.11 WLANs, data-centers or rural broadband
networks). To keep the guidelines generic, a number of potential networks). To keep the guidelines generic, a number of potential
router components and algorithms (such as DiffServ) are omitted. router components and algorithms (such as DiffServ) are omitted.
The goals of this document can thus be summarized as follows: The goals of this document can thus be summarized as follows:
o The present characterization guidelines provide a non-exhaustive o The present characterization guidelines provide a non-exhaustive
list of scenarios to help ascertain whether an AQM is not only list of scenarios to help ascertain whether an AQM is not only
better than drop-tail (with a BDP-sized buffer), but also safe to better than drop-tail (with a BDP-sized buffer), but also safe to
particular evaluation toolset and (2) can be used for various particular evaluation toolset and (2) can be used for various
deployment contexts; testers are free to select a toolset that is deployment contexts; testers are free to select a toolset that is
best suited for the environment in which their proposal will be best suited for the environment in which their proposal will be
deployed. deployed.
o The present characterization guidelines are intended to provide o The present characterization guidelines are intended to provide
guidance for better selecting an AQM for a specific environment; guidance for better selecting an AQM for a specific environment;
it is not required that an AQM proposal is evaluated following it is not required that an AQM proposal is evaluated following
these guidelines for its standardization. these guidelines for its standardization.
1.2. Requirements Language 1.3. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
1.3. Glossary 1.4. Glossary
o AQM: [RFC7567] separately describes the Active Queue Managment o application-limited traffic: a type of traffic that does not have
(AQM) algorithm implemented in a router from the scheduling of an unlimited amount of data to transmit.
packets sent by the router. The rest of this memo refers to the
AQM as a dropping/marking policy as a separate feature to any o AQM: the Active Queue Managment (AQM) algorithm implemented in a
interface scheduling scheme. router can be separated from the scheduling of packets sent by the
router. The rest of this memo refers to the AQM as a dropping/
marking policy as a separate feature to any interface scheduling
scheme [RFC7567].
o BDP: Bandwidth Delay Product.
o buffer: a physical volume of memory in which a queue or set of o buffer: a physical volume of memory in which a queue or set of
queues are stored. queues are stored.
o buffer occupancy: amount of data that are stored in a buffer, o buffer occupancy: amount of data that are stored in a buffer,
measured in bytes or packets. measured in bytes or packets.
o buffer size: maximum buffer occupancy, that is the maximum amount o buffer size: maximum buffer occupancy, that is the maximum amount
of data that may be stored in a buffer, measured in bytes or of data that may be stored in a buffer, measured in bytes or
packets. packets.
o IW10: TCP initial congestion window set to 10 packets.
o latency: one-way delay of packets across Internet paths. This
definition suits transport layer definition of the latency, that
shall not be confused with an application layer view of the
latency.
o goodput: goodput is defined as the number of bits per unit of time o goodput: goodput is defined as the number of bits per unit of time
forwarded to the correct destination minus any bits lost or forwarded to the correct destination minus any bits lost or
retransmitted [RFC2647]. retransmitted [RFC2647]. The goodput should be determined for
each flow and not for aggregates of flows.
o SQRT: the square root function. o SQRT: the square root function.
o ROUND: the round function. o ROUND: the round function.
2. End-to-end metrics 2. End-to-end metrics
End-to-end delay is the result of propagation delay, serialization End-to-end delay is the result of propagation delay, serialization
delay, service delay in a switch, medium-access delay and queuing delay, service delay in a switch, medium-access delay and queuing
delay, summed over the network elements along the path. AQM schemes delay, summed over the network elements along the path. AQM schemes
Some metrics listed in this section are not suited to every type of Some metrics listed in this section are not suited to every type of
traffic detailed in the rest of this document. It is therefore not traffic detailed in the rest of this document. It is therefore not
necessary to measure all of the following metrics: the chosen metric necessary to measure all of the following metrics: the chosen metric
may not be relevant to the context of the evaluation scenario (e.g., may not be relevant to the context of the evaluation scenario (e.g.,
latency vs. goodput trade-off in application-limited traffic latency vs. goodput trade-off in application-limited traffic
scenarios). Guidance is provided for each metric. scenarios). Guidance is provided for each metric.
2.1. Flow completion time 2.1. Flow completion time
The flow completion time is an important performance metric for the The flow completion time is an important performance metric for the
end-user when the flow size is finite. Considering the fact that an end-user when the flow size is finite. The definition of the flow
AQM scheme may drop/mark packets, the flow completion time is size may be source of contradictions, thus, this metric can consider
directly linked to the dropping/marking policy of the AQM scheme. a flow as a single file. Considering the fact that an AQM scheme may
This metric helps to better assess the performance of an AQM drop/mark packets, the flow completion time is directly linked to the
depending on the flow size. The Flow Completion Time (FCT) is dropping/marking policy of the AQM scheme. This metric helps to
related to the flow size (Fs) and the goodput for the flow (G) as better assess the performance of an AQM depending on the flow size.
follows: The Flow Completion Time (FCT) is related to the flow size (Fs) and
the goodput for the flow (G) as follows:
FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] ) FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] )
If this metric is used to evaluate the performance of web transfers, If this metric is used to evaluate the performance of web transfers,
it is suggested to rather consider the time needed to download all it is suggested to rather consider the time needed to download all
the objects that compose the web page, as this makes more sense in the objects that compose the web page, as this makes more sense in
terms of user experience than assessing the time needed to download terms of user experience than assessing the time needed to download
each object. each object.
2.2. Flow start up time 2.2. Flow start up time
The flow start up time is the time between the request has been sent The flow start up time is the time between the request has been sent
from the client and the server starts to transmit data. The amount from the client and the server starts to transmit data. The amount
terms of user experience than assessing the time needed to download terms of user experience than assessing the time needed to download
each object. each object.
2.2. Flow start up time 2.2. Flow start up time
The flow start up time is the time between the request has been sent The flow start up time is the time between the request has been sent
from the client and the server starts to transmit data. The amount from the client and the server starts to transmit data. The amount
of packets dropped by an AQM may seriously affect the waiting period of packets dropped by an AQM may seriously affect the waiting period
during which the data transfer has not started. This metric would during which the data transfer has not started. This metric would
specifically focus on the operations such as DNS lookups, TCP opens specifically focus on the operations such as DNS lookups, TCP opens
of SSL handshakes. and SSL handshakes.
2.3. Packet loss 2.3. Packet loss
Packet loss can occur en-route, this can impact the end-to-end Packet loss can occur en-route, this can impact the end-to-end
performance measured at receiver. performance measured at receiver.
The tester SHOULD evaluate loss experienced at the receiver using one The tester should evaluate loss experienced at the receiver using one
of the two metrics: of the two metrics:
o the packet loss ratio: this metric is to be frequently measured o the packet loss ratio: this metric is to be frequently measured
during the experiment. The long-term loss ratio is of interest during the experiment. The long-term loss ratio is of interest
for steady-state scenarios only; for steady-state scenarios only;
o the interval between consecutive losses: the time between two o the interval between consecutive losses: the time between two
losses is to be measured. losses is to be measured.
The packet loss ratio can be assessed by simply evaluating the loss The packet loss ratio can be assessed by simply evaluating the loss
ratio as a function of the number of lost packets and the total ratio as a function of the number of lost packets and the total
number of packets sent. This might not be easily done in laboratory number of packets sent. This might not be easily done in laboratory
testing, for which these guidelines advice the tester: testing, for which these guidelines advice the tester:
o to check that for every packet, a corresponding packet was o to check that for every packet, a corresponding packet was
received within a reasonable time, as explained in [RFC2680]. received within a reasonable time [RFC2680].
o to keep a count of all packets sent, and a count of the non- o to keep a count of all packets sent, and a count of the non-
duplicate packets received, as explained in the section 10 of duplicate packets received [RFC2544].
[RFC2544].
The interval between consecutive losses, which is also called a gap, The interval between consecutive losses, which is also called a gap,
is a metric of interest for VoIP traffic and, as a result, has been is a metric of interest for VoIP traffic [RFC3611].
further specified in [RFC3611].
2.4. Packet loss synchronization 2.4. Packet loss synchronization
One goal of an AQM algorithm is to help to avoid global One goal of an AQM algorithm is to help to avoid global
synchronization of flows sharing a bottleneck buffer on which the AQM synchronization of flows sharing a bottleneck buffer on which the AQM
operates ([RFC2309],[RFC7567]). The "degree" of packet-loss operates ([RFC2309],[RFC7567]). The "degree" of packet-loss
synchronization between flows SHOULD be assessed, with and without synchronization between flows should be assessed, with and without
the AQM under consideration. the AQM under consideration.
As discussed e.g., in [HASS2008], loss synchronization among flows Loss synchronization among flows may be quantified by several
may be quantified by several slightly different metrics that capture slightly different metrics that capture different aspects of the same
different aspects of the same issue. However, in real-world issue [HASS2008]. However, in real-world measurements the choice of
measurements the choice of metric could be imposed by practical metric could be imposed by practical considerations -- e.g., whether
considerations -- e.g., whether fine-grained information on packet fine-grained information on packet losses in the bottleneck available
losses in the bottleneck available or not. For the purpose of AQM or not. For the purpose of AQM characterization, a good candidate
characterization, a good candidate metric is the global metric is the global synchronization ratio, measuring the proportion
synchronization ratio, measuring the proportion of flows losing of flows losing packets during a loss event. This metric can be used
packets during a loss event. [JAY2006] used this metric in real- in real-world experiments to characterize synchronization along
world experiments to characterize synchronization along arbitrary arbitrary Internet paths [JAY2006].
Internet paths; the full methodology is described in [JAY2006].
If an AQM scheme is evaluated using real-life network environments, If an AQM scheme is evaluated using real-life network environments,
it is worth pointing out that some network events, such as failed it is worth pointing out that some network events, such as failed
link restoration may cause synchronized losses between active flows link restoration may cause synchronized losses between active flows
and thus confuse the meaning of this metric. and thus confuse the meaning of this metric.
2.5. Goodput 2.5. Goodput
The goodput has been defined in section 3.17 of [RFC2647] as the The goodput has been defined as the number of bits per unit of time
number of bits per unit of time forwarded to the correct destination forwarded to the correct destination interface, minus any bits lost
interface, minus any bits lost or retransmitted. This definition or retransmitted [RFC2647]. This definition requires that the test
induces that the test setup needs to be qualified to assure that it setup needs to be qualified to assure that it is not generating
is not generating losses on its own. losses on its own.
Measuring the end-to-end goodput provides an appreciation of how well Measuring the end-to-end goodput provides an appreciation of how well
an AQM scheme improves transport and application performance. The an AQM scheme improves transport and application performance. The
measured end-to-end goodput is linked to the dropping/marking policy measured end-to-end goodput is linked to the dropping/marking policy
of the AQM scheme -- e.g., the fewer the number of packet drops, the of the AQM scheme -- e.g., the fewer the number of packet drops, the
fewer packets need retransmission, minimizing the impact of AQM on fewer packets need retransmission, minimizing the impact of AQM on
transport and application performance. Additionally, an AQM scheme transport and application performance. Additionally, an AQM scheme
may resort to Explicit Congestion Notification (ECN) marking as an may resort to Explicit Congestion Notification (ECN) marking as an
initial means to control delay. Again, marking packets instead of initial means to control delay. Again, marking packets instead of
dropping them reduces the number of packet retransmissions and dropping them reduces the number of packet retransmissions and
may depend on other factors than the introduction of an AQM, such as may depend on other factors than the introduction of an AQM, such as
link layer losses due to external noise or corruption, fluctuating link layer losses due to external noise or corruption, fluctuating
bandwidths (802.11 WLANs), heavy congestion levels or transport bandwidths (802.11 WLANs), heavy congestion levels or transport
layer's rate reduction by congestion control mechanism. layer's rate reduction by congestion control mechanism.
2.6. Latency and jitter 2.6. Latency and jitter
The latency, or the one-way delay metric, is discussed in [RFC2679]. The latency, or the one-way delay metric, is discussed in [RFC2679].
There is a consensus on an adequate metric for the jitter, that There is a consensus on an adequate metric for the jitter, that
represents the one-way delay variations for packets from the same represents the one-way delay variations for packets from the same
flow: the Packet Delay Variation (PDV), detailed in [RFC5481], serves flow: the Packet Delay Variation (PDV) serves well all use cases
well all use cases. [RFC5481].
The end-to-end latency includes components other than just the The end-to-end latency includes components other than just the
queuing delay, such as the signal processing delay, transmission queuing delay, such as the signal processing delay, transmission
delay and the processing delay. Moreover, the jitter is caused by delay and the processing delay. Moreover, the jitter is caused by
variations in queuing and processing delay (e.g., scheduling variations in queuing and processing delay (e.g., scheduling
effects). The introduction of an AQM scheme would impact these effects). The introduction of an AQM scheme would impact end-to-end
metrics (end-to-end latency and jitter) and therefore they should be latency and jitter, and therefore these metrics should be considered
considered in the end-to-end evaluation of performance. in the end-to-end evaluation of performance.
2.7. Discussion on the trade-off between latency and goodput 2.7. Discussion on the trade-off between latency and goodput
The metrics presented in this section may be considered as explained The metrics presented in this section may be considered as explained
in the rest of this document, in order to discuss and quantify the in the rest of this document, in order to discuss and quantify the
trade-off between latency and goodput. trade-off between latency and goodput.
With regards to the goodput, and in addition to the long-term With regards to the goodput, and in addition to the long-term
stationary goodput value, it is RECOMMENDED to take measurements stationary goodput value, it is recommended to take measurements
every multiple of the minimum RTT (minRTT) between A and B. It is every multiple of the minimum RTT (minRTT) between A and B. It is
suggested to take measurements at least every K x minRTT (to smooth suggested to take measurements at least every K x minRTT (to smooth
out the fluctuations), with K=10. Higher values for K are encouraged out the fluctuations), with K=10. Higher values for K are encouraged
whenever it is more appropriate for the presentation of the results. whenever it is more appropriate for the presentation of the results.
The value for K may depend on the network's path characteristics. The value for K may depend on the network's path characteristics.
The measurement period MUST be disclosed for each experiment and when The measurement period must be disclosed for each experiment and when
results/values are compared across different AQM schemes, the results/values are compared across different AQM schemes, the
comparisons SHOULD use exactly the same measurement periods. With comparisons should use exactly the same measurement periods. With
regards to latency, it is RECOMMENDED to take the samples on per- regards to latency, it is recommended to take the samples on per-
packet basis whenever possible depending on the features provided by packet basis whenever possible depending on the features provided by
hardware/software and the impact of sampling itself on the hardware hardware/software and the impact of sampling itself on the hardware
performance. It is generally RECOMMENDED to provide at least 10 performance. It is generally recommended to provide at least 10
samples per RTT. samples per RTT.
From each of these sets of measurements, the cumulative density From each of these sets of measurements, the cumulative density
function (CDF) of the considered metrics SHOULD be computed. If the function (CDF) of the considered metrics should be computed. If the
considered scenario introduces dynamically varying parameters, considered scenario introduces dynamically varying parameters,
temporal evolution of the metrics could also be generated. For each temporal evolution of the metrics could also be generated. For each
scenario, the following graph may be generated: the x-axis shows scenario, the following graph may be generated: the x-axis shows
queuing delay (that is the average per-packet delay in excess of queuing delay (that is the average per-packet delay in excess of
minimum RTT), the y-axis the goodput. Ellipses are computed such as minimum RTT), the y-axis the goodput. Ellipses are computed such as
detailed in [WINS2014]: "We take each individual [...] run [...] as detailed in [WINS2014]: "We take each individual [...] run [...] as
one point, and then compute the 1-epsilon elliptic contour of the one point, and then compute the 1-epsilon elliptic contour of the
maximum-likelihood 2D Gaussian distribution that explains the points. maximum-likelihood 2D Gaussian distribution that explains the points.
[...] we plot the median per-sender throughput and queueing delay as [...] we plot the median per-sender throughput and queueing delay as
a circle. [...] The orientation of an ellipse represents the a circle. [...] The orientation of an ellipse represents the
| + | | | | + | | + | | | | + |
| | | | | | | | | | | | | | | |
| + | | | | + | | + | | | | + |
| SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y | | SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y |
+--------------+ +--------------+ +--------------+ +--------------+
Figure 1: Topology and notations Figure 1: Topology and notations
Figure 1 is a generic topology where: Figure 1 is a generic topology where:
o sender with different traffic characteristics (i.e., traffic o senders with different traffic characteristics (i.e., traffic
profiles) can be introduced; profiles) can be introduced;
o the timing of each flow could be different (i.e., when does each o the timing of each flow could be different (i.e., when does each
flow start and stop); flow start and stop);
o each traffic profile can comprise various number of flows; o each traffic profile can comprise various number of flows;
o each link is characterized by a couple (one-way delay, capacity); o each link is characterized by a couple (one-way delay, capacity);
o flows are generated at A and sent to B, sharing a bottleneck (the o flows are generated at A and sent to B, sharing a bottleneck (the
link between routers L and R); link between routers L and R);
o the tester SHOULD consider both scenarios of asymmetric and o the tester should consider both scenarios of asymmetric and
symmetric bottleneck links in terms of bandwidth. In case of symmetric bottleneck links in terms of bandwidth. In case of
asymmetric link, the capacity from senders to receivers is higher asymmetric link, the capacity from senders to receivers is higher
than the one from receivers to senders; the symmetric link than the one from receivers to senders; the symmetric link
scenario provides a basic understanding of the operation of the scenario provides a basic understanding of the operation of the
AQM mechanism whereas the asymmetric link scenario evaluates an AQM mechanism whereas the asymmetric link scenario evaluates an
AQM mechanism in a more realistic setup; AQM mechanism in a more realistic setup;
o in asymmetric link scenarios, the tester SHOULD study the bi- o in asymmetric link scenarios, the tester should study the bi-
directional traffic between A and B (downlink and uplink) with the directional traffic between A and B (downlink and uplink) with the
AQM mechanism deployed on one direction only. The tester MAY AQM mechanism deployed on one direction only. The tester may
additionally consider a scenario with AQM mechanism being deployed additionally consider a scenario with AQM mechanism being deployed
on both directions. In each scenario, the tester SHOULD on both directions. In each scenario, the tester should
investigate the impact of drop policy of the AQM on TCP ACK investigate the impact of drop policy of the AQM on TCP ACK
packets and its impact on the performance. packets and its impact on the performance.
Although this topology may not perfectly reflect actual topologies, Although this topology may not perfectly reflect actual topologies,
the simple topology is commonly used in the world of simulations and the simple topology is commonly used in the world of simulations and
small testbeds. It can be considered as adequate to evaluate AQM small testbeds. It can be considered as adequate to evaluate AQM
proposals, similarly to the topology proposed in proposals [I-D.irtf-iccrg-tcpeval]. Testers ought to pay attention
[I-D.irtf-iccrg-tcpeval]. Testers ought to pay attention to the to the topology that has been used to evaluate an AQM scheme when
topology that has been used to evaluate an AQM scheme when comparing comparing this scheme with a newly proposed AQM scheme.
this scheme with a newly proposed AQM scheme.
3.2. Buffer size 3.2. Buffer size
The size of the buffers should be carefully chosen, and MAY be set to The size of the buffers should be carefully chosen, and may be set to
the bandwidth-delay product; the bandwidth being the bottleneck the bandwidth-delay product; the bandwidth being the bottleneck
capacity and the delay the largest RTT in the considered network. capacity and the delay the largest RTT in the considered network.
The size of the buffer can impact the AQM performance and is a The size of the buffer can impact the AQM performance and is a
dimensioning parameter that will be considered when comparing AQM dimensioning parameter that will be considered when comparing AQM
proposals. proposals.
If a specific buffer size is required, the tester MUST justify and If a specific buffer size is required, the tester must justify and
detail the way the maximum queue size is set. Indeed, the maximum detail the way the maximum queue size is set. Indeed, the maximum
size of the buffer may affect the AQM's performance and its choice size of the buffer may affect the AQM's performance and its choice
SHOULD be elaborated for a fair comparison between AQM proposals. should be elaborated for a fair comparison between AQM proposals.
While comparing AQM schemes the buffer size SHOULD remain the same While comparing AQM schemes the buffer size should remain the same
across the tests. across the tests.
3.3. Congestion controls 3.3. Congestion controls
This document considers running three different congestion control This document considers running three different congestion control
algorithms between A and B algorithms between A and B
o Standard TCP congestion control: the base-line congestion control o Standard TCP congestion control: the base-line congestion control
is TCP NewReno with SACK, as explained in [RFC5681]. is TCP NewReno with SACK [RFC5681].
o Aggressive congestion controls: a base-line congestion control for o Aggressive congestion controls: a base-line congestion control for
this category is TCP Cubic [I-D.ietf-tcpm-cubic]. this category is TCP Cubic [I-D.ietf-tcpm-cubic].
o Less-than Best Effort (LBE) congestion controls: an LBE congestion o Less-than Best Effort (LBE) congestion controls: an LBE congestion
control 'results in smaller bandwidth and/or delay impact on control 'results in smaller bandwidth and/or delay impact on
standard TCP than standard TCP itself, when sharing a bottleneck standard TCP than standard TCP itself, when sharing a bottleneck
with it.' [RFC6297] with it.' [RFC6297]
Other transport congestion controls can OPTIONALLY be evaluated in Other transport congestion controls can OPTIONALLY be evaluated in
addition. Recent transport layer protocols are not mentioned in the addition. Recent transport layer protocols are not mentioned in the
following sections, for the sake of simplicity. following sections, for the sake of simplicity.
4. Methodology, Metrics, AQM Comparisons, Packet Sizes, Scheduling and 4. Methodology, Metrics, AQM Comparisons, Packet Sizes, Scheduling and
ECN ECN
4.1. Methodology 4.1. Methodology
A description of each test setup SHOULD be detailed to allow this A description of each test setup should be detailed to allow this
test to be compared with other tests. This also allows others to test to be compared with other tests. This also allows others to
replicate the tests if needed. This test setup SHOULD detail replicate the tests if needed. This test setup should detail
software and hardware versions. The tester could make its data software and hardware versions. The tester could make its data
available. available.
The proposals SHOULD be evaluated on real-life systems, or they MAY The proposals should be evaluated on real-life systems, or they may
be evaluated with event-driven simulations (such as ns-2, ns-3, be evaluated with event-driven simulations (such as ns-2, ns-3,
OMNET, etc). The proposed scenarios are not bound to a particular OMNET, etc). The proposed scenarios are not bound to a particular
evaluation toolset. evaluation toolset.
The tester is encouraged to make the detailed test setup and the The tester is encouraged to make the detailed test setup and the
results publicly available. results publicly available.
4.2. Comments on metrics measurement 4.2. Comments on metrics measurement
The document presents the end-to-end metrics that ought to be used to The document presents the end-to-end metrics that ought to be used to
evaluate the trade-off between latency and goodput in Section 2. In evaluate the trade-off between latency and goodput in Section 2. In
addition to the end-to-end metrics, the queue-level metrics (normally addition to the end-to-end metrics, the queue-level metrics (normally
collected at the device operating the AQM) provide a better collected at the device operating the AQM) provide a better
understanding of the AQM behavior under study and the impact of its understanding of the AQM behavior under study and the impact of its
internal parameters. Whenever it is possible (e.g., depending on the internal parameters. Whenever it is possible (e.g., depending on the
features provided by the hardware/software), these guidelines advice features provided by the hardware/software), these guidelines advise
to consider queue-level metrics, such as link utilization, queuing to consider queue-level metrics, such as link utilization, queuing
delay, queue size or packet drop/mark statistics in addition to the delay, queue size or packet drop/mark statistics in addition to the
AQM-specific parameters. However, the evaluation MUST be primarily AQM-specific parameters. However, the evaluation must be primarily
based on externally observed end-to-end metrics. based on externally observed end-to-end metrics.
These guidelines do not aim to detail on the way these metrics can be These guidelines do not aim to detail on the way these metrics can be
measured, since the way these metrics are measured is expected to measured, since the way these metrics are measured is expected to
depend on the evaluation toolset. depend on the evaluation toolset.
4.3. Comparing AQM schemes 4.3. Comparing AQM schemes
This document recognizes that these guidelines may be used for This document recognizes that these guidelines may be used for
comparing AQM schemes. comparing AQM schemes.
AQM schemes need to be compared against both performance and AQM schemes need to be compared against both performance and
deployment categories. In addition, this section details how best to deployment categories. In addition, this section details how best to
achieve a fair comparison of AQM schemes by avoiding certain achieve a fair comparison of AQM schemes by avoiding certain
pitfalls. pitfalls.
4.3.1. Performance comparison 4.3.1. Performance comparison
AQM schemes should be compared against the generic scenarios that are AQM schemes should be compared against the generic scenarios that are
summarized in Section 13. AQM schemes MAY be compared for specific summarized in Section 13. AQM schemes may be compared for specific
network environments such as data centers, home networks, etc. If an network environments such as data centers, home networks, etc. If an
AQM scheme has parameter(s) that were externally tuned for AQM scheme has parameter(s) that were externally tuned for
optimization or other purposes, these values MUST be disclosed. optimization or other purposes, these values must be disclosed.
AQM schemes belong to different varieties such as queue-length based AQM schemes belong to different varieties such as queue-length based
schemes (ex. RED) or queueing-delay based scheme (ex. CoDel, PIE). schemes (ex. RED) or queueing-delay based scheme (ex. CoDel, PIE).
AQM schemes expose different control knobs associated with different AQM schemes expose different control knobs associated with different
semantics. For example, while both PIE and CoDel are queueing-delay semantics. For example, while both PIE and CoDel are queueing-delay
based schemes and each expose a knob to control the queueing delay -- based schemes and each expose a knob to control the queueing delay --
PIE's "queueing delay reference" vs. CoDel's "queueing delay target", PIE's "queueing delay reference" vs. CoDel's "queueing delay target",
the two tuning parameters of the two schemes have different the two tuning parameters of the two schemes have different
semantics, resulting in different control points. Such differences semantics, resulting in different control points. Such differences
in AQM schemes can be easily overlooked while making comparisons. in AQM schemes can be easily overlooked while making comparisons.
This document RECOMMENDS the following procedures for a fair This document recommends the following procedures for a fair
performance comparison between the AQM schemes: performance comparison between the AQM schemes:
1. comparable control parameters and comparable input values: 1. Similar control parameters and implications: Testers should be
carefully identify the set of parameters that control similar aware of the control parameters of the different schemes that
behavior between the two AQM schemes and ensure these parameters control similar behavior. Testers should also be aware of the
have comparable input values. For example, to compare how well a input value ranges and corresponding implications. For example,
queue-length based AQM scheme controls queueing delay vs. a consider two different schemes - (A) queue-length based AQM
queueing-delay based AQM scheme, a tester can identify the scheme, and (B) queueing-delay based scheme. A and B are likely
parameters of the schemes that control queue delay and ensure to have different kinds of control inputs to control the target
that their input values are comparable. Similarly, to compare delay - target queue length in A vs. target queuing delay in B,
how well two AQM schemes accommodate packet bursts, the tester for example. Setting parameter values such as 100MB for A vs.
can identify burst-related control parameters and ensure they are 10ms for B will have different implications depending on
configured with similar values. Additionally, it would be evaluation context. Such context-dependent implications must be
preferable if an AQM proposal listed such parameters and considered before drawing conclusions on performance comparisons.
discussed how each relates to network characteristics such as Also, it would be preferable if an AQM proposal listed such
capacity, average RTT etc. parameters and discussed how each relates to network
characteristics such as capacity, average RTT etc.
2. compare over a range of input configurations: there could be 2. compare over a range of input configurations: there could be
situations when the set of control parameters that affect a situations when the set of control parameters that affect a
specific behavior have different semantics between the two AQM specific behavior have different semantics between the two AQM
schemes. As mentioned above, PIE has tuning parameters to schemes. As mentioned above, PIE has tuning parameters to
control queue delay that has a different semantics from those control queue delay that has a different semantics from those
used in CoDel. In such situations, these schemes need to be used in CoDel. In such situations, these schemes need to be
compared over a range of input configurations. For example, compared over a range of input configurations. For example,
compare PIE vs. CoDel over the range of target delay input compare PIE vs. CoDel over the range of target delay input
configurations. configurations.
4.3.2. Deployment comparison 4.3.2. Deployment comparison
AQM schemes MUST be compared against deployment criteria such as the AQM schemes must be compared against deployment criteria such as the
parameter sensitivity (Section 8.3), auto-tuning (Section 12) or parameter sensitivity (Section 8.3), auto-tuning (Section 12) or
implementation cost (Section 11). implementation cost (Section 11).
4.4. Packet sizes and congestion notification 4.4. Packet sizes and congestion notification
An AQM scheme may be considering packet sizes while generating An AQM scheme may be considering packet sizes while generating
congestion signals. [RFC7141] discusses the motivations behind this. congestion signals [RFC7141]. For example, control packets such as
For example, control packets such as DNS requests/responses, TCP DNS requests/responses, TCP SYNs/ACKs are small, but their loss can
SYNs/ACKs are small, but their loss can severely impact the severely impact the application performance. An AQM scheme may
application performance. An AQM scheme may therefore be biased therefore be biased towards small packets by dropping them with
towards small packets by dropping them with smaller probability smaller probability compared to larger packets. However, such an AQM
compared to larger packets. However, such an AQM scheme is unfair to scheme is unfair to data senders generating larger packets. Data
data senders generating larger packets. Data senders, malicious or senders, malicious or otherwise, are motivated to take advantage of
otherwise, are motivated to take advantage of such AQM scheme by such AQM scheme by transmitting smaller packets, and could result in
transmitting smaller packets, and could result in unsafe deployments unsafe deployments and unhealthy transport and/or application
and unhealthy transport and/or application designs. designs.
An AQM scheme SHOULD adhere to the recommendations outlined in An AQM scheme should adhere to the recommendations outlined in
[RFC7141], and SHOULD NOT provide undue advantage to flows with [RFC7141], and should not provide undue advantage to flows with
smaller packets [RFC7567]. smaller packets [RFC7567]. In order to evaluate if an AQM scheme is
biased towards flows with smaller size packets, sender A in Figure 1
can be instantiated with two long standing TCP flows with different
packet sizes; such as 500 bytes vs. 1500 bytes respectively, and
metrics can be compared for these two flows.
4.5. Interaction with ECN 4.5. Interaction with ECN
Deployed AQM algorithms SHOULD implement Explicit Congestion ECN [RFC3168] is an alternative that allows AQM schemes to signal
Notification (ECN) as well as loss to signal congestion to endpoints receivers about network congestion that does not use packet drop.
[RFC7567]. ECN [RFC3168] is an alternative that allows AQM schemes There are benefits of providing ECN support for an AQM scheme
to signal receivers about network congestion that does not use packet [WELZ2015].
drop. The benefits of providing ECN support for an AQM scheme are
described in [WELZ2015]. Section 3 of [WELZ2015] describes expected
operation of routers enabling ECN. AQM schemes SHOULD NOT drop or
remark packets solely because the ECT(0) or ECT(1) codepoints are
used, and when ECN-capable SHOULD set a CE-mark on ECN-capable
packets in the presence of incipient congestion.
If the tested AQM scheme can support ECN [RFC7567], the testers MUST If the tested AQM scheme can support ECN [RFC7567], the testers must
discuss and describe the support of ECN. Since these guidelines can discuss and describe the support of ECN. Since these guidelines can
be used to evaluate the performance of the tested AQM with and be used to evaluate the performance of the tested AQM with and
without ECN markings, they could also be used to quantify the without ECN markings, they could also be used to quantify the
interest of enabling ECN. interest of enabling ECN.
4.6. Interaction with Scheduling 4.6. Interaction with Scheduling
A network device may use per-flow or per-class queuing with a A network device may use per-flow or per-class queuing with a
scheduling algorithm to either prioritize certain applications or scheduling algorithm to either prioritize certain applications or
classes of traffic, limit the rate of transmission, or to provide classes of traffic, limit the rate of transmission, or to provide
isolation between different traffic flows within a common class isolation between different traffic flows within a common class
[RFC7567]. [RFC7567].
The scheduling and the AQM conjointly impact on the end-to-end The scheduling and the AQM conjointly impact on the end-to-end
performance. Therefore, the AQM proposal MUST discuss the performance. Therefore, the AQM proposal must discuss the
feasibility to add scheduling combined with the AQM algorithm. This feasibility to add scheduling combined with the AQM algorithm. This
discussion as an instance, MAY explain whether the dropping policy is discussion as an instance, may explain whether the dropping policy is
applied when packets are being enqueued or dequeued. applied when packets are being enqueued or dequeued.
These guidelines do not propose guidelines to assess the performance These guidelines do not propose guidelines to assess the performance
of scheduling algorithms. Indeed, as opposed to characterizing AQM of scheduling algorithms. Indeed, as opposed to characterizing AQM
schemes that is related to their capacity to control the queuing schemes that is related to their capacity to control the queuing
delay in a queue, characterizing scheduling schemes is related to the delay in a queue, characterizing scheduling schemes is related to the
scheduling itself and its interaction with the AQM scheme. As one scheduling itself and its interaction with the AQM scheme. As one
example, the scheduler may create sub-queues and the AQM scheme may example, the scheduler may create sub-queues and the AQM scheme may
be applied on each of the sub-queues, and/or the AQM could be applied be applied on each of the sub-queues, and/or the AQM could be applied
on the whole queue. Also, schedulers might, such as FQ-CoDel on the whole queue. Also, schedulers might, such as FQ-CoDel
5.1. TCP-friendly sender 5.1. TCP-friendly sender
5.1.1. TCP-friendly sender with the same initial congestion window 5.1.1. TCP-friendly sender with the same initial congestion window
This scenario helps to evaluate how an AQM scheme reacts to a TCP- This scenario helps to evaluate how an AQM scheme reacts to a TCP-
friendly transport sender. A single long-lived, non application- friendly transport sender. A single long-lived, non application-
limited, TCP NewReno flow, with an Initial congestion Window (IW) set limited, TCP NewReno flow, with an Initial congestion Window (IW) set
to 3 packets, transfers data between sender A and receiver B. Other to 3 packets, transfers data between sender A and receiver B. Other
TCP friendly congestion control schemes such as TCP-friendly rate TCP friendly congestion control schemes such as TCP-friendly rate
control [RFC5348] etc MAY also be considered. control [RFC5348] etc may also be considered.
For each TCP-friendly transport considered, the graph described in For each TCP-friendly transport considered, the graph described in
Section 2.7 could be generated. Section 2.7 could be generated.
5.1.2. TCP-friendly sender with different initial congestion windows 5.1.2. TCP-friendly sender with different initial congestion windows
This scenario can be used to evaluate how an AQM scheme adapts to a This scenario can be used to evaluate how an AQM scheme adapts to a
traffic mix consisting of TCP flows with different values of the IW. traffic mix consisting of TCP flows with different values of the IW.
For this scenario, two types of flows MUST be generated between For this scenario, two types of flows must be generated between
sender A and receiver B: sender A and receiver B:
o A single long-lived non application-limited TCP NewReno flow; o A single long-lived non application-limited TCP NewReno flow;
o A single application-limited TCP NewReno flow, with an IW set to 3 o A single application-limited TCP NewReno flow, with an IW set to 3
or 10 packets. The size of the data transferred must be strictly or 10 packets. The size of the data transferred must be strictly
higher than 10 packets and should be lower than 100 packets. higher than 10 packets and should be lower than 100 packets.
The transmission of the non application-limited flow must start The transmission of the non application-limited flow must start first
before the transmission of the application-limited flow and only and the transmission of the application-limited flow starts after the
after the steady state has been reached by non application-limited non application-limited flow has reached steady state.
flow.
For each of these scenarios, the graph described in Section 2.7 could For each of these scenarios, the graph described in Section 2.7 could
be generated for each class of traffic (application-limited and non be generated for each class of traffic (application-limited and non
application-limited). The completion time of the application-limited application-limited). The completion time of the application-limited
TCP flow could be measured. TCP flow could be measured.
5.2. Aggressive transport sender 5.2. Aggressive transport sender
This scenario helps testers to evaluate how an AQM scheme reacts to a This scenario helps testers to evaluate how an AQM scheme reacts to a
transport sender that is more aggressive than a single TCP-friendly transport sender that is more aggressive than a single TCP-friendly
sender. We define 'aggressiveness' as a higher increase factor than sender. We define 'aggressiveness' as a higher increase factor than
standard upon a successful transmission and/or a lower than standard standard upon a successful transmission and/or a lower than standard
decrease factor upon a unsuccessful transmission (e.g., in case of decrease factor upon a unsuccessful transmission (e.g., in case of
congestion controls with Additive-Increase Multiplicative-Decrease congestion controls with Additive-Increase Multiplicative-Decrease
(AIMD) principle, a larger AI and/or MD factors). A single long- (AIMD) principle, a larger AI and/or MD factors). A single long-
lived, non application-limited, TCP Cubic flow transfers data between lived, non application-limited, TCP Cubic flow transfers data between
sender A and receiver B. Other aggressive congestion control schemes sender A and receiver B. Other aggressive congestion control schemes
MAY also be considered. may also be considered.
For each flavor of aggressive transports, the graph described in For each flavor of aggressive transports, the graph described in
Section 2.7 could be generated. Section 2.7 could be generated.
5.3. Unresponsive transport sender 5.3. Unresponsive transport sender
This scenario helps testers to evaluate how an AQM scheme reacts to a This scenario helps testers to evaluate how an AQM scheme reacts to a
transport sender that is less responsive than TCP. Note that faulty transport sender that is less responsive than TCP. Note that faulty
transport implementations on an end host and/or faulty network transport implementations on an end host and/or faulty network
elements en-route that "hide" congestion signals in packet headers elements en-route that "hide" congestion signals in packet headers
the first scenario, the rate of the UDP traffic should not be greater the first scenario, the rate of the UDP traffic should not be greater
than the bottleneck capacity, and should be higher than half of the than the bottleneck capacity, and should be higher than half of the
bottleneck capacity. For each type of traffic, the graph described bottleneck capacity. For each type of traffic, the graph described
in Section 2.7 could be generated. in Section 2.7 could be generated.
5.4. Less-than Best Effort transport sender 5.4. Less-than Best Effort transport sender
This scenario helps to evaluate how an AQM scheme reacts to LBE This scenario helps to evaluate how an AQM scheme reacts to LBE
congestion controls that 'results in smaller bandwidth and/or delay congestion controls that 'results in smaller bandwidth and/or delay
impact on standard TCP than standard TCP itself, when sharing a impact on standard TCP than standard TCP itself, when sharing a
bottleneck with it.' [RFC6297]. The potential fateful interaction bottleneck with it.' [RFC6297]. There are potential fateful
when AQM and LBE techniques are combined has been shown in interactions when AQM and LBE techniques are combined [GONG2014];
[GONG2014]; this scenario helps to evaluate whether the coexistence this scenario helps to evaluate whether the coexistence of the
of the proposed AQM and LBE techniques may be possible. proposed AQM and LBE techniques may be possible.
A single long-lived non application-limited TCP NewReno flow A single long-lived non application-limited TCP NewReno flow
transfers data between sender A and receiver B. Other TCP-friendly transfers data between sender A and receiver B. Other TCP-friendly
congestion control schemes MAY also be considered. Single long-lived congestion control schemes may also be considered. Single long-lived
non application-limited LEDBAT [RFC6817] flows transfer data between non application-limited LEDBAT [RFC6817] flows transfer data between
sender A and receiver B. We recommend to set the target delay and sender A and receiver B. We recommend to set the target delay and
gain values of LEDBAT respectively to 5 ms and 10 [TRAN2014]. Other gain values of LEDBAT respectively to 5 ms and 10 [TRAN2014]. Other
LBE congestion control schemes, any of those listed in [RFC6297], MAY LBE congestion control schemes [RFC6297], may also be considered.
also be considered.
For each of the TCP-friendly and LBE transports, the graph described For each of the TCP-friendly and LBE transports, the graph described
in Section 2.7 could be generated. in Section 2.7 could be generated.
6. Round Trip Time Fairness 6. Round Trip Time Fairness
6.1. Motivation 6.1. Motivation
An AQM scheme's congestion signals (via drops or ECN marks) must An AQM scheme's congestion signals (via drops or ECN marks) must
reach the transport sender so that a responsive sender can initiate reach the transport sender so that a responsive sender can initiate
its congestion control mechanism and adjust the sending rate. This its congestion control mechanism and adjust the sending rate. This
procedure is thus dependent on the end-to-end path RTT. When the RTT procedure is thus dependent on the end-to-end path RTT. When the RTT
varies, the onset of congestion control is impacted, and in turn varies, the onset of congestion control is impacted, and in turn
impacts the ability of an AQM scheme to control the queue. It is impacts the ability of an AQM scheme to control the queue. It is
therefore important to assess the AQM schemes for a set of RTTs therefore important to assess the AQM schemes for a set of RTTs
between A and B (e.g., from 5 ms to 200 ms). between A and B (e.g., from 5 ms to 200 ms).
The asymmetry in terms of difference in intrinsic RTT between various The asymmetry in terms of difference in intrinsic RTT between various
paths sharing the same bottleneck SHOULD be considered so that the paths sharing the same bottleneck should be considered so that the
fairness between the flows can be discussed since in this scenario, a fairness between the flows can be discussed since in this scenario, a
flow traversing on shorter RTT path may react faster to congestion flow traversing on shorter RTT path may react faster to congestion
and recover faster from it compared to another flow on a longer RTT and recover faster from it compared to another flow on a longer RTT
path. The introduction of AQM schemes may potentially improve this path. The introduction of AQM schemes may potentially improve the
type of fairness. RTT fairness.
Introducing an AQM scheme may cause the unfairness between the flows, Introducing an AQM scheme may cause the unfairness between the flows,
even if the RTTs are identical. This potential unfairness SHOULD be even if the RTTs are identical. This potential unfairness should be
investigated as well. investigated as well.
6.2. Recommended tests 6.2. Recommended tests
The RECOMMENDED topology is detailed in Figure 1. The recommended topology is detailed in Figure 1.
To evaluate the RTT fairness, for each run, two flows divided into To evaluate the RTT fairness, for each run, two flows are divided
two categories. Category I whose RTT between sender A and receiver B into two categories. Category I whose RTT between sender A and
SHOULD be 100ms. Category II which RTT between sender A and receiver receiver B should be 100ms. Category II which RTT between sender A
B should be in the range [5ms;560ms] inclusive. The maximum value and receiver B should be in the range [5ms;560ms] inclusive. The
for the RTT represents the RTT of a satellite link that, according to maximum value for the RTT represents the RTT of a satellite link
section 2 of [RFC2488] should be at least 558ms. [RFC2488].
A set of evaluated flows MUST use the same congestion control A set of evaluated flows must use the same congestion control
algorithm: all the generated flows could be single long-lived non algorithm: all the generated flows could be single long-lived non
application-limited TCP NewReno flows. application-limited TCP NewReno flows.
6.3. Metrics to evaluate the RTT fairness 6.3. Metrics to evaluate the RTT fairness
The outputs that MUST be measured are: (1) the cumulative average The outputs that must be measured are: (1) the cumulative average
goodput of the flow from Category I, goodput_Cat_I (Section 2.5); (2) goodput of the flow from Category I, goodput_Cat_I (Section 2.5); (2)
the cumulative average goodput of the flow from Category II, the cumulative average goodput of the flow from Category II,
goodput_Cat_II (Section 2.5); (3) the ratio goodput_Cat_II/ goodput_Cat_II (Section 2.5); (3) the ratio goodput_Cat_II/
goodput_Cat_I; (4) the average packet drop rate for each category goodput_Cat_I; (4) the average packet drop rate for each category
(Section 2.3). (Section 2.3).
7. Burst Absorption 7. Burst Absorption
"AQM mechanisms need to control the overall queue sizes, to ensure "AQM mechanisms need to control the overall queue sizes, to ensure
that arriving bursts can be accommodated without dropping packets" that arriving bursts can be accommodated without dropping packets"
in the buffer for bursts of arriving packets. The tolerance to in the buffer for bursts of arriving packets. The tolerance to
bursts of packets depends upon the number of packets in the queue, bursts of packets depends upon the number of packets in the queue,
which is directly linked to the AQM algorithm. Moreover, an AQM which is directly linked to the AQM algorithm. Moreover, an AQM
scheme may implement a feature controlling the maximum size of scheme may implement a feature controlling the maximum size of
accepted bursts, that can depend on the buffer occupancy or the accepted bursts, that can depend on the buffer occupancy or the
currently estimated queuing delay. The impact of the buffer size on currently estimated queuing delay. The impact of the buffer size on
the burst allowance may be evaluated. the burst allowance may be evaluated.
7.2. Recommended tests 7.2. Recommended tests
For this scenario, tester MUST evaluate how the AQM performs with the For this scenario, tester must evaluate how the AQM performs with the
following traffic generated from sender A to receiver B: following traffic generated from sender A to receiver B:
o Web traffic with IW10; o Web traffic with IW10;
o Bursty video frames; o Bursty video frames;
o Constant Bit Rate (CBR) UDP traffic. o Constant Bit Rate (CBR) UDP traffic.
o A single non application-limited bulk TCP flow as background o A single non application-limited bulk TCP flow as background
traffic. traffic.
Figure 2 presents the various cases for the traffic that MUST be Figure 2 presents the various cases for the traffic that must be
generated between sender A and receiver B. generated between sender A and receiver B.
+-------------------------------------------------+ +-------------------------------------------------+
|Case| Traffic Type | |Case| Traffic Type |
| +-----+------------+----+--------------------+ | +-----+------------+----+--------------------+
| |Video|Web (IW 10)| CBR| Bulk TCP Traffic | | |Video|Web (IW 10)| CBR| Bulk TCP Traffic |
+----|-----|------------|----|--------------------| +----|-----|------------|----|--------------------|
|I | 0 | 1 | 1 | 0 | |I | 0 | 1 | 1 | 0 |
+----|-----|------------|----|--------------------| +----|-----|------------|----|--------------------|
|II | 0 | 1 | 1 | 1 | |II | 0 | 1 | 1 | 1 |
|III | 1 | 1 | 1 | 0 | |III | 1 | 1 | 1 | 0 |
+----|-----|------------|----|--------------------| +----|-----|------------|----|--------------------|
|IV | 1 | 1 | 1 | 1 | |IV | 1 | 1 | 1 | 1 |
+----+-----+------------+----+--------------------+ +----+-----+------------+----+--------------------+
Figure 2: Bursty traffic scenarios Figure 2: Bursty traffic scenarios
A new web page download could start after the previous web page A new web page download could start after the previous web page
download is finished. Each web page could be composed by at least 50 download is finished. Each web page could be composed by at least 50
objects and the size of each object should be at least 1kB. 6 TCP objects and the size of each object should be at least 1kB. 6 TCP
parallel connections SHOULD be generated to download the objects, parallel connections should be generated to download the objects,
each parallel connections having an initial congestion window set to each parallel connections having an initial congestion window set to
10 packets. 10 packets.
For each of these scenarios, the graph described in Section 2.7 could For each of these scenarios, the graph described in Section 2.7 could
be generated for each application. Metrics such as end-to-end be generated for each application. Metrics such as end-to-end
latency, jitter, flow completion time MAY be generated. For the latency, jitter, flow completion time may be generated. For the
cases of frame generation of bursty video traffic as well as the cases of frame generation of bursty video traffic as well as the
choice of web traffic pattern, these details and their presentation choice of web traffic pattern, these details and their presentation
are left to the testers. are left to the testers.
8. Stability 8. Stability
8.1. Motivation 8.1. Motivation
The safety of an AQM scheme is directly related to its stability The safety of an AQM scheme is directly related to its stability
under varying operating conditions such as varying traffic profiles under varying operating conditions such as varying traffic profiles
Whether the target context is a not stable environment, the ability Whether the target context is a not stable environment, the ability
of an AQM scheme to maintain its control over the queuing delay and of an AQM scheme to maintain its control over the queuing delay and
buffer occupancy can be challenged. This document proposes buffer occupancy can be challenged. This document proposes
guidelines to assess the behavior of AQM schemes under varying guidelines to assess the behavior of AQM schemes under varying
congestion levels and varying draining rates. congestion levels and varying draining rates.
8.2. Recommended tests 8.2. Recommended tests
Note that the traffic profiles explained below comprises non Note that the traffic profiles explained below comprises non
application-limited TCP flows. For each of the below scenarios, the application-limited TCP flows. For each of the below scenarios, the
graphs described in Section 2.7 SHOULD be generated, and the goodput graphs described in Section 2.7 should be generated, and the goodput
of the various flows should be cumulated. For Section 8.2.5 and of the various flows should be cumulated. For Section 8.2.5 and
Section 8.2.6 they SHOULD incorporate the results in per-phase basis Section 8.2.6 they should incorporate the results in per-phase basis
as well. as well.
Wherever the notion of time has explicitly mentioned in this Wherever the notion of time has explicitly mentioned in this
subsection, time 0 starts from the moment all TCP flows have already subsection, time 0 starts from the moment all TCP flows have already
reached their congestion avoidance phase. reached their congestion avoidance phase.
8.2.1. Definition of the congestion Level 8.2.1. Definition of the congestion Level
In these guidelines, the congestion levels are represented by the In these guidelines, the congestion levels are represented by the
projected packet drop rate, had a drop-tail queue was chosen instead projected packet drop rate, had a drop-tail queue was chosen instead
o Experiment 1: the capacity varies between two values within a o Experiment 1: the capacity varies between two values within a
large time-scale. As an example, the following phases may be large time-scale. As an example, the following phases may be
considered: phase I - 100Mbps during 0-20s; phase II - 10Mbps considered: phase I - 100Mbps during 0-20s; phase II - 10Mbps
during 20-40s; phase I again, and so on. during 20-40s; phase I again, and so on.
o Experiment 2: the capacity varies between two values within a o Experiment 2: the capacity varies between two values within a
short time-scale. As an example, the following phases may be short time-scale. As an example, the following phases may be
considered: phase I - 100Mbps during 0-100ms; phase II - 10Mbps considered: phase I - 100Mbps during 0-100ms; phase II - 10Mbps
during 100-200ms; phase I again, and so on. during 100-200ms; phase I again, and so on.
The tester MAY choose a phase time-interval value different than what The tester may choose a phase time-interval value different than what
is stated above, if the network's path conditions (such as bandwidth- is stated above, if the network's path conditions (such as bandwidth-
delay product) necessitate. In this case the choice of such time- delay product) necessitate. In this case the choice of such time-
interval value SHOULD be stated and elaborated. interval value should be stated and elaborated.
The tester MAY additionally evaluate the two mentioned scenarios The tester may additionally evaluate the two mentioned scenarios
(short-term and long-term capacity variations), during and/or (short-term and long-term capacity variations), during and/or
including TCP slow-start phase. including TCP slow-start phase.
More realistic fluctuating capacity patterns MAY be considered. The More realistic fluctuating capacity patterns may be considered. The
tester MAY choose to incorporate realistic scenarios with regards to tester may choose to incorporate realistic scenarios with regards to
common fluctuation of bandwidth in state-of-the-art technologies. common fluctuation of bandwidth in state-of-the-art technologies.
The scenario consists of TCP NewReno flows between sender A and The scenario consists of TCP NewReno flows between sender A and
receiver B. To better assess the impact of draining rates on the AQM receiver B. To better assess the impact of draining rates on the AQM
behavior, the tester MUST compare its performance with those of drop- behavior, the tester must compare its performance with those of drop-
tail and SHOULD provide a reference document for their proposal tail and should provide a reference document for their proposal
discussing performance and deployment compared to those of drop-tail. discussing performance and deployment compared to those of drop-tail.
Burst traffic, such as presented in Section 7.2, could also be Burst traffic, such as presented in Section 7.2, could also be
considered to assess the impact of varying available capacity on the considered to assess the impact of varying available capacity on the
burst absorption of the AQM. burst absorption of the AQM.
8.3. Parameter sensitivity and stability analysis 8.3. Parameter sensitivity and stability analysis
The control law used by an AQM is the primary means by which the The control law used by an AQM is the primary means by which the
queuing delay is controlled. Hence understanding the control law is queuing delay is controlled. Hence understanding the control law is
critical to understanding the behavior of the AQM scheme. The critical to understanding the behavior of the AQM scheme. The
Transports operating under the control of AQM experience the effect Transports operating under the control of AQM experience the effect
of multiple control loops that react over different timescales. It of multiple control loops that react over different timescales. It
is therefore important that proposed AQM schemes are seen to be is therefore important that proposed AQM schemes are seen to be
stable when they are deployed at multiple points of potential stable when they are deployed at multiple points of potential
congestion along an Internet path. The pattern of congestion signals congestion along an Internet path. The pattern of congestion signals
(loss or ECN-marking) arising from AQM methods also need to not (loss or ECN-marking) arising from AQM methods also need to not
adversely interact with the dynamics of the transport protocols that adversely interact with the dynamics of the transport protocols that
they control. they control.
AQM proposals SHOULD provide background material showing control AQM proposals should provide background material showing control
theoretic analysis of the AQM control law and the input parameter theoretic analysis of the AQM control law and the input parameter
space within which the control law operates as expected; or could use space within which the control law operates as expected; or could use
another way to discuss the stability of the control law. For another way to discuss the stability of the control law. For
parameters that are auto-tuned, the material SHOULD include stability parameters that are auto-tuned, the material should include stability
analysis of the auto-tuning mechanism(s) as well. Such analysis analysis of the auto-tuning mechanism(s) as well. Such analysis
helps to understand an AQM control law better and the network helps to understand an AQM control law better and the network
conditions/deployments under which the AQM is stable. conditions/deployments under which the AQM is stable.
9. Various Traffic Profiles 9. Various Traffic Profiles
This section provides guidelines to assess the performance of an AQM This section provides guidelines to assess the performance of an AQM
proposal for various traffic profiles such as traffic with different proposal for various traffic profiles such as traffic with different
applications or bi-directional traffic. applications or bi-directional traffic.
o Bulk TCP transfer o Bulk TCP transfer
o Web traffic o Web traffic
o VoIP o VoIP
o Constant Bit Rate (CBR) UDP traffic o Constant Bit Rate (CBR) UDP traffic
o Adaptive video streaming o Adaptive video streaming
Various traffic mixes can be considered. These guidelines RECOMMEND Various traffic mixes can be considered. These guidelines recommend
to examine at least the following example: 1 bi-directional VoIP; 6 to examine at least the following example: 1 bi-directional VoIP; 6
Web pages download (such as detailed in Section 7.2); 1 CBR; 1 Web pages download (such as detailed in Section 7.2); 1 CBR; 1
Adaptive Video; 5 bulk TCP. Any other combinations could be Adaptive Video; 5 bulk TCP. Any other combinations could be
considered and should be carefully documented. considered and should be carefully documented.
For each scenario, the graph described in Section 2.7 could be For each scenario, the graph described in Section 2.7 could be
generated for each class of traffic. Metrics such as end-to-end generated for each class of traffic. Metrics such as end-to-end
latency, jitter and flow completion time MAY be reported. latency, jitter and flow completion time may be reported.
9.2. Bi-directional traffic 9.2. Bi-directional traffic
Control packets such as DNS requests/responses, TCP SYNs/ACKs are Control packets such as DNS requests/responses, TCP SYNs/ACKs are
small, but their loss can severely impact the application small, but their loss can severely impact the application
performance. The scenario proposed in this section will help in performance. The scenario proposed in this section will help in
assessing whether the introduction of an AQM scheme increases the assessing whether the introduction of an AQM scheme increases the
loss probability of these important packets. loss probability of these important packets.
For this scenario, traffic MUST be generated in both downlink and For this scenario, traffic must be generated in both downlink and
uplink, such as defined in Section 3.1. These guidelines RECOMMEND uplink, such as defined in Section 3.1. These guidelines recommend
to consider a mild congestion level and the traffic presented in to consider a mild congestion level and the traffic presented in
Section 8.2.2 in both directions. In this case, the metrics reported Section 8.2.2 in both directions. In this case, the metrics reported
MUST be the same as in Section 8.2 for each direction. must be the same as in Section 8.2 for each direction.
The traffic mix presented in Section 9.1 MAY also be generated in The traffic mix presented in Section 9.1 may also be generated in
both directions. both directions.
10. Multi-AQM Scenario 10. Multi-AQM Scenario
10.1. Motivation 10.1. Motivation
Transports operating under the control of AQM experience the effect Transports operating under the control of AQM experience the effect
of multiple control loops that react over different timescales. It of multiple control loops that react over different timescales. It
is therefore important that proposed AQM schemes are seen to be is therefore important that proposed AQM schemes are seen to be
stable when they are deployed at multiple points of potential stable when they are deployed at multiple points of potential
congestion along an Internet path. The pattern of congestion signals congestion along an Internet path. The pattern of congestion signals
(loss or ECN-marking) arising from AQM methods also need to not (loss or ECN-marking) arising from AQM methods also need to not
adversely interact with the dynamics of the transport protocols that adversely interact with the dynamics of the transport protocols that
they control. they control.
11. Implementation cost 11. Implementation cost
11.1. Motivation 11.1. Motivation
Successful deployment of AQM is directly related to its cost of Successful deployment of AQM is directly related to its cost of
implementation. Network devices can need hardware or software implementation. Network devices can need hardware or software
implementations of the AQM mechanism. Depending on a device's implementations of the AQM mechanism. Depending on a device's
capabilities and limitations, the device may or may not be able to capabilities and limitations, the device may or may not be able to
implement some or all parts of their AQM logic. implement some or all parts of their AQM logic.
AQM proposals SHOULD provide pseudo-code for the complete AQM scheme, AQM proposals should provide pseudo-code for the complete AQM scheme,
highlighting generic implementation-specific aspects of the scheme highlighting generic implementation-specific aspects of the scheme
such as "drop-tail" vs. "drop-head", inputs (e.g., current queuing such as "drop-tail" vs. "drop-head", inputs (e.g., current queuing
delay, queue length), computations involved, need for timers, etc. delay, queue length), computations involved, need for timers, etc.
This helps to identify costs associated with implementing the AQM This helps to identify costs associated with implementing the AQM
scheme on a particular hardware or software device. This also scheme on a particular hardware or software device. This also
facilitates discsusions around which kind of devices can easily facilitates discsusions around which kind of devices can easily
support the AQM and which cannot. support the AQM and which cannot.
11.2. Recommended discussion 11.2. Recommended discussion
AQM proposals SHOULD highlight parts of their AQM logic that are AQM proposals should highlight parts of their AQM logic that are
device dependent and discuss if and how AQM behavior could be device dependent and discuss if and how AQM behavior could be
impacted by the device. For example, a queueing-delay based AQM impacted by the device. For example, a queueing-delay based AQM
scheme requires current queuing delay as input from the device. If scheme requires current queuing delay as input from the device. If
the device already maintains this value, then it can be trivial to the device already maintains this value, then it can be trivial to
implement the their AQM logic on the device. If the device provides implement the their AQM logic on the device. If the device provides
indirect means to estimate the queuing delay (for example: indirect means to estimate the queuing delay (for example:
timestamps, dequeuing rate), then the AQM behavior is sensitive to timestamps, dequeuing rate), then the AQM behavior is sensitive to
the precision of the queuing delay estimations are for that device. the precision of the queuing delay estimations are for that device.
Highlighting the sensitivity of an AQM scheme to queuing delay Highlighting the sensitivity of an AQM scheme to queuing delay
estimations helps implementers to identify appropriate means of estimations helps implementers to identify appropriate means of
Any AQM scheme is likely to have parameters whose values affect the Any AQM scheme is likely to have parameters whose values affect the
control law and behaviour of an AQM. Exposing all these parameters control law and behaviour of an AQM. Exposing all these parameters
as control parameters to a network operator (or user) can easily as control parameters to a network operator (or user) can easily
result in a unsafe AQM deployment. Unexpected AQM behavior ensues result in a unsafe AQM deployment. Unexpected AQM behavior ensues
when parameter values are set improperly. A minimal number of when parameter values are set improperly. A minimal number of
control parameters minimizes the number of ways a user can break a control parameters minimizes the number of ways a user can break a
system where an AQM scheme is deployed at. Fewer control parameters system where an AQM scheme is deployed at. Fewer control parameters
make the AQM scheme more user-friendly and easier to deploy and make the AQM scheme more user-friendly and easier to deploy and
debug. debug.
[RFC7567] states "AQM algorithms SHOULD NOT require tuning of initial "AQM algorithms should not require tuning of initial or configuration
or configuration parameters in common use cases." A scheme ought to parameters in common use cases." [RFC7567]. A scheme ought to
expose only those parameters that control the macroscopic AQM expose only those parameters that control the macroscopic AQM
behavior such as queue delay threshold, queue length threshold, etc. behavior such as queue delay threshold, queue length threshold, etc.
Additionally, the safety of an AQM scheme is directly related to its Additionally, the safety of an AQM scheme is directly related to its
stability under varying operating conditions such as varying traffic stability under varying operating conditions such as varying traffic
profiles and fluctuating network conditions, as described in profiles and fluctuating network conditions, as described in
Section 8. Operating conditions vary often and hence the AQM needs Section 8. Operating conditions vary often and hence the AQM needs
to remain stable under these conditions without the need for to remain stable under these conditions without the need for
additional external tuning. If AQM parameters require tuning under additional external tuning. If AQM parameters require tuning under
these conditions, then the AQM must self-adapt necessary parameter these conditions, then the AQM must self-adapt necessary parameter
values by employing auto-tuning techniques. values by employing auto-tuning techniques.
12.2. Recommended discussion 12.2. Recommended discussion
In order to understand an AQM's deployment considerations and In order to understand an AQM's deployment considerations and
performance under a specific environment, AQM proposals SHOULD performance under a specific environment, AQM proposals should
describe the parameters that control the macroscopic AQM behavior, describe the parameters that control the macroscopic AQM behavior,
and identify any parameters that require tuning to operational and identify any parameters that require tuning to operational
conditions. It could be interesting to also discuss that even if an conditions. It could be interesting to also discuss that even if an
AQM scheme may not adequately auto-tune its parameters, the resulting AQM scheme may not adequately auto-tune its parameters, the resulting
performance may not be optimal, but close to something reasonable. performance may not be optimal, but close to something reasonable.
If there are any fixed parameters within the AQM, their setting If there are any fixed parameters within the AQM, their setting
SHOULD be discussed and justified, to help understand whether a fixed should be discussed and justified, to help understand whether a fixed
parameter value is applicable for a particular environment. parameter value is applicable for a particular environment.
If an AQM scheme is evaluated with parameter(s) that were externally If an AQM scheme is evaluated with parameter(s) that were externally
tuned for optimization or other purposes, these values MUST be tuned for optimization or other purposes, these values must be
disclosed. disclosed.
13. Conclusion 13. Conclusion
Figure 4 lists the scenarios and their requirements. Figure 4 lists the scenarios and their requirements for an extended
characterization of an AQM scheme.
+------------------------------------------------------------------+ +------------------------------------------------------------------+
|Scenario |Sec. |Requirement | |Scenario |Sec. |Requirement |
+------------------------------------------------------------------+ +------------------------------------------------------------------+
+------------------------------------------------------------------+ +------------------------------------------------------------------+
|Interaction with ECN | 4.5 |MUST be discussed if supported | |Interaction with ECN | 4.5 |MUST be discussed if supported |
+------------------------------------------------------------------+ +------------------------------------------------------------------+
|Interaction with Scheduling| 4.6 |Feasibility MUST be discussed | |Interaction with Scheduling| 4.6 |Feasibility MUST be discussed |
+------------------------------------------------------------------+ +------------------------------------------------------------------+
|Transport Protocols |5. | | |Transport Protocols |5. | |
| TCP-friendly sender | 5.1 |Scenario MUST be considered | | TCP-friendly sender | 5.1 |Scenario MUST be considered |
| Aggressive sender | 5.2 |Scenario MUST be considered | | Aggressive sender | 5.2 |Scenario MUST be considered |
| Unresponsive sender | 5.3 |Scenario MUST be considered | | Unresponsive sender | 5.3 |Scenario MUST be considered |
| LBE sender | 5.4 |Scenario MAY be considered | | LBE sender | 5.4 |Scenario MAY be considered |
+------------------------------------------------------------------+ +------------------------------------------------------------------+
|Round Trip Time Fairness | 6.2 |Scenario MUST be considered | |Round Trip Time Fairness | 6.2 |Scenario MUST be considered |
+------------------------------------------------------------------+ +------------------------------------------------------------------+
|Burst Absorption | 7.2 |Scenario MUST be considered | |Burst Absorption | 7.2 |Scenario MUST be considered |
17. Security Considerations 17. Security Considerations
Some security considerations for AQM are identified in [RFC7567].This Some security considerations for AQM are identified in [RFC7567].This
document, by itself, presents no new privacy nor security issues. document, by itself, presents no new privacy nor security issues.
18. References 18. References
18.1. Normative References 18.1. Normative References
[I-D.ietf-tcpm-cubic]
Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and
R. Scheffenegger, "CUBIC for Fast Long-Distance Networks",
draft-ietf-tcpm-cubic-01 (work in progress), January 2016.
[I-D.irtf-iccrg-tcpeval]
Hayes, D., Ros, D., Andrew, L., and S. Floyd, "Common TCP
Evaluation Suite", draft-irtf-iccrg-tcpeval-01 (work in
progress), July 2014.
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7,
RFC 793, DOI 10.17487/RFC0793, September 1981,
<http://www.rfc-editor.org/info/rfc793>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, 1997. Requirement Levels", RFC 2119, 1997.
[RFC2488] Allman, M., Glover, D., and L. Sanchez, "Enhancing TCP
Over Satellite Channels using Standard Mechanisms",
BCP 28, RFC 2488, DOI 10.17487/RFC2488, January 1999,
<http://www.rfc-editor.org/info/rfc2488>.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, Network Interconnect Devices", RFC 2544,
DOI 10.17487/RFC2544, March 1999, DOI 10.17487/RFC2544, March 1999,
<http://www.rfc-editor.org/info/rfc2544>. <http://www.rfc-editor.org/info/rfc2544>.
[RFC2647] Newman, D., "Benchmarking Terminology for Firewall [RFC2647] Newman, D., "Benchmarking Terminology for Firewall
Performance", RFC 2647, DOI 10.17487/RFC2647, August 1999, Performance", RFC 2647, DOI 10.17487/RFC2647, August 1999,
<http://www.rfc-editor.org/info/rfc2647>. <http://www.rfc-editor.org/info/rfc2647>.
[RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679, Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679,
September 1999, <http://www.rfc-editor.org/info/rfc2679>. September 1999, <http://www.rfc-editor.org/info/rfc2679>.
[RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Packet Loss Metric for IPPM", RFC 2680, Packet Loss Metric for IPPM", RFC 2680,
DOI 10.17487/RFC2680, September 1999, DOI 10.17487/RFC2680, September 1999,
<http://www.rfc-editor.org/info/rfc2680>. <http://www.rfc-editor.org/info/rfc2680>.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001,
<http://www.rfc-editor.org/info/rfc3168>.
[RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed.,
"RTP Control Protocol Extended Reports (RTCP XR)",
RFC 3611, DOI 10.17487/RFC3611, November 2003,
<http://www.rfc-editor.org/info/rfc3611>.
[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification",
RFC 5348, DOI 10.17487/RFC5348, September 2008,
<http://www.rfc-editor.org/info/rfc5348>.
[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
March 2009, <http://www.rfc-editor.org/info/rfc5481>. March 2009, <http://www.rfc-editor.org/info/rfc5481>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<http://www.rfc-editor.org/info/rfc5681>.
[RFC6297] Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort
Transport Protocols", RFC 6297, DOI 10.17487/RFC6297, June
2011, <http://www.rfc-editor.org/info/rfc6297>.
[RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind,
"Low Extra Delay Background Transport (LEDBAT)", RFC 6817,
DOI 10.17487/RFC6817, December 2012,
<http://www.rfc-editor.org/info/rfc6817>.
[RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion
Notification", RFC 7141, 2014.
[RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF
Recommendations Regarding Active Queue Management", Recommendations Regarding Active Queue Management",
BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015,
<http://www.rfc-editor.org/info/rfc7567>. <http://www.rfc-editor.org/info/rfc7567>.
18.2. Informative References 18.2. Informative References
[ANEL2014] [ANEL2014]
Anelli, P., Diana, R., and E. Lochin, "FavorQueue: a Anelli, P., Diana, R., and E. Lochin, "FavorQueue: a
Parameterless Active Queue Management to Improve TCP Parameterless Active Queue Management to Improve TCP
[HASS2008] [HASS2008]
Hassayoun, S. and D. Ros, "Loss Synchronization and Router Hassayoun, S. and D. Ros, "Loss Synchronization and Router
Buffer Sizing with High-Speed Versions of TCP", IEEE Buffer Sizing with High-Speed Versions of TCP", IEEE
INFOCOM Workshops , 2008. INFOCOM Workshops , 2008.
[HOEI2015] [HOEI2015]
Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys,
J., and E. Dumazet, "FlowQueue-Codel", IETF (Work-in- J., and E. Dumazet, "FlowQueue-Codel", IETF (Work-in-
Progress) , January 2015. Progress) , January 2015.
[I-D.ietf-aqm-codel]
Nichols, K., Jacobson, V., McGregor, A., and J. Iyengar,
"Controlled Delay Active Queue Management", draft-ietf-
aqm-codel-03 (work in progress), March 2016.
[I-D.ietf-aqm-pie]
Pan, R., Natarajan, P., and F. Baker, "PIE: A Lightweight
Control Scheme To Address the Bufferbloat Problem", draft-
ietf-aqm-pie-07 (work in progress), April 2016.
[I-D.ietf-tcpm-cubic]
Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and
R. Scheffenegger, "CUBIC for Fast Long-Distance Networks",
draft-ietf-tcpm-cubic-01 (work in progress), January 2016.
[I-D.irtf-iccrg-tcpeval]
Hayes, D., Ros, D., Andrew, L., and S. Floyd, "Common TCP
Evaluation Suite", draft-irtf-iccrg-tcpeval-01 (work in
progress), July 2014.
[JAY2006] Jay, P., Fu, Q., and G. Armitage, "A preliminary analysis [JAY2006] Jay, P., Fu, Q., and G. Armitage, "A preliminary analysis
of loss synchronisation between concurrent TCP flows", of loss synchronisation between concurrent TCP flows",
Australian Telecommunication Networks and Application Australian Telecommunication Networks and Application
Conference (ATNAC) , 2006. Conference (ATNAC) , 2006.
[MORR2000] [MORR2000]
Morris, R., "Scalable TCP congestion control", IEEE Morris, R., "Scalable TCP congestion control", IEEE
INFOCOM , 2000. INFOCOM , 2000.
[NICH2012] [RFC0793] Postel, J., "Transmission Control Protocol", STD 7,
Nichols, K. and V. Jacobson, "Controlling Queue Delay", RFC 793, DOI 10.17487/RFC0793, September 1981,
ACM Queue , 2012. <http://www.rfc-editor.org/info/rfc793>.
[PAN2013] Pan, R., Natarajan, P., Piglione, C., Prabhu, MS.,
Subramanian, V., Baker, F., and B. VerSteeg, "PIE: A
lightweight control scheme to address the bufferbloat
problem", IEEE HPSR , 2013.
[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering,
S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G.,
Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, Partridge, C., Peterson, L., Ramakrishnan, K., Shenker,
S., Wroclawski, J., and L. Zhang, "Recommendations on S., Wroclawski, J., and L. Zhang, "Recommendations on
Queue Management and Congestion Avoidance in the Queue Management and Congestion Avoidance in the
Internet", RFC 2309, April 1998. Internet", RFC 2309, April 1998.
[RFC2488] Allman, M., Glover, D., and L. Sanchez, "Enhancing TCP
Over Satellite Channels using Standard Mechanisms",
BCP 28, RFC 2488, DOI 10.17487/RFC2488, January 1999,
<http://www.rfc-editor.org/info/rfc2488>.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001,
<http://www.rfc-editor.org/info/rfc3168>.
[RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed.,
"RTP Control Protocol Extended Reports (RTCP XR)",
RFC 3611, DOI 10.17487/RFC3611, November 2003,
<http://www.rfc-editor.org/info/rfc3611>.
[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification",
RFC 5348, DOI 10.17487/RFC5348, September 2008,
<http://www.rfc-editor.org/info/rfc5348>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<http://www.rfc-editor.org/info/rfc5681>.
[RFC6297] Welzl, M. and D. Ros, "A Survey of Lower-than-Best-Effort
Transport Protocols", RFC 6297, DOI 10.17487/RFC6297, June
2011, <http://www.rfc-editor.org/info/rfc6297>.
[RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind,
"Low Extra Delay Background Transport (LEDBAT)", RFC 6817,
DOI 10.17487/RFC6817, December 2012,
<http://www.rfc-editor.org/info/rfc6817>.
[RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion
Notification", RFC 7141, 2014.
[TRAN2014] [TRAN2014]
Trang, S., Kuhn, N., Lochin, E., Baudoin, C., Dubois, E., Trang, S., Kuhn, N., Lochin, E., Baudoin, C., Dubois, E.,
and P. Gelard, "On The Existence Of Optimal LEDBAT and P. Gelard, "On The Existence Of Optimal LEDBAT
Parameters", IEEE ICC 2014 - Communication QoS, Parameters", IEEE ICC 2014 - Communication QoS,
Reliability and Modeling Symposium , 2014. Reliability and Modeling Symposium , 2014.
[WELZ2015] [WELZ2015]
Welzl, M. and G. Fairhurst, "The Benefits to Applications Welzl, M. and G. Fairhurst, "The Benefits to Applications
of using Explicit Congestion Notification (ECN)", IETF of using Explicit Congestion Notification (ECN)", IETF
(Work-in-Progress) , June 2015. (Work-in-Progress) , June 2015.

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/
_______________________________________________
Gen-art mailing list
Gen-art <at> ietf.org
https://www.ietf.org/mailman/listinfo/gen-art
Dale R. Worley | 18 May 22:03 2016

Gen-ART IETF Telechat review of draft-ietf-netmod-rfc6020bis-12

I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed by
the IESG for the IETF Chair. Please wait for direction from your
document shepherd or AD before posting a new version of the draft.

For more information, please see the FAQ at
<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Document: draft-ietf-netmod-rfc6020bis-12
Reviewer: Dale R. Worley
Review Date: 2016-05-12
IETF LC End Date: 2016-05-12
IESG Telechat date: 2016-05-19

Summary: This draft is basically ready for publication, but has nits
that should be fixed before publication.  I have a large number of
editorial comments.  A few of these comments address technical
uncertainties, but I strongly suspect that the technical issues have
long since been fixed, as this is a revision of Yang 1, and the
only needed work is clarifying the text.

For more information, please see the IETF last call Gen-ART review
thread:  http://www.ietf.org/mail-archive/web/ietf/current/msg98051.html

Dale
Kuhn Nicolas | 17 May 12:53 2016
Picon

Re: Gen-ART review of draft-ietf-aqm-eval-guidelines-11

Hi again, 

FYI, attached to this email is what could be a new ID for the document. 

Kind regards,

Nicolas

-----Message d'origine-----
De : Kuhn Nicolas 
Envoyé : mardi 17 mai 2016 10:31
À : Ralph Droms (rdroms); gen-art <at> ietf.org
Cc : draft-ietf-aqm-eval-guidelines.all <at> tools.ietf.org; Mirja Kuehlewind (IETF)
Objet : RE: Gen-ART review of draft-ietf-aqm-eval-guidelines-11

Thanks a lot for this review. We have proposed answer to the different points and hope that this is more
clear. When we believe that it was needed, we have suggested text changes. If you think that this should
result in the submission of a new ID, please let us know. 

Kind regards, 

The authors

-----Message d'origine-----
De : Ralph Droms (rdroms) [mailto:rdroms <at> cisco.com] Envoyé : jeudi 28 avril 2016 21:23 À :
gen-art <at> ietf.org Cc : draft-ietf-aqm-eval-guidelines.all <at> tools.ietf.org
Objet : Gen-ART review of draft-ietf-aqm-eval-guidelines-11

I am the assigned Gen-ART reviewer for this draft. The General Area Review Team (Gen-ART) reviews all IETF
documents being processed by the IESG for the IETF Chair.  Please treat these comments just like any other
last call comments.

For more information, please see the FAQ at

<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Document: draft-ietf-aqm-eval-guidelines-11
Reviewer: Ralph Droms
Review Date: 2016-04-28
IETF LC End Date: 2016-05-04
IESG Telechat date: 2016-05-19

Summary: This draft is on the right track but has open issues, described in the review.

In general, I think the document could be read, implemented and used to generate useful characterizations
of AQM schemes.  However, the motivations for some of the measurements and scenarios seems weak to me,
which might compromise the weight given to the conclusions drawn from the guidelines.

Major issues:

None.  However, the list of minor issues and nits, taken together, could be considered a major issue to be
resolved before publication.

Minor issues:

#####

I often react to the use of RFC 2119 language in an Informational document by asking is that language really
necessary?  I'll ask the question here: in the context of this Informational document, which appears to be
entirely advisory in providing guidelines, what does the use of RFC 2119 "requirements language" add to
the meaning of the document.

>> Authors: Indeed, the use of RFC 2119 language is not mandatory for such information document. However,
using it enables us to introduce weight in the different parameterizations of the tests. Even though, it
is not mandatory, we believe that it eases the reading of the document, for someone familiar with the IETF wording.

#####

Figure 1 is not clear to me.  Where are the physical links and interfaces?  Are there multiple physical
senders and receivers or are "senders A" instantiated on a single host (does it make a difference)?  Are
there static-sized buffers for each interface or do all the buffers share one memory space?

>> Authors: We acknowledge that Figure 1 is not very clear. We have voluntarily omitted precisions on the
amount of senders, receivers and traffic classes since the instantiation on a specific testbed would
remove the generality of the figure and the described architecture. We believe that the text helps in
reading the figure. Also, the rationale of this figure is to explain the notation more than going deeper in
the topology that is anyway very generic.    

#####

In section 3.1, is there a need to say something about the relative capacities of the various links and the
rates at which the various flows generate traffic?

>> Authors: These capacities are described in a later section when needed, and to remain high level and not
focus on any applicability context (wi-fi, rural satellite access, fibber access, etc.) they are not
specified for the whole document. The rates at which the flows generate traffic is specified for each
further described scenario.

#####

I would have trouble following the guidelines set out in section 4.3.1.  I can understand the need for
consideration of the tunable control parameters when comparing different AQM schemes.  However, I don't
know what "comparable" means for control parameters that are likely quite different between AQM
schemes.  I also think one would want to compare optimal control settings for the different schemes, to
compare best-case performance.  Or, for AQM schemes whose performance is highly dependent on
operational conditions, one might want to compare settings that are sub-optimal for any particular test
condition but that give better performance over a wide range of conditions.

>> Authors: The intent of the first recommendation is to make testers be aware of which control points
control which behavior and be conscious to make apples to apples comparison. 
To further precise this, we could change the text is section 4.3.1 as follows :
"1. Similar control parameters and implications: Testers should be aware of the control parameters of the
different schemes that control similar behavior. Testers should also be aware of the input value ranges
and corresponding implications. For example, consider two different schemes - (A) queue-length based
AQM scheme, and (B) queueing-delay based scheme. A and B are likely to have different kinds of control
inputs to control the target delay - target queue length in A vs. target queuing delay in B, for example.
Setting parameter values such as 100MB for A vs. 10ms for B will have different implications depending on
evaluation context.  Such context-dependent implications must be considered before drawing
conclusions on performance comparisons. Also, it would be preferable if an AQM proposal listed such
parameters and discussed how each relates to network characteristics such as capacity, average RTT etc."

#####

Section 4.4 seems to give advice to the AQM designer rather than describe guidelines for
characterization.  Section 4.4 should either be rewritten to give guidelines for structuring
measurements to account for varying packet sizes or the section should be elided.

>> Authors: We could to modify text for 2nd paragraph of 4.4, if you think that this clarifies the issue.
" An AQM scheme SHOULD adhere to the recommendations outlined in [RFC7141], and SHOULD NOT provide undue
advantage to flows with smaller packets [RFC7567]. In order to evaluate if an AQM scheme is biased towards
flows with smaller size packets, sender A in Figure 1 can be instantiated with two long standing TCP flows
with different packet sizes - 500 bytes vs. 1500 bytes, respectively and metrics such as goodput, loss
rate can be compared for these two flows. "

#####

In section 4.5, what is the motivation for giving the advice about ECN to AQM designers?  I can understand
that ECN will have affect the impact of AQM, but for this document I think the section should focus on
measurement guidlines that account for that impact.

>> Authors: The scope of introducing this part is mostly related to 
>> remain the tester that if their AQM supports ECN, this must be 
>> presented, since it could be seen as an important move for the 
>> deployment of ECN

#####

The specific topology in section 10 does not seem well-motivated to me.  Why is router R with no AQM included
in the topology?  The choice of measurements is similarly not well-motivated.  Why would it not be of
interest to run all the tests described earlier in the document?

>> Authors: It is worth pointing out that this specific topology is just a suggestion. The router without
AQM has been included to remind the tester that the positioning of multiple AQMs have to be looked at with
routers that do not have AQMs in mind. Since the placement of AQMs is mostly expected to be on "the last
mile", we believe that evaluating all the tests described earlier in the document may not be needed.
However, in case of multiple AQMs, their interactions should be considered.

#####

Nits/editorial comments:

There are several instances of the word "advice" which should be replaced with "advise"; e.g., in section 2.3.

>> Authors: Thanks.

#####

Last sentence of the abstract: I don't get the meaning of "precautionary characterizations of AQM
schemes".  I recommend that the phrase be reworded

>> Authors: After another read of the abstract, we believe that the "precautionary" may not be needed. What
about : « characterizations of AQM schemes".  

#####

Section 1, first paragraph: The last sentence doesn't follow the rest of the paragraph and I recommend that
it be elided.

>> Authors: IMHO, we may even remove that sentence.

#####

Section 1, third paragraph: This text is redundant with the text in the Glossary section:

   When speaking of a specific queue in this
   document, "buffer occupancy" refers to the amount of data (measured
   in bytes or packets) that are in the queue, and the "maximum buffer
   size" refers to the maximum buffer occupancy.

>> Authors: It may be clearer to remove that sentence.

#####

Section 1, third paragraph:

OLD:

   In real
   implementations of switches, a global memory is often shared between
   the available devices, and thus, the maximum buffer size may vary
   over the time.

NEW:

   In switches and routers, a global memory space is often shared
   between the available interfaces, and thus, the maximum buffer size
   for any given interface may vary over the time.

>> Authors: Thanks.

#####

Section 1, fifth paragraph, last sentence: Is this document just concerned with "deployability" or more
generally with "applicability, performance and deployability"?

>> Authors: We believe that the « deployability » involves the performance and the applicability, but we
can further specify if you think it is important.

#####

Section 1.1, first paragraph: Would it be helpful to qualify "goodput" as "goodput in individual flows",
to contrast with "goodput at a router"?  If "goodput" is well-known in this community to be "flow goodput",
no change is needed.

>> Authors: IMHO, it is well-known to be flow goodput.

#####

Section 1.1, second paragraph: What is "BDP", as in "BDP-sized buffer"?

>> Authors: Indeed, the BDP has not been defined. We could add it in the glossary ?

#####

Section 1.1, third paragraph, first sentence: "environment" is unclear; perhaps "deployment scenario"
or "use case"?  I'm not sure what was meant.

>> Authors: Indeed, deployment scenario may be clearer.

#####

Section 1.3: for completeness, a definition of "latency" would be useful.

>> Authors: We can add a definition of latency in the glossary.

#####

Section 2, first paragraph: I'm going to sound pedantic here, but it seems to me this section is not just
intended "to better quantify (1) the reduction of latency, (2) maximization of goodput and (3) the
trade-off between these two" but also to define other performance metrics that get to the goal from the
abstract of describing "various criteria for performing precautionary characterizations of AQM schemes"

>> Authors: We believe that the "criterias for performaning characterizations" is covered by the last
sentence of the paragraphe :
"This section provides normative requirements for metrics that can be
   used to assess the performance of an AQM scheme.
"  

#####

Section 2, last paragraph: What is "application-limited traffic"?  Later, what is "non
application-limited" traffic?

>> Authors: We may propose a definition in the application limited traffic. A maybe more accurate name
could be "Rate-Limited Traffic" ?

#####

Section 2.2: s/of/and/

>> Authors: Thanks.

#####

Section 2.5, first paragraph: s/induces/requires/  ???

>> Authors: Agree.

#####

Section 2.6, second paragraph is just not clear to me.  In particular, what is the antecedent of "they":

   The introduction of an AQM scheme would impact these
   metrics (end-to-end latency and jitter) and therefore they should be
   considered in the end-to-end evaluation of performance."

>> Authors: « They » refers the « these metrics ».

#####

Section 5.1: is this new text correct?

OLD:

   The transmission of the non application-limited flow must start
   before the transmission of the application-limited flow and only
   after the steady state has been reached by non application-limited
   flow.

NEW:

   The transmission of the non application-limited flow must start
   first and the transmission of the application-limited flow starts
   after the non application-limited flow has reached steady state.

>> Authors: Yes. It seems better to me.

#####

Throughout section 5, the text like "the graph described in Section 2.7 could be generated" seems redundant.

>> Authors: Indeed. We do not want readers to have to look at all the documents, but rather can directly jump
on the test of their interest.

#####

Section 6.1, second paragraph: I don't understand the last sentence.  What is the "type of fairness" that
the AQM scheme might affect?

>> Authors: « Type of fairness «  refers to the rtt fairness.

#####

Section 6.2, second paragraph: The words up to the first period don't constitute a sentence.

>> Authors: I do not understand the comment.

#####

In section 7.2, what is "IW10"?  Why are the particular traffic flow chosen in figure 2?  s/could/should/ in
the last paragraph?

>> Authors: This can be added in the glossary. IW10 means Initial congestion Window set to 10. They have been
chosen because they will generate bursts of data.

_______________________________________________
Gen-art mailing list
Gen-art <at> ietf.org
https://www.ietf.org/mailman/listinfo/gen-art
Kuhn Nicolas | 17 May 10:31 2016
Picon

Re: Gen-ART review of draft-ietf-aqm-eval-guidelines-11

Thanks a lot for this review. We have proposed answer to the different points and hope that this is more
clear. When we believe that it was needed, we have suggested text changes. If you think that this should
result in the submission of a new ID, please let us know. 

Kind regards, 

The authors

-----Message d'origine-----
De : Ralph Droms (rdroms) [mailto:rdroms <at> cisco.com] 
Envoyé : jeudi 28 avril 2016 21:23
À : gen-art <at> ietf.org
Cc : draft-ietf-aqm-eval-guidelines.all <at> tools.ietf.org
Objet : Gen-ART review of draft-ietf-aqm-eval-guidelines-11

I am the assigned Gen-ART reviewer for this draft. The General Area Review Team (Gen-ART) reviews all IETF
documents being processed by the IESG for the IETF Chair.  Please treat these comments just like any other
last call comments.

For more information, please see the FAQ at

<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Document: draft-ietf-aqm-eval-guidelines-11
Reviewer: Ralph Droms
Review Date: 2016-04-28
IETF LC End Date: 2016-05-04
IESG Telechat date: 2016-05-19

Summary: This draft is on the right track but has open issues, described in the review.

In general, I think the document could be read, implemented and used to generate useful characterizations
of AQM schemes.  However, the motivations for some of the measurements and scenarios seems weak to me,
which might compromise the weight given to the conclusions drawn from the guidelines.

Major issues:

None.  However, the list of minor issues and nits, taken together, could be considered a major issue to be
resolved before publication.

Minor issues:

#####

I often react to the use of RFC 2119 language in an Informational document by asking is that language really
necessary?  I'll ask the question here: in the context of this Informational document, which appears to be
entirely advisory in providing guidelines, what does the use of RFC 2119 "requirements language" add to
the meaning of the document.

>> Authors: Indeed, the use of RFC 2119 language is not mandatory for such information document. However,
using it enables us to introduce weight in the different parameterizations of the tests. Even though, it
is not mandatory, we believe that it eases the reading of the document, for someone familiar with the IETF wording.

#####

Figure 1 is not clear to me.  Where are the physical links and interfaces?  Are there multiple physical
senders and receivers or are "senders A" instantiated on a single host (does it make a difference)?  Are
there static-sized buffers for each interface or do all the buffers share one memory space?

>> Authors: We acknowledge that Figure 1 is not very clear. We have voluntarily omitted precisions on the
amount of senders, receivers and traffic classes since the instantiation on a specific testbed would
remove the generality of the figure and the described architecture. We believe that the text helps in
reading the figure. Also, the rationale of this figure is to explain the notation more than going deeper in
the topology that is anyway very generic.    

#####

In section 3.1, is there a need to say something about the relative capacities of the various links and the
rates at which the various flows generate traffic?

>> Authors: These capacities are described in a later section when needed, and to remain high level and not
focus on any applicability context (wi-fi, rural satellite access, fibber access, etc.) they are not
specified for the whole document. The rates at which the flows generate traffic is specified for each
further described scenario.

#####

I would have trouble following the guidelines set out in section 4.3.1.  I can understand the need for
consideration of the tunable control parameters when comparing different AQM schemes.  However, I don't
know what "comparable" means for control parameters that are likely quite different between AQM
schemes.  I also think one would want to compare optimal control settings for the different schemes, to
compare best-case performance.  Or, for AQM schemes whose performance is highly dependent on
operational conditions, one might want to compare settings that are sub-optimal for any particular test
condition but that give better performance over a wide range of conditions.

>> Authors: The intent of the first recommendation is to make testers be aware of which control points
control which behavior and be conscious to make apples to apples comparison. 
To further precise this, we could change the text is section 4.3.1 as follows :
"1. Similar control parameters and implications: Testers should be aware of the control parameters of the
different schemes that control similar behavior. Testers should also be aware of the input value ranges
and corresponding implications. For example, consider two different schemes - (A) queue-length based
AQM scheme, and (B) queueing-delay based scheme. A and B are likely to have different kinds of control
inputs to control the target delay - target queue length in A vs. target queuing delay in B, for example.
Setting parameter values such as 100MB for A vs. 10ms for B will have different implications depending on
evaluation context.  Such context-dependent implications must be considered before drawing
conclusions on performance comparisons. Also, it would be preferable if an AQM proposal listed such
parameters and discussed how each relates to network characteristics such as capacity, average RTT etc."

#####

Section 4.4 seems to give advice to the AQM designer rather than describe guidelines for
characterization.  Section 4.4 should either be rewritten to give guidelines for structuring
measurements to account for varying packet sizes or the section should be elided.

>> Authors: We could to modify text for 2nd paragraph of 4.4, if you think that this clarifies the issue.
" An AQM scheme SHOULD adhere to the recommendations outlined in [RFC7141], and SHOULD NOT provide undue
advantage to flows with smaller packets [RFC7567]. In order to evaluate if an AQM scheme is biased towards
flows with smaller size packets, sender A in Figure 1 can be instantiated with two long standing TCP flows
with different packet sizes - 500 bytes vs. 1500 bytes, respectively and metrics such as goodput, loss
rate can be compared for these two flows. "

#####

In section 4.5, what is the motivation for giving the advice about ECN to AQM designers?  I can understand
that ECN will have affect the impact of AQM, but for this document I think the section should focus on
measurement guidlines that account for that impact.

>> Authors: The scope of introducing this part is mostly related to remain the tester that if their AQM
supports ECN, this must be presented, since it could be seen as an important move for the deployment of ECN

#####

The specific topology in section 10 does not seem well-motivated to me.  Why is router R with no AQM included
in the topology?  The choice of measurements is similarly not well-motivated.  Why would it not be of
interest to run all the tests described earlier in the document?

>> Authors: It is worth pointing out that this specific topology is just a suggestion. The router without
AQM has been included to remind the tester that the positioning of multiple AQMs have to be looked at with
routers that do not have AQMs in mind. Since the placement of AQMs is mostly expected to be on "the last
mile", we believe that evaluating all the tests described earlier in the document may not be needed.
However, in case of multiple AQMs, their interactions should be considered.

#####

Nits/editorial comments:

There are several instances of the word "advice" which should be replaced with "advise"; e.g., in section 2.3.

>> Authors: Thanks.

#####

Last sentence of the abstract: I don't get the meaning of "precautionary characterizations of AQM
schemes".  I recommend that the phrase be reworded

>> Authors: After another read of the abstract, we believe that the "precautionary" may not be needed. What
about : « characterizations of AQM schemes".  

#####

Section 1, first paragraph: The last sentence doesn't follow the rest of the paragraph and I recommend that
it be elided.

>> Authors: IMHO, we may even remove that sentence.

#####

Section 1, third paragraph: This text is redundant with the text in the Glossary section:

   When speaking of a specific queue in this
   document, "buffer occupancy" refers to the amount of data (measured
   in bytes or packets) that are in the queue, and the "maximum buffer
   size" refers to the maximum buffer occupancy.

>> Authors: It may be clearer to remove that sentence.

#####

Section 1, third paragraph:

OLD:

   In real
   implementations of switches, a global memory is often shared between
   the available devices, and thus, the maximum buffer size may vary
   over the time.

NEW:

   In switches and routers, a global memory space is often shared
   between the available interfaces, and thus, the maximum buffer size
   for any given interface may vary over the time.

>> Authors: Thanks.

#####

Section 1, fifth paragraph, last sentence: Is this document just concerned with "deployability" or more
generally with "applicability, performance and deployability"?

>> Authors: We believe that the « deployability » involves the performance and the applicability, but we
can further specify if you think it is important.

#####

Section 1.1, first paragraph: Would it be helpful to qualify "goodput" as "goodput in individual flows",
to contrast with "goodput at a router"?  If "goodput" is well-known in this community to be "flow goodput",
no change is needed.

>> Authors: IMHO, it is well-known to be flow goodput.

#####

Section 1.1, second paragraph: What is "BDP", as in "BDP-sized buffer"?

>> Authors: Indeed, the BDP has not been defined. We could add it in the glossary ?

#####

Section 1.1, third paragraph, first sentence: "environment" is unclear; perhaps "deployment scenario"
or "use case"?  I'm not sure what was meant.

>> Authors: Indeed, deployment scenario may be clearer.

#####

Section 1.3: for completeness, a definition of "latency" would be useful.

>> Authors: We can add a definition of latency in the glossary.

#####

Section 2, first paragraph: I'm going to sound pedantic here, but it seems to me this section is not just
intended "to better quantify (1) the reduction of latency, (2) maximization of goodput and (3) the
trade-off between these two" but also to define other performance metrics that get to the goal from the
abstract of describing "various criteria for performing precautionary characterizations of AQM schemes"

>> Authors: We believe that the "criterias for performaning characterizations" is covered by the last
sentence of the paragraphe :
"This section provides normative requirements for metrics that can be
   used to assess the performance of an AQM scheme.
"  

#####

Section 2, last paragraph: What is "application-limited traffic"?  Later, what is "non
application-limited" traffic?

>> Authors: We may propose a definition in the application limited traffic. A maybe more accurate name
could be "Rate-Limited Traffic" ?

#####

Section 2.2: s/of/and/

>> Authors: Thanks.

#####

Section 2.5, first paragraph: s/induces/requires/  ???

>> Authors: Agree.

#####

Section 2.6, second paragraph is just not clear to me.  In particular, what is the antecedent of "they":

   The introduction of an AQM scheme would impact these
   metrics (end-to-end latency and jitter) and therefore they should be
   considered in the end-to-end evaluation of performance."

>> Authors: « They » refers the « these metrics ».

#####

Section 5.1: is this new text correct?

OLD:

   The transmission of the non application-limited flow must start
   before the transmission of the application-limited flow and only
   after the steady state has been reached by non application-limited
   flow.

NEW:

   The transmission of the non application-limited flow must start
   first and the transmission of the application-limited flow starts
   after the non application-limited flow has reached steady state.

>> Authors: Yes. It seems better to me.

#####

Throughout section 5, the text like "the graph described in Section 2.7 could be generated" seems redundant.

>> Authors: Indeed. We do not want readers to have to look at all the documents, but rather can directly jump
on the test of their interest.

#####

Section 6.1, second paragraph: I don't understand the last sentence.  What is the "type of fairness" that
the AQM scheme might affect?

>> Authors: « Type of fairness «  refers to the rtt fairness.

#####

Section 6.2, second paragraph: The words up to the first period don't constitute a sentence.

>> Authors: I do not understand the comment.

#####

In section 7.2, what is "IW10"?  Why are the particular traffic flow chosen in figure 2?  s/could/should/ in
the last paragraph?

>> Authors: This can be added in the glossary. IW10 means Initial congestion Window set to 10. They have been
chosen because they will generate bursts of data.

Gmane