Just pasted answers to the draft-03 review:

 

http://www.ietf.org/mail-archive/web/bmwg/current/msg02413.html

 

==========PJ ===============

OK, I'm happy with that. Please clarify this in section 4.3.3, eg:

 

      Various Flow monitoring implementations might use different

      default values regarding the export of Control Information

      [RFC5470]. Note that Control Information includes IPFIX Options and Templates [RFC5101].

      The Flow Export corresponding to Control Information ...

==========PJ ===============

 

Clarified - see the text please

 

 

==========PJ ===============

Note that Control Information includes IPFIX Options and Templates [RFC5101].

==========PJ ===============

 

Yes, I know that - what is your point here please ??

 

 

==========PJ ===============

Per Brian's feedback, please add to section 4.5:

 

    Packet sampling and flow sampling is out of scope of this document.

    This document applies to situations without packet or flow sampling.

==========PJ ===============

 

Added

 

==========PJ ===============

Section 1: remove "the":

 

    is provided in the section 3.3.

(And many more times, throughout the document - eg, at the end

of Section 3.1, in Section 3.4.1, at the very end of 3.4.2, ...  Please search yourself.)

==========PJ ===============

 

Searched and removed or occurences.

 

==========PJ ===============

In section 1, at the bottom of page 2, write "DUT's" and remove "support":

 

    The only restriction may be the DUT's lack

    of support for Flow monitoring support of the particular traffic

    type.

==========PJ ===============

 

Changed

 

==========PJ ===============

Section 2.2.5:

 

     mention that there SHOULD NOT be any export filtering, so that all the expired cache entries are exported. If there is export filtering and it can't be disabled, this needs to be noted.

==========PJ ===============

 

Added as suggested

 

==========PJ ===============

Also in section 3.1, I had said:

==========PJ ===============

 

I answered this in the previous review already:

http://www.ietf.org/mail-archive/web/bmwg/current/msg02413.html

 

==========PJ ===============

Section 3.4.1:

 

   In this scenario every packet seen by the DUT creates a

   new Cache entry and forces the DUT to the full Cache processing fill the Cache

   instead of just updating packet and byte counters on of an already

   existing Cache entry.

==========PJ ===============

 

Changed

 

==========PJ ===============

Section 3.4.2: the indentation of the second line in each point is wrong - compare with other a/b/c immediately above and below:

==========PJ ===============

 

Changed

 

==========PJ ===============

Section 4.1:

 

s/if/whether/

 

   The ideal way to implement the measurement is by using a single

   device to provide the sender and receiver capabilities with a sending

   port and a receiving port. This allows for an easy check if whether all the

   traffic sent by the sender was re-transmitted by the DUT and received

   at the receiver.

 

PJ: If the sender and receiver are independent (ie, two devices), then there needs to be another link directly from the sender to the receiver (ie, not through the DUT), so the receiver can make that comparison.

 

- how can the comparison be made if the tester is unable to implement your suggestion of a single device?

==========PJ ===============

 

Changed. In our tests we don't have one device but we still know

what was sent by configuration of the sender and receiver. It is

not ideal as stated in the draft but still not very difficult to

figure out. I don't see any importance/relevance of this - it is

a standard function of any commercial traffic generator.

 

==========PJ ===============

Section 4.1:

    In all measurements, the export

    interface MUST have enough bandwidth to transmit Flow Export data

    without congestion. In other words, the export interface MUST NOT be

    a bottleneck during the measurement.

 

PJ: What if the receiving interface, or the collector itself, is a bottleneck? Previously you said it doesn't matter, and I disagreed.

 

- data could be lost at the receiving end, making the DUT look worse than it really is.

==========PJ ===============

 

I answered this in the previous review already:

http://www.ietf.org/mail-archive/web/bmwg/current/msg02413.html

 

I believe this is all covered in the section 4.4 already

 

==========PJ ===============

Section 4.2:

 

    The DUT export interface (see Figure 2) MUST be configured with

    sufficient output buffers to avoid dropping the Flow Export data due

    to a simple lack of resources in the interface hardware.

 

PJ: then it might not be configured as it would in real life conditions. Wouldn't it be better to record the real-life results than to tweak the settings just for testing purposes?

 

- eg, consider if the DUT operates with insufficient buffers in the real-world scenario, and therefore drops a lot more data than when testing under "ideal" conditions in the lab.

==========PJ ===============

 

I would like to disagree here. We are not measuring real life

scenario here. We are trying to measure the maximum what the

DUT can do. In real life you would also configure the export interface

(if there is one dedicated at all - see the discussion in Appendix

B) not to drop the export packets.

 

==========PJ ===============

Section 4.3:

 

    The DUT SHOULD support IPFIX [RFC5101]

 

Say why?

==========PJ ===============

 

This has been changed several times as the result of the previous reviews. I don't

know what text you would like there. Please suggest something.

The previous text was "The DUT SHOULD support IPFIX [RFC5101] to allow easier

                       results comparison"

 

and your/someone else's question was "What difference it would make ??" - sorry no idea

what you want from me here, it just feels like common sense to have same protocol

for testing everywhere if possible ??

 

==========PJ ===============

Section 4.3.1:

 

      In the case when both ingress and egress Flow monitoring is

      enabled on one DUT the results analysis needs to take into account

      that each Flow will be represented in the DUT Cache by two Flow

      Records (one for each direction) and therefore also the Flow

      Export will contain those two Flow Records.

 

PJ: you assume a single cache. ingress and egress may potentially be recorded in separate caches. This may, or may not, impact the performance.

 

- so mention that whether the combined ingress and egress traffic is measured in one cache or each separately in its own cache, may impact performance and should be recorded.

==========PJ ===============

 

I have added text in 4.3.2.l If it is not sufficient please suggest

something more appropriate.

 

==========PJ ===============

Section 4.3.2: add "of", remove "the", and move "instantly" :

 

      The Cache's Inactive and Active Timeouts MUST be known and taken

      into account when designing the measurement as specified in

      section 5. If the Flow monitoring implementation allows only

      timeouts of zero (e.g. immediate timeout or non-existent Cache) then

      the measurement conditions in the section 5 are fulfilled

      inherently without any additional configuration. The DUT simply

      instantly exports instantly information about every single packet.

 

PJ: Assuming that the cache works with timeouts. What if it uses some other mechanism, eg number of packets in the flow?

 

- state what impact other such mechanisms may have.

==========PJ ===============

 

Changed the text. I don't know about any such impact. Could you elaborate

please ??

 

==========PJ ===============

Section 4.3.3: add "of" :

 

    Section 10 of [RFC5101] and section 8.1 of [RFC5470] discuss

==========PJ ===============

 

Changed.

 

==========PJ ===============

Section 4.3.3: add "of" :

 

    Section 10 of [RFC5101] and section 8.1 of [RFC5470] discuss

 

==========PJ ===============

 

Added

 

==========PJ ===============

Section 4.3.3:

    The Exporting Process SHOULD be configured with IPFIX [RFC5101] as

    the protocol to use to format the Flow Export data. If the Flow

    monitoring implementation does not support it, proprietary

    protocols MAY be used.

 

PJ: what difference would this make to the testing?

 

Eg, NFv5 is a proprietary protocol which has smaller data records and no templates – so it requires less export bandwidth. Wouldn't the results look better if NFv5 was used rather than IPFIX?

 

- at least state that since proprietary protocols may make a considerable difference to the testing, the exact protocol being tested (and any related configuration parameters) MUST be recorded and only similar protocols should be compared.

==========PJ ===============

 

This section has been edited many times to satisfy various reviewers

ideas. I have tried another one. Of it does not suit please provide

some text. This also answers your comment about 4.3.

 

==========PJ ===============

Section 4.3.5, remove "on" :

 

      The test report should therefore contain information

      containing on how many Metering and Exporting processes were

      configured on the DUT for the selected Observation Points.

==========PJ ===============

 

Removed

 

==========PJ ===============

Section 4.3.6: insert "The"

 

    The forwarding performance document [RFC5695] specifies

==========PJ ===============

 

==========PJ ===============

Section 4.4:

 

   However if the Collector is also used to decode the Flow Export data

   then it SHOULD support IPFIX [RFC5101] for easier results analysis.

 

PJ: Again, why does IPFIX make it easier?

==========PJ ===============

 

Please suggest some text, this has gone through many variations as well.

It is answered in the discussion above.

 

==========PJ ===============

Section 4.9.1:

 

   A packet with destination IP address equal to A is sent every 10

   seconds, so the Cache entry would be refreshed in the Cache every 10

   seconds. However, the Inactive Timeout is 5 seconds, so the Cache

   entries will expire from the Cache due to the Inactive Timeout and

   when a new packet is sent with the same IP address A it will create a

   new entry in the Cache.

 

PJ: theoretically. In practice... the DUT has to check those 10,000 cache entries within the 10 seconds to ensure that expired cache entries are exported. If it checks 1,000 cache entries per second, it may only just be ready to expire the existing cache entry when the new packet arrives. Therefore the new packet may sometimes be added to the existing cache entry, giving occasional 2 packet flows!

 

- so note that this behaviour depends upon the design an efficiency of the cache ager, and incidences of multi-packet flows observed during this test should be noted.

==========PJ ===============

 

Added text as suggested.

 

 

==========PJ ===============

At the end of section 4.9.1:

 

   with large Cache Sizes and high packet rate where the DUT's actual

==========PJ ===============

 

Changed

 

 

==========PJ ===============

Section 4.9.2:

 

   So each stream has a packet rate of 10 packets per second. The packets

 

   A packet with destination IP address equal to A is sent every 0.1

   second, so it means that the Cache entry is refreshed in the Cache  

==========PJ ===============

 

Changed

 

==========PJ ===============

I think you could work around all the config issues by simply saying that the DUT should conform to the IPFIX Config model described in draft-ietf-ipfix-configuration-model (which is in the RFC Editor's queue, waiting for the IPFIX MIB).

==========PJ ===============

 

Tried to add some text to 5.1, if it is not suitable please provide

some.

 

==========PJ ===============

Section 5.3: the "Page 19" header/footer is broken - search for [Page 19].

==========PJ ===============

 

Couldn't see it broken, but tried

 

==========PJ ===============

PJ: show that calculation here.

 

- show the exact calculation you have in mind, to ensure that every reader calculates the same.

==========PJ ===============

 

The formula is just above the text you point to, so it is trivial

to figure out:

 

The measurement time interval is then calculated as the difference

   (stop time) - (start time) - (Inactive Timeout).

 

==========PJ ===============

PJ: I'm confused by the final part. What are you trying to avoid?

==========PJ ===============

 

I have answered this in the previous review, Please read

http://www.ietf.org/mail-archive/web/bmwg/current/msg02413.html

to progress any further or simply suggest some text

 

==========PJ ===============

Section 7: "The" shouldn't be capitalised:

==========PJ ===============

 

Changed

 

==========PJ ===============

B.1 add "the" and "a":

==========PJ ===============

 

Added

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Jan,

I reviewed the changes from -02 to -03, then the changes from -01 to -03, and finally I reviewed what you've done in -03 in response to my previous feedback.

The new version is a big improvement over -01 which I last reviewed - so I have many minor comments, along with several points you seem to have missed or overlooked.

Please see my comments inline below.

I look forward to seeing a -04.

P.