headers
Carter Bullard | 1 Oct 2003 02:10

RE: [ipfix] Variant Field Types.

Gentle people,
   It is terribly important that IPFIX support the
transport of vendor specific data.  Because of this,
I expect IPFIX to support variant field types.  
One could argue that variant field types are simply the
implementation of a very large number of explicit
types (4 bit int, 8 bit int, 16 bit int, 4-bit src
address, 4 bit protocol field, etc....).

   I support the notion of variant field types because
it is the only mechanism for improving transport efficiency,
which I believe should be the primary design goal for IPFIX.
I'm not convinced that collector complexity should have
any impact on IPFIX's data model.  Let IPFIX be efficient,
expect the collectors do what is needed to support it.

Carter

-----Original Message-----
From: majordomo listserver [mailto:majordomo <at> mil.doit.wisc.edu] On Behalf Of
Ganesh Sadasivan
Sent: Tuesday, September 30, 2003 6:50 PM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: ipfix <at> net.doit.wisc.edu
Subject: Re: [ipfix] Variant Field Types.

Jeff,

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
(Continue reading)

Permalink | Reply | 0 comments |
headers
MEYER,JEFFREY D (HP-Cupertino,ex1 | 1 Oct 2003 02:37
Picon
Favicon

RE: [ipfix] Variant Field Types.

Carter,

  IPFIX supports vendor specific data.  The information model
is extensible.  So you're covered on that point.

  Regarding transport efficiency, the proposed compromise was
that one explicitly indicate the maximal size of an information
item in the model, but allow the template to indicate that fewer 
bytes are used in the record.

  Keep in mind that the protocol enables two entities to 
communicate, an exporter and a collector.  Implementation 
feasability and practicality on both ends make for something
that will be adopted.

  From an implementation perspective, the exporters are likely
to be more purpose built in their behavior, while the collectors
will likely be more general purpose to support a variety of 
exporters.  As such the exporters will likely have a fixed 
information model which they simple declare, whereas the collectors
will likely require configuration for given devices.

  So from a collector perspective, the machine readable and unambiguous 
information model is important, and as proposed does not constrain the 
exporter.

Regards,

  Jeff Meyer
(Continue reading)

Permalink | Reply | 0 comments |
headers
Stewart Bryant | 1 Oct 2003 08:47
Picon
Favicon

Re: [ipfix] Variant Field Types.

We need to consider IEs on a case by case basis, but I
can see that the collector designer would have difficulty
with completely open-ended IE sizes.

One approach would be for ipfix to declare a recommended
maximum for an object, but allow the exporter to exceed that.
The other is, as Jeff proposes, to set a hard limit. Either
way I think that we agree that the exporter can choose to
use a smaller IE size.

In the soft limit case, presumably the exporter would only
exceed the recommended limit for good reason, and a collector
unable to operate with an IE that large would have to flag
an error.

The difference between a hard and a soft upper bound it how
far out we set the upper limit. The twin problems of
undersizing and backwards compatibility are notorious in
this industry leading to all sorts of work-around. A hard
limit design would therefore need to set the limit way
above what currently seemed reasonable. Would this then
lead to unreasonable storage requirements in the collector?

Although we could perhaps regard the collector storage
capacity as approximating infinity, what about an aggregator?

Stewart

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> Hi,
(Continue reading)

Permalink | Reply | 0 comments |
headers
Mark Fullmer | 1 Oct 2003 17:15

Re: [ipfix] Variant Field Types.

I'm in favor of setting upper bounds, for example counters don't go
past 64 bits.  If you don't do this, what's a collector to do with say
a 72 bit counter?  You can't store it in a uint_64_t anymore so collectors
would have to implement arbitrary precision arithmetic to handle potential
cases.  I don't think this is reasonable to impose on the collectors.

The issue I have with the variable length fields as currently specified
is the potential for template explosion.  If a template has 8
counters, each of which could be 1..8 byte then you have
8^8 potential templates for that one stream of flows.

I was hoping for the sake of the collectors we could keep the active
templates down to something < 64K so a template lookup could be done
with an index instead of a hash.

mark

On Tue, Sep 30, 2003 at 04:20:07PM -0400, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> Ganesh,
>  
>   I'm not clear on how this benefits the collector.  Tal and myself produce
> different implementations of carrier deployed products which collect from
> 100's of devices and dozens of protocols, perhaps our experience here is
> worth something, and it tells us variant is undesireable on the collector...
>  
>   Regarding your other point:
>  
> One way to solve this problem is if the collector assumes the largest size
> for 
> computations?
(Continue reading)

Permalink | Reply | 1 comment |
headers
Ganesh Sadasivan | 1 Oct 2003 20:34
Picon
Favicon

Re: [ipfix] Variant Field Types.

Mark,

Mark Fullmer wrote:
I'm in favor of setting upper bounds, for example counters don't go

I agree that a field even if variable in length is limited by an upper bound and this is limit is set on a
case by case basis. The whole point of my first e-mail is rather than nailing a field to an explicit
type like uint64_t, to let implementors know that  they can send/receive these fields in smaller
sizes too.
past 64 bits. If you don't do this, what's a collector to do with say a 72 bit counter? You can't store it in a uint_64_t anymore so collectors would have to implement arbitrary precision arithmetic to handle potential cases. I don't think this is reasonable to impose on the collectors. The issue I have with the variable length fields as currently specified is the potential for template explosion. If a template has 8 counters, each of which could be 1..8 byte then you have 8^8 potential templates for that one stream of flows.
Yes a separate template is needed for varying length. Template explosion because if this would be
an implementation choice rather than that imposed by the definition itself. Also the same would
have happened if we explicitly define field types for the counters of varying length.

Thanks
Ganesh
I was hoping for the sake of the collectors we could keep the active templates down to something < 64K so a template lookup could be done with an index instead of a hash. mark On Tue, Sep 30, 2003 at 04:20:07PM -0400, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
Ganesh, I'm not clear on how this benefits the collector. Tal and myself produce different implementations of carrier deployed products which collect from 100's of devices and dozens of protocols, perhaps our experience here is worth something, and it tells us variant is undesireable on the collector... Regarding your other point: One way to solve this problem is if the collector assumes the largest size for computations? This was my point in the e-mail to Stewart. Have the information model be EXPLICIT about what is the largest size which is being modeled for a data item and declare it. Allow the exporter to declare a smaller size in the template, if the exporter knows that an individual record will always be accomodated (based on its implementation) in the smaller size. So for instance this might say that byteCount should be declared in the information model as a 64-bit int, while AS is probably sufficient at 32-bits. This accomodates the: exporter - can use smaller size if known n/w - can reduce BW if know a smaller size is used info model - explicit is better than "vague" collector - addresses persistence and aggregation issues Can we call this a compromise? Regards, Jeff Meyer -----Original Message----- From: Ganesh Sadasivan [mailto:gsadasiv <at> cisco.com] Sent: Tuesday, September 30, 2003 10:45 AM To: MEYER,JEFFREY D (HP-Cupertino,ex1) Cc: ipfix <at> net.doit.wisc.edu Subject: Re: [ipfix] Variant Field Types. MEYER,JEFFREY D (HP-Cupertino,ex1) wrote: Hi, Although the use of variant types may seem convenient from the perspective of an exporter, I think the various consumers of the data would benefit from sticking with an explicit type system. It would benefit the exporter, the n/w, flexibility of interpretation of fields in the information model and the collector. Consider operations which involve summing fields, storing them to a DB, etc. One way to solve this problem is if the collector assumes the largest size for computations? Explicit typing ensures that these consuming applications can handle these values appropriately and also makes the information modeller think about how these types are to be used. The examples you cite: - ingressPort - egressPort - packetCount - byteCount - sourceAS - destinationAS - nextHopAS - droppedPacketCount - samplingInterval - droppedByteCount fall into a few categories, identifiers (such as ports and AS's), counts and intervals. For the identifiers, I believe the size is fixed by the relevant RFC's (i.e. why would I have a 1 byte Port?). For counts, the use of 32 vs. 64 bits is a valid question, however these can be handled by making them explicit (i.e. the information model says if I get this value it is a 32-bit quantity or a 64-bit quantity and the code will address it accordingly). For intervals, the resolution would determine which size is most appropriate. For AS the size could be 2 or 4 bytes. You may prefer to use a 1 byte port if you are a small system as against a 4 byte for a gigantic system. Both have their own implementation reasons. The option always exist to define variations on these data items explicitly. E.g. byteCount32 and byteCount64. Yes. But is'nt this not more of an overhead to introduce new field type as the requirements come in and then make sure that it gets deployed in exporter and collector? If there is a need to vary the sizes for various fields I'd much rather see the rationale for each discrete sizing made explicit vs. forcing all the downstream consumers of this \ information have to deal with special cases and ambiguity. This is not a special case handling as you can see that the list above covers about 40% of the fields defined today in the information model. It is way to handle ambiguity arising out of implementation differences. Thanks Ganesh Regards, Jeff Meyer -----Original Message----- From: Ganesh Sadasivan [ mailto:gsadasiv <at> cisco.com <mailto:gsadasiv <at> cisco.com> ] Sent: Monday, September 29, 2003 6:00 PM To: ipfix <at> net.doit.wisc.edu <mailto:ipfix <at> net.doit.wisc.edu> Subject: [ipfix] Variant Field Types. Hi, The information model <draft-ietf-ipfix-info-01.txt> only has types for fixed length objects and variable length string like objects. There are a number of information element types (listed below) that would be better defined in terms of fixed-size run-time specified objects. The type variation comes due to difference in the size of the objects used because of an implementation, as demanded by the network or otherwise. The way to make this work is to use the template length field to determine the exact sub-type: The definitions of a variant field type can be stated as: type varUnsignedInt if (element.length == 1) type = unsignedByte else if (element.length <= 2) type = unsignedShort else if (element.length <= 4) type = unsignedInt else if (element.length <= 8) type = unsignedLong else type = undefined type varInt if (element.length == 1) type = byte else if (element.length <= 2) type = short else if (element.length <= 4) type = int else if (element.length <= 8) type = long else type = undefined The candidates that fit into the description above are (I may have missed some also): - ingressPort - egressPort - packetCount - byteCount - sourceAS - destinationAS - nextHopAS - droppedPacketCount - samplingInterval - droppedByteCount Please comment. Thanks Ganesh
-- Help mailto:majordomo <at> net.doit.wisc.edu and say "help" in message body Unsubscribe mailto:majordomo <at> net.doit.wisc.edu and say "unsubscribe ipfix" in message body Archive http://ipfix.doit.wisc.edu/archive/

Permalink | Reply | 0 comments |
headers
MEYER,JEFFREY D (HP-Cupertino,ex1 | 1 Oct 2003 20:59
Picon
Favicon

RE: [ipfix] Variant Field Types.

Hi,
 
  I think the details of using smaller encoding in an implementation belong in the protocol specification vs. the information model.  The information model already references the protocol specification for how fields of a given type are encoded.
 
  For specific fields, there is still a decision on what would be the maximal type.  E.g. do ports need to be 32-bit or 16?  Do AS numbers need to be 32 or 64?  Given the unit of measure for SampleInterval, what is its limit?  These details would go in the information model.
 
Regards,
 
  Jeff Meyer
-----Original Message-----
From: Ganesh Sadasivan [mailto:gsadasiv <at> cisco.com]
Sent: Wednesday, October 01, 2003 11:35 AM
To: Mark Fullmer
Cc: MEYER,JEFFREY D (HP-Cupertino,ex1); ipfix <at> net.doit.wisc.edu
Subject: Re: [ipfix] Variant Field Types.

Mark,

Mark Fullmer wrote:
I'm in favor of setting upper bounds, for example counters don't go

I agree that a field even if variable in length is limited by an upper bound and this is limit is set on a
case by case basis. The whole point of my first e-mail is rather than nailing a field to an explicit
type like uint64_t, to let implementors know that  they can send/receive these fields in smaller
sizes too.
past 64 bits. If you don't do this, what's a collector to do with say a 72 bit counter? You can't store it in a uint_64_t anymore so collectors would have to implement arbitrary precision arithmetic to handle potential cases. I don't think this is reasonable to impose on the collectors. The issue I have with the variable length fields as currently specified is the potential for template explosion. If a template has 8 counters, each of which could be 1..8 byte then you have 8^8 potential templates for that one stream of flows.
Yes a separate template is needed for varying length. Template explosion because if this would be
an implementation choice rather than that imposed by the definition itself. Also the same would
have happened if we explicitly define field types for the counters of varying length.

Thanks
Ganesh
I was hoping for the sake of the collectors we could keep the active templates down to something < 64K so a template lookup could be done with an index instead of a hash. mark On Tue, Sep 30, 2003 at 04:20:07PM -0400, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
Ganesh, I'm not clear on how this benefits the collector. Tal and myself produce different implementations of carrier deployed products which collect from 100's of devices and dozens of protocols, perhaps our experience here is worth something, and it tells us variant is undesireable on the collector... Regarding your other point: One way to solve this problem is if the collector assumes the largest size for computations? This was my point in the e-mail to Stewart. Have the information model be EXPLICIT about what is the largest size which is being modeled for a data item and declare it. Allow the exporter to declare a smaller size in the template, if the exporter knows that an individual record will always be accomodated (based on its implementation) in the smaller size. So for instance this might say that byteCount should be declared in the information model as a 64-bit int, while AS is probably sufficient at 32-bits. This accomodates the: exporter - can use smaller size if known n/w - can reduce BW if know a smaller size is used info model - explicit is better than "vague" collector - addresses persistence and aggregation issues Can we call this a compromise? Regards, Jeff Meyer -----Original Message----- From: Ganesh Sadasivan [mailto:gsadasiv <at> cisco.com] Sent: Tuesday, September 30, 2003 10:45 AM To: MEYER,JEFFREY D (HP-Cupertino,ex1) Cc: ipfix <at> net.doit.wisc.edu Subject: Re: [ipfix] Variant Field Types. MEYER,JEFFREY D (HP-Cupertino,ex1) wrote: Hi, Although the use of variant types may seem convenient from the perspective of an exporter, I think the various consumers of the data would benefit from sticking with an explicit type system. It would benefit the exporter, the n/w, flexibility of interpretation of fields in the information model and the collector. Consider operations which involve summing fields, storing them to a DB, etc. One way to solve this problem is if the collector assumes the largest size for computations? Explicit typing ensures that these consuming applications can handle these values appropriately and also makes the information modeller think about how these types are to be used. The examples you cite: - ingressPort - egressPort - packetCount - byteCount - sourceAS - destinationAS - nextHopAS - droppedPacketCount - samplingInterval - droppedByteCount fall into a few categories, identifiers (such as ports and AS's), counts and intervals. For the identifiers, I believe the size is fixed by the relevant RFC's (i.e. why would I have a 1 byte Port?). For counts, the use of 32 vs. 64 bits is a valid question, however these can be handled by making them explicit (i.e. the information model says if I get this value it is a 32-bit quantity or a 64-bit quantity and the code will address it accordingly). For intervals, the resolution would determine which size is most appropriate. For AS the size could be 2 or 4 bytes. You may prefer to use a 1 byte port if you are a small system as against a 4 byte for a gigantic system. Both have their own implementation reasons. The option always exist to define variations on these data items explicitly. E.g. byteCount32 and byteCount64. Yes. But is'nt this not more of an overhead to introduce new field type as the requirements come in and then make sure that it gets deployed in exporter and collector? If there is a need to vary the sizes for various fields I'd much rather see the rationale for each discrete sizing made explicit vs. forcing all the downstream consumers of this \ information have to deal with special cases and ambiguity. This is not a special case handling as you can see that the list above covers about 40% of the fields defined today in the information model. It is way to handle ambiguity arising out of implementation differences. Thanks Ganesh Regards, Jeff Meyer -----Original Message----- From: Ganesh Sadasivan [ mailto:gsadasiv <at> cisco.com <mailto:gsadasiv <at> cisco.com> ] Sent: Monday, September 29, 2003 6:00 PM To: ipfix <at> net.doit.wisc.edu <mailto:ipfix <at> net.doit.wisc.edu> Subject: [ipfix] Variant Field Types. Hi, The information model <draft-ietf-ipfix-info-01.txt> only has types for fixed length objects and variable length string like objects. There are a number of information element types (listed below) that would be better defined in terms of fixed-size run-time specified objects. The type variation comes due to difference in the size of the objects used because of an implementation, as demanded by the network or otherwise. The way to make this work is to use the template length field to determine the exact sub-type: The definitions of a variant field type can be stated as: type varUnsignedInt if (element.length == 1) type = unsignedByte else if (element.length <= 2) type = unsignedShort else if (element.length <= 4) type = unsignedInt else if (element.length <= 8) type = unsignedLong else type = undefined type varInt if (element.length == 1) type = byte else if (element.length <= 2) type = short else if (element.length <= 4) type = int else if (element.length <= 8) type = long else type = undefined The candidates that fit into the description above are (I may have missed some also): - ingressPort - egressPort - packetCount - byteCount - sourceAS - destinationAS - nextHopAS - droppedPacketCount - samplingInterval - droppedByteCount Please comment. Thanks Ganesh
-- Help mailto:majordomo <at> net.doit.wisc.edu and say "help" in message body Unsubscribe mailto:majordomo <at> net.doit.wisc.edu and say "unsubscribe ipfix" in message body Archive http://ipfix.doit.wisc.edu/archive/

Permalink | Reply | 1 comment |
headers
Reinaldo Penno | 2 Oct 2003 18:41

[ipfix] Ipfix Protocol open items

Hello,

Have we zeroed in a certain transport protocol? TCP, SCTP-PR or something else?

I need to write something to explain how Netflow runs over TCP, but it seems to me somewhat of a lower priority if this is not going to be the default protocol. We should focus on explaning how this should work on the "choosen one".

There were also several open items like packet length (size of field, does it replaces the count header, etc, etc).
Template management, etc
Periodic issue of counts per record/template/etc to ensure we know how many records we lost.

It seems to me one of those discussions that die out and we never reach a conclusion. We should have a new version of the protocol draft soon. Can we start reaally reaching a consensus on these things?

I'll talk offline with the other editors and send a list of open issue so we can tackle them on by one.


Regards,

Reinaldo

Permalink | Reply | 1 comment |
headers
Stewart Bryant | 2 Oct 2003 19:13
Picon
Favicon

Re: [ipfix] Ipfix Protocol open items


Reinaldo Penno wrote:

> Hello,
> 
> Have we zeroed in a certain transport protocol? TCP, SCTP-PR or 
> something else?
> 
> I need to write something to explain how Netflow runs over TCP, but it 
> seems to me somewhat of a lower priority if this is not going to be the 
> default protocol. We should focus on explaning how this should work on 
> the "choosen one".

I think that perhaps the question should be put another way. There are
WG members who propose SCTP-PR as the default, and SCTP-PR has
advantage because of its better flexibility in coping with congestion.

So the real question is whether there are WG members who think that
the default should be TCP. If no group comes forward, then we have
our answer. If there is support then we need to perform a detailed
analysis of the relative merits of the two protocols.

Stewart

--
Help        mailto:majordomo <at> net.doit.wisc.edu and say "help" in message body
Unsubscribe mailto:majordomo <at> net.doit.wisc.edu and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/
Permalink | Reply | 0 comments |
headers
Carter Bullard | 2 Oct 2003 19:27

[ipfix] STCP as a default transport not

I believe that SCTP-PR is not deployed widely enough
to be considered a default for any IETF technology.
It is not on my Linux RedHat 9.0 distribution, nor
my Mac OS-X or my Windows XP system.  I don't see it
available on any Solaris machine that I have access
to.

What IETF technology is currently using SCTP as a
default transport?

Carter

> -----Original Message-----
> From: majordomo listserver
> [mailto:majordomo <at> mil.doit.wisc.edu] On Behalf Of Stewart Bryant
> Sent: Thursday, October 02, 2003 12:14 PM
> To: Reinaldo Penno
> Cc: ipfix-chairs <at> net.doit.wisc.edu; ipfix <at> net.doit.wisc.edu
> Subject: Re: [ipfix] Ipfix Protocol open items
>
>
>
>
> Reinaldo Penno wrote:
>
> > Hello,
> >
> > Have we zeroed in a certain transport protocol? TCP, SCTP-PR or
> > something else?
> >
> > I need to write something to explain how Netflow runs over
> TCP, but it
> > seems to me somewhat of a lower priority if this is not
> going to be the
> > default protocol. We should focus on explaning how this
> should work on
> > the "choosen one".
>
> I think that perhaps the question should be put another way. There are
> WG members who propose SCTP-PR as the default, and SCTP-PR has
> advantage because of its better flexibility in coping with congestion.
>
> So the real question is whether there are WG members who think that
> the default should be TCP. If no group comes forward, then we have
> our answer. If there is support then we need to perform a detailed
> analysis of the relative merits of the two protocols.
>
> Stewart
>
>
> --
> Help        mailto:majordomo <at> net.doit.wisc.edu and say "help"
> in message body
> Unsubscribe mailto:majordomo <at> net.doit.wisc.edu and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
>

--
Help        mailto:majordomo <at> net.doit.wisc.edu and say "help" in message body
Unsubscribe mailto:majordomo <at> net.doit.wisc.edu and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/
Permalink | Reply | 1 comment |
headers
MEYER,JEFFREY D (HP-Cupertino,ex1 | 2 Oct 2003 19:32
Picon
Favicon

RE: [ipfix] Ipfix Protocol open items

Hi,

  Given the ubiquity of TCP and the challenge of using SCTP 
"out of the box" on many platforms, I would strongly recommend
making TCP the default.

  I am not aware of Microsoft support for SCTP, nor support on
the Java platform.

  My understanding of some of the experience w/ Diameter interoperability
sessions, is that TCP was the dominant transport, even though 
Diameter calls out both SCTP and TCP as required protocols.

  An alternative view would be that building more on top of SCTP
will drive more adoption and help address the "out of the box" issue.
I'm a bit skeptical on this point, since SCTP seems more kernel
space than user space, it will be more challenging to get OS
and JVM vendors to quickly support it.

Regards,

  Jeff Meyer

> -----Original Message-----
> From: Stewart Bryant [mailto:stbryant <at> cisco.com]
> Sent: Thursday, October 02, 2003 10:14 AM
> To: Reinaldo Penno
> Cc: ipfix-chairs <at> net.doit.wisc.edu; ipfix <at> net.doit.wisc.edu
> Subject: Re: [ipfix] Ipfix Protocol open items
> 
> 
> 
> 
> Reinaldo Penno wrote:
> 
> > Hello,
> > 
> > Have we zeroed in a certain transport protocol? TCP, SCTP-PR or 
> > something else?
> > 
> > I need to write something to explain how Netflow runs over 
> TCP, but it 
> > seems to me somewhat of a lower priority if this is not 
> going to be the 
> > default protocol. We should focus on explaning how this 
> should work on 
> > the "choosen one".
> 
> I think that perhaps the question should be put another way. There are
> WG members who propose SCTP-PR as the default, and SCTP-PR has
> advantage because of its better flexibility in coping with congestion.
> 
> So the real question is whether there are WG members who think that
> the default should be TCP. If no group comes forward, then we have
> our answer. If there is support then we need to perform a detailed
> analysis of the relative merits of the two protocols.
> 
> Stewart
> 
> 
> --
> Help        mailto:majordomo <at> net.doit.wisc.edu and say "help" 
> in message body
> Unsubscribe mailto:majordomo <at> net.doit.wisc.edu and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 

--
Help        mailto:majordomo <at> net.doit.wisc.edu and say "help" in message body
Unsubscribe mailto:majordomo <at> net.doit.wisc.edu and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/
Permalink | Reply | 0 comments |

Gmane