headers
Anton Okmianski | 5 Oct 2004 22:41
Picon
Favicon

RE: Sturctured Data & Unicode

Rainer:

I second David's opinion.  I don't see how control characters in tag
names are any different than control characters in message content
which we allow.  

BTW, I am back from 2 week vacation. Sorry for a long silence on the
message size and fragmentation issue.  I am going to revisit it this
week. I am consulting with more people. 

Thanks,
Anton. 

> -----Original Message-----
> From: syslog-sec-bounces <at> willers.employees.org 
> [mailto:syslog-sec-bounces <at> willers.employees.org] On Behalf 
> Of Rainer Gerhards
> Sent: Monday, September 20, 2004 12:48 PM
> To: ietfdbh <at> comcast.net; syslog-sec <at> employees.org
> Subject: RE: [Syslog-sec] Sturctured Data & Unicode
> 
> 
> David,
> 
> I mostly agree with you. I have updated the draft in your 
> spirit. There is one issue left - that is control characters 
> in the structured data names. I would like to make sure that 
> only printable characters are used for names. But what does 
> printable mean in Unicode? As of my understanding, stating 
> something like "printable Unicode characters" does not really
(Continue reading)

Permalink | Reply | 751 comments |
headers
Anton Okmianski | 12 Oct 2004 23:25
Picon
Favicon

UDP message size issue proposal

Hi!

Sorry for a long delay on this issue - I was on a 2 week vacation.  I have spoken with a number of TCP/UDP/IP
experts regarding the sizing issue.  I am ready to propose the following changes:

1. Syslog-protocol will state that the max message will be defined by the transport layer.  

2. Syslog-transport-udp will support messages up to maximum UDP datagram size of 64K. UDP is a very bad
choice for large message transmissions, so it does not make sense for us to stretch it by adding our own
fragmentation without other transmission control features such as found in TCP.

3. Syslog-transport-udp will rely on IP fragmentation and we will get rid of "proprietary" fragmentation
which was designed to handle messages over 64K and deal with various non-compliant network hosts.    

4. Syslog-transport-udp will recommend sending messages within the boundaries of prevalent MTU size on a
given network to be safe. It will recommend Ethernet's 1500 bytes less headers and will draw reader
attention to the minimum MTU size hosts on the network are required to support for IPv6 (576 bytes) and IPv6
(1280 bytes).   

5. Path MTU discovery may not work robustly and some TCP/IP stacks may not support UDP packets of full 64K
size and truncate them.  We will mention this and bite this bullet.  We should not restrict the protocol due
to incompliant implementations because it is a moving target and penalizes compliant implementations
with extra overhead. The above size recommendation would partially deal with this, but leave the final
choice up to the administrator.      

6. We will get rid of most syslog transport headers for UDP as they will no longer be needed.  The only thing
that will be left is the transport protocol version prefixed to every syslog message.  Should we even
bother with that?  

This is a major change to the syslog-transport-udp.  I'd like to get positive feedback before I proceed with
(Continue reading)

Permalink | Reply | 0 comments |
headers
Rainer Gerhards | 13 Oct 2004 11:17
Gravatar

RE: UDP message size issue proposal

Anton,

I agree to your conclusion.

Although I am the one who always voted for larger sizes, the discussion
has shown that this yields to unnecessary complexity. I can satisfy my
needs with a vendor-extension structured data element, which I think
shows the flexibility of the new drafts.

So I support your move. I would tend to even remove the transport
version header. If there is need to, we could always include that in a
later release (e.g. "v1", which would make it clearly distingshable from
the current frame format). I see little need to do so, though.

Regarding -protocol, I think we should still keep an upper limit in the
spec. Even I don't see any reason why a syslog message should go above
16MB. So for -protocol, I propose the following new text:

####
5.1  Message Length

   The maximum length of any syslog message is 16,777,216 octets.  Any
   receiver receiving a larger message MUST discard the message.  A
   diagnostic message SHOULD be logged in this case.

   A receiver MUST be able to accept messages that are 480 octets (or
   less) in length.  A receiver SHOULD be able to accept messages that
   are 65,535 octets (or less) in length.  It is RECOMMENDED that
   receivers be able to accept messages up to the maximum message length
   of 16,777,216 octets.
(Continue reading)

Permalink | Reply | 628 comments |
headers
Rainer Gerhards | 13 Oct 2004 11:19
Gravatar

syslog-interntional

Hi List,

this is more or less a housekeeping call. I think -protocol addresses
all internationalization issues now. We still lists syslog-international
on the WG home page. I think now would be the right time to abandon that
project, as there is no more need for it.

Rainer
_______________________________________________
Syslog-sec mailing list
Syslog-sec <at> www.employees.org
http://www.employees.org/mailman/listinfo/syslog-sec
Permalink | Reply | 0 comments |
headers
Anton Okmianski | 13 Oct 2004 16:37
Picon
Favicon

RE: UDP message size issue proposal

Rainer:

Glad you liked the latest proposal. I received two other votes for it
as well (from Andrew Ross and John Moehrke) and no disagreements.
So, I will go ahead with the proposal. 

I will remove version field from udp transport as well.  I agree that
in the unlikely scenario that we would need a new version of UDP
transport, we can add the version field at the very front which does
not conflict with syslog-protocol headers.  

One clarification question on suggested text below. 

> ####
> 5.1  Message Length
> 
>    The maximum length of any syslog message is 16,777,216 octets.
Any
>    receiver receiving a larger message MUST discard the message.  A
>    diagnostic message SHOULD be logged in this case.
> 
>    A receiver MUST be able to accept messages that are 480 octets
(or
>    less) in length.  A receiver SHOULD be able to accept messages
that
>    are 65,535 octets (or less) in length.  It is RECOMMENDED that
>    receivers be able to accept messages up to the maximum 
> message length
>    of 16,777,216 octets.
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
David B Harrington | 13 Oct 2004 16:38
Picon

RE: UDP message size issue proposal

Hi,

The RFC2119 keyword usage needs to be tightened up. I suggest:

>From RFC2119:
"MUST   This word, or the terms "REQUIRED" or "SHALL", mean that the
definition is an absolute requirement of the specification."
"SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course."

MUST is used to specify compliance requirements. Your text " A
receiver MUST be able to accept messages that are 480 octets (or less)
in length." could be interpreted to mean that I am compliant if I
accept only messages of 4 bytes, since 4 bytes would fit within your
"480 octets (or less)" requirement. "up to and including 480 octets"
is clearer (and consistent with the the same restriction in RFC3417). 

SHOULD and RECOMMENDED are equivalent. So I suggest that 
 " A receiver SHOULD be able to accept messages that
   are 65,535 octets (or less) in length.  It is RECOMMENDED that
   receivers be able to accept messages up to the maximum message
length
   of 16,777,216 octets." 
be reduced to 
 " A receiver SHOULD be able to accept messages up to the maximum
message length of 16,777,216 octets." 

If they cannot support that size, which would be a valid reason to
(Continue reading)

Permalink | Reply | 626 comments |
headers
David B Harrington | 13 Oct 2004 17:09
Picon

Maximum message size

Hi,

I am concerned that we are in danger of promoting uses of syslog that
will defeat its current usefulness as a human-readable log of system
activity.

Do network operators really believe the 16,777,216 octet size is a
needed improvement to syslog, or are we designing this size in
explicitly for the vendors of applications who want to use syslog as a
programmatic interface? I believe Syslog should be designed to meet
the needs of operators. Most of the discussion ssems to be from
vendors of syslog receivers or applications. Other protocols such as
FTP might be better used for such a specialty use case. 

Does allowing messages of 16,777,216 octets to be accumulated within
the system log make it harder to use some widely-available minimal
text editors, like Notepad, to view the system log? Will having huge
application-specific messages in the system log make it harder for an
operator to locate useful troubleshooting messages within the system
log? Can the information in such a huge message fit within an 80x25
screen, when an operator is troubleshooting network problems and needs
to use a bare-bones terminal attached to the serial port of a device
to view the system log? 

Syslog was designing to be an operator's tool, and Syslog should be
designed to meet the needs of operators.  Will allowing messages of
this size negatively impact its usefulness in the primary use case?  

What do network **operators** think about the need for messages of
this size? Have we deliberately asked them their opinion on this by,
(Continue reading)

Permalink | Reply | 1 comment |
headers
Marshall Glen | 13 Oct 2004 18:56
Picon

RE: Maximum message size


While I'd agree that a 16M octet size is extreme, we do need to accommodate
larger syslog messages.  See RFC3881 for an example.  

Communication of healthcare application security audit data to a central
repository is the key use case for RFC3881.  The end-users are healthcare
security and privacy officers, and they will need aggregated data from
multiple IT-enable healthcare systems to assess security conformance and
track patterns of inappropriate data access.  The design of an XML schema to
meet the end-user information needs leads to a message size on the order of
a few kilobytes.  It needs to be sent reliably and securely to a central
repository for after-the-fact analysis.

Best regards,
Glen Marshall

-----Original Message-----
From: David B Harrington [mailto:ietfdbh <at> comcast.net]
Sent: Wednesday, October 13, 2004 11:10 AM
To: 'Rainer Gerhards'; 'Anton Okmianski'; syslog-sec <at> employees.org
Subject: [Syslog-sec] Maximum message size 

Hi,

I am concerned that we are in danger of promoting uses of syslog that
will defeat its current usefulness as a human-readable log of system
activity.

Do network operators really believe the 16,777,216 octet size is a
needed improvement to syslog, or are we designing this size in
(Continue reading)

Permalink | Reply | 0 comments |
headers
Andrew Ross | 14 Oct 2004 02:35

RE: Maximum message size


David, 

You make a good point on the end use of syslog messages. I'm not a big
fan of large messages, I'd be happy limiting it to 64K even over TCP. It
makes it readable by humans and parsable by standard file parsing
programs.

Some standard text editors will only allow a max line length of 4096
characters.

If we are to assume that a "syslog" message is intended to be displayed
on a console, logged to a file and maybe included in some sort of
paging/e-mail alert system, then the message size really needs to be
kept fairly short.

Rainer's experience is that some devices/apps will send very large
diagnostic messages of around 1MB in size.

I can understand the need in this case to be able to handle large
message sizes. However, I don't plan on accepting messages any larger
than 64K in my implementation of -protocol.

A client would have to make a REALLY good case for me to want to accept
larger messages, especially 16MB in size. FTP, or even HTTP POST would
be the best solution in this case. 

Syslog messages are meant for displaying, logging to disk and alerting.
Not large binary style core dump info or database style records.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Rainer Gerhards | 14 Oct 2004 13:34
Gravatar

RE: Maximum message size

David and all,

many thanks for all the good comments.

I agree that 16 MB for a single message is far more than actually should
be used. Let's look a bit at the history of this:

Initially, we noticed that the 1024 octet limit for current syslog is
too short. I think there is still general agreement on this. We had a
good discussion about the size issue in September 2003. A link to an
archive is here:

http://www.syslog.cc/ietf/autoarc/msg00855.html

The general issue is that 1K is too short for many applications. For
example, Marshall Glen mentions the healthcare needs (RFC3881) for a
larger message. As far as I know, the IHE initiative currently
standardizes on a maximum XML stream size of 32K (which is then supposed
to be transmitted via syslog). I think a message size limit of 64K would
probably be suffcient for a long time to come (and leaves some headroom
for message content increases).

Besides that, I myself identified a need - in very rare occasions - to
send information larger than 64K. In extreme cases, this information
could be as large as one megabyte. Again, these were cases that might
happen once in a year. All of this, of course, only if an operator
*really* wants to submit that large information. So the initial approach
and suggestion was to introduce a way to build "message groups". That
would have been a way to split "oversize messages" (as they were called
then) into multiple syslog messages. They key to that concept was that
(Continue reading)

Permalink | Reply | 4 comments |

Gmane