1 Oct 2010 01:21
Re: Nondeterministic output
Carter Bullard <carter <at> qosient.com>
2010-09-30 23:21:09 GMT
2010-09-30 23:21:09 GMT
Hey George,
OMG, did you uncover a most interesting bug !!! Seems that we had a race condition in
one of the memory allocation structures, and poor placement of a mutex, resulting in
us not always getting a message buffer when we want one (exacerbated because
when writing to stdout, we use a lot of message passing buffers to hold the data on
the device output queue), so we would schedule the message from the flow modeler
to the output processor (allowing us to continue to aggregate into the flow record, causing
inconsistent output), or we would throw the message away, resulting in lost flow records.
Wow. Its fixed, and a lot of routines are now very clean as a result.
I'll be uploading a new version of argus tonight onto the server, and I'll send mail to the list.
Carter
On Sep 30, 2010, at 5:55 AM, George Jones wrote:
Thanks. Look forward to testing the fix....my whole process is stdout pipelines...
On Sep 30, 2010 2:34 AM, "Carter Bullard" <carter <at> qosient.com> wrote:
> Hey George,
> Well, there are a number of things going on at the same time, but I'm getting a grip
> on this issue. When writing to a file, argus seems to be very consistent. When
> writing to stdout, it seems to be having some issues with getting all the records out
> of the engine, and it seems that when we zero out a record (when we write a flow
> status record, we maintain the cache, but zero out the metrics) things may not behave
> as we would like, ....., but only when we have to queue records for output. We do this
> when we are delivering flow records to the output socket/file descriptor faster than they
> can be written out the device. When we have partially written a record, and we are
> still queuing outgoing flow records, we get into a bad situation where we only clear
> one record every turn, and we have a turn every 0.020 seconds, so we get really slow.
> That is where the trouble then begins.
>
> So the short story is, when you write to disk, all is good, when you write to stdout, all
> is not. I'm working this now, but it is a head scratcher and so it may take a few days.
>
> Sorry for the inconvenience,
>
> Carter
>
> On Sep 21, 2010, at 11:05 AM, George Jones wrote:
>
>> The following command produces different output:
>>
>> cat foo.pcap | argus -U 64 -r - -w /tmp/foo.ar
>> cat foo.pcap | argus -U 64 -r - -w /tmp/bar.ar
>>
>> cksum(1) shows the content differs (but byte count is the same).
>>
>> More disturbing is different numbers of records output from identical runs on the same input, etc.
>>
>> cat foo.pcap | argus -U 64 -r - -w - | racluster -r - -w - | ra -r - | tee /tmp/1.out
>> cat foo.pcap | argus -U 64 -r - -w - | racluster -r - -w - | ra -r - | tee /tmp/2.out
>>
>> results in slightly different output. Sometimes there are slight differences in the flgs (packet ordering, I think),
>> but in a file of several thousand records, I'm getting 6 or so additional records in one output file vs the other.
>>
>> Confused,
>> ---George Jones
>
>
RSS Feed