headers
Andrew Pollock | 1 Apr 2003 03:09
Picon
Favicon

Re: Using tcpdump input

On Mon, Mar 31, 2003 at 08:32:25AM -0500, Carter Bullard wrote:
> Hey Andrew,
>   This looks an awful lot like double counting, so I would
> suggest eliminating that as a possibility, using the
> strategies that I sent earlier.  Remember, you can have
> argus log the packets that it receives to a file,
> by turning on the ARGUS_PACKET_CAPTURE_FILE variable.
> By turning down the volume a bit, you could use this
> to determine if your getting two copies of the same
> packet.

Carter, we've run Argus with a capture file specified, and sure enough
we're seeing every packet twice, so it is double counting. Revisiting your
previous email on double counting, I don't think anything applies. A 
tcpdump on the same interface isn't seeing the packets twice.

Andrew
Permalink | Reply | 0 comments |
headers
Andrew Pollock | 1 Apr 2003 04:32
Picon
Favicon

Re: Using tcpdump input

On Tue, Mar 25, 2003 at 08:33:37AM -0500, Carter Bullard wrote:
> 
>    The type of argus configurations that can cause double
> counting generally are those where argus opens the same
> interface twice, or in a router and the 'any' interface
> was used.  This can happen accidentally if you read multiple
> argus.conf files, where both have an interface definition.
> If you run argus normally with the "-F conf" option, remove
> the option and see if argus is still getting packets.
> That is a sure sign of a problem.

We've run Argus by hand without a -F option, and are still seeing double 
with Argus natively than what it is seeing with tcpdump input. We're 
running Argus (and tcpdump) on a box that is attached to a switch's span 
port.

>    Hope this helps to solve your dilemma.

Dilemma persists. Hair is thinning.
Permalink | Reply | 0 comments |
headers
Carter Bullard | 1 Apr 2003 05:26

RE: Using tcpdump input

More than likely, argus is opening whatever
interface twice.  This is not impossible,
so the solution will be in understanding how argus
is being called and the contents of your argus.conf
file.  One quick approach is to run argus with
the -X option as the first option on the command
line.  If this resolves the problem the it will
be straight forward.

Carter

> -----Original Message-----
> From: Andrew Pollock [mailto:andrew-argus <at> andrew.net.au] 
> Sent: Monday, March 31, 2003 8:10 PM
> To: Carter Bullard
> Cc: argus-info <at> lists.andrew.cmu.edu
> Subject: Re: Using tcpdump input
> 
> 
> On Mon, Mar 31, 2003 at 08:32:25AM -0500, Carter Bullard wrote:
> > Hey Andrew,
> >   This looks an awful lot like double counting, so I would
> > suggest eliminating that as a possibility, using the
> > strategies that I sent earlier.  Remember, you can have
> > argus log the packets that it receives to a file,
> > by turning on the ARGUS_PACKET_CAPTURE_FILE variable.
> > By turning down the volume a bit, you could use this
> > to determine if your getting two copies of the same
> > packet.
>
(Continue reading)

Permalink | Reply | 2 comments |
headers
Andrew Pollock | 1 Apr 2003 05:58
Picon
Favicon

Re: Using tcpdump input

On Mon, Mar 31, 2003 at 10:26:05PM -0500, Carter Bullard wrote:
> More than likely, argus is opening whatever
> interface twice.  This is not impossible,
> so the solution will be in understanding how argus
> is being called and the contents of your argus.conf
> file.  One quick approach is to run argus with
> the -X option as the first option on the command
> line.  If this resolves the problem the it will
> be straight forward.

Now we're making some progress. I get identical results.
Soooo.... How come Argus under normal operations is opening the interface 
twice?

I've got eth1 in the /etc/argus.conf, and argus is invoked with a -F 
/etc/argus.conf and not with a -i

Andrew
Permalink | Reply | 1 comment |
headers
Andrew Pollock | 1 Apr 2003 06:27
Picon
Favicon

Re: Using tcpdump input

On Tue, Apr 01, 2003 at 01:58:30PM +1000, Andrew Pollock wrote:
> On Mon, Mar 31, 2003 at 10:26:05PM -0500, Carter Bullard wrote:
> > More than likely, argus is opening whatever
> > interface twice.  This is not impossible,
> > so the solution will be in understanding how argus
> > is being called and the contents of your argus.conf
> > file.  One quick approach is to run argus with
> > the -X option as the first option on the command
> > line.  If this resolves the problem the it will
> > be straight forward.
> 
> Now we're making some progress. I get identical results.
> Soooo.... How come Argus under normal operations is opening the interface 
> twice?
> 
> I've got eth1 in the /etc/argus.conf, and argus is invoked with a -F 
> /etc/argus.conf and not with a -i

Furthermore, the blurb in /etc/argus.conf says:

#-----------------------------------------------------------------------------#
# By default, Argus will open the first appropriate interface on a
# system that it encounters.  For systems that have only one network
# interface, this is a reasonable thing to do.  But, when there are
# more than one interface suitable interface, you may want to specify
# which interface(s) Argus should read data from.
#
# Argus can read packets from multiple interfaces at the same time,
# although this is limited to 2 interfaces at this time.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Andrew Pollock | 1 Apr 2003 06:45
Picon
Favicon

The double-counting saga

Sigh.

We have gotten to the bottom of the problem, it would seem.

The problem would appear to be specific to Debian's Argus implementation
(predating my maintenance of the packages) whereby the /etc/init.d/argus
script is invoking Argus with a -F /etc/argus.conf, but Argus is also
compiled with /etc/argus.conf as it's config file, so it's essentially
reading the configuration twice, once implicitly and once explicitly,
hence it opens the specified interface twice, and counts everything twice.

Is there an easy way to remove duplicates from existing Argus logs?

Andrew
Permalink | Reply | 2 comments |
headers
Carter Bullard | 1 Apr 2003 06:48

RE: Using tcpdump input

Hey Andrew,
   By default, argus opens and processes the system
/etc/argus.conf file.  Using the "-F /etc/argus.conf",
you are asking argus to process this file twice, so
its getting the open interface directive two times.

   This is becoming a gottcha that needs to be
eliminated as this is the second time its come
up in a few years.  I'll look into it.

Carter

> -----Original Message-----
> From: Andrew Pollock [mailto:andrew-argus <at> andrew.net.au] 
> Sent: Monday, March 31, 2003 10:59 PM
> To: Carter Bullard
> Cc: argus-info <at> lists.andrew.cmu.edu
> Subject: Re: Using tcpdump input
> 
> 
> On Mon, Mar 31, 2003 at 10:26:05PM -0500, Carter Bullard wrote:
> > More than likely, argus is opening whatever
> > interface twice.  This is not impossible,
> > so the solution will be in understanding how argus
> > is being called and the contents of your argus.conf
> > file.  One quick approach is to run argus with
> > the -X option as the first option on the command
> > line.  If this resolves the problem the it will
> > be straight forward.
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Carter Bullard | 1 Apr 2003 06:50

RE: The double-counting saga

So are you getting two duplicate records or are you getting
records with 2x counts?  The duplicate records are easy
to remove, we could write a simple client to
adjust the counts and bytes.

Carter

> -----Original Message-----
> From: owner-argus-info <at> lists.andrew.cmu.edu 
> [mailto:owner-argus-info <at> lists.andrew.cmu.edu] On Behalf Of 
> Andrew Pollock
> Sent: Monday, March 31, 2003 11:45 PM
> To: argus-info <at> lists.andrew.cmu.edu
> Subject: The double-counting saga
> 
> 
> Sigh.
> 
> We have gotten to the bottom of the problem, it would seem.
> 
> The problem would appear to be specific to Debian's Argus 
> implementation
> (predating my maintenance of the packages) whereby the 
> /etc/init.d/argus
> script is invoking Argus with a -F /etc/argus.conf, but Argus is also
> compiled with /etc/argus.conf as it's config file, so it's essentially
> reading the configuration twice, once implicitly and once explicitly,
> hence it opens the specified interface twice, and counts 
> everything twice.
>
(Continue reading)

Permalink | Reply | 1 comment |
headers
Andrew Pollock | 1 Apr 2003 07:31
Picon
Favicon

Re: The double-counting saga

On Mon, Mar 31, 2003 at 11:50:56PM -0500, Carter Bullard wrote:
> So are you getting two duplicate records or are you getting
> records with 2x counts?  The duplicate records are easy
> to remove, we could write a simple client to
> adjust the counts and bytes.

It's packet duplication, not record duplication.

Andrew
Permalink | Reply | 0 comments |
headers
Andrew Pollock | 2 Apr 2003 07:16
Picon
Favicon

ragator

Carter,

My understanding of ragator when used without a flowfile is that it'll 
just aggregate multiple records for the same flow into one, where it can.

Is this correct?

I was looking at a month's worth of logs for a client today, and with 
racount, it said:

racount    records       total_pkts         src_pkts         dst_pkts      
total_bytes        src_bytes        dst_bytes
    sum    4339943        430446602        284435604        146010998     
118126342743      53028657986      65097684757

When I ran it through ragator first, I got:

racount    records       total_pkts         src_pkts         dst_pkts      
total_bytes        src_bytes        dst_bytes
    sum    3959765        430446602        284418909        146027693     
118126342743      53003592942      65122749801

(Sorry for the formatting).

The aggregated record count was lower, which is what I expected, however 
the packet and byte counts no longer matched, which I didn't expect.

The total packets and total bytes still match, but not the src and dst 
counts.
(Continue reading)

Permalink | Reply | 0 comments |

Gmane