headers
Carter Bullard | 1 Jun 2003 20:17

RE: filter expressions and flows

Hey Jose,
   Sorry for the delay in responding!  The filters do
have a personality of their own, but the rules are pretty
simple.  Since the starting base for the filter compiler
was tcpdump, there are a lot of similarities, but the
ra* programs have a different abstraction to work with,
so there are some real differences.

   The filter expression "host x.y.z.w" implies "ip host
x.y.z.w", that is why "arp and host x.y.z.w" is going to
give you an expression error.  But "arp host x.y.z.w"
is stating that you want the host value returned in an
arp request, so the and is ok!

   The filter wants to fill in the blanks, if any are
left out, such as "host x.y.z.w".  What kind of host?
protocol? version?  Without this info, it has to fill
in the blanks.  Now the error "ip proto tcp" comes from
the fact that "tcp" generates "ip proto tcp" in the
lexical analyzer, which then generates "ip proto ip proto
tcp".  This should be fixed, but since tcpdump generates
the same error here, suggests that there is a fundamental
problem with the compiler strategy.  But I'll look into
it.

You ask,

> It seems that using the host particle alone is the same as 
> "ip host" and
> again this is different from tcpdump.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Andrew Pollock | 5 Jun 2003 03:40
Picon
Favicon

Me and my usual questions

Hi,

I'm the never ending purgatory that is my life, I'm trying to make Argus 
produce consistently sensible data, and as usual, failing miserably.

I'm going to go back to square one:

We have a network topology like this:

              LB        LB
               |         |
            +--------------+
            |              |
Argus ------| Switch       |
            |              |
            +--------------+
             | | | | | | | 
             |     |   |
           Client  | Client  
               Client

We have a switch, to which two load balancers are attached (one being 
active at any point in time) and we have our clients connected to the 
switch as well. Argus is running on a server plugged into a port on the 
switch that spans the two ports that the load balancers are plugged into.

Each client has a /24 of their own, with the exception of a couple of 
clients that have a single /32

Today's focus is one of the clients that have just a /32
(Continue reading)

Permalink | Reply | 0 comments |
headers
Carter Bullard | 5 Jun 2003 17:21

RE: Me and my usual questions

Hey Andrew,
   So how do they differ?  are the totals the same
but the src and dst counters mixed, or is one low?
They should all count the same total pkts and bytes.

Carter

> -----Original Message-----
> From: owner-argus-info <at> lists.andrew.cmu.edu 
> [mailto:owner-argus-info <at> lists.andrew.cmu.edu] On Behalf Of 
> Andrew Pollock
> Sent: Wednesday, June 04, 2003 9:40 PM
> To: argus-info <at> lists.andrew.cmu.edu
> Subject: Me and my usual questions
> 
> 
> Hi,
> 
> I'm the never ending purgatory that is my life, I'm trying to 
> make Argus 
> produce consistently sensible data, and as usual, failing miserably.
> 
> I'm going to go back to square one:
> 
> We have a network topology like this:
> 
>               LB        LB
>                |         |
>             +--------------+
>             |              |
(Continue reading)

Permalink | Reply | 0 comments |
headers
Carter Bullard | 5 Jun 2003 17:23

RE: Me and my usual questions

Andrew,
   Svc only counts udp and tcp traffic, as you have
to have a port to do the svc counting.  Try your ramon
calls with the "tcp or udp" filter to verify.

Carter

> -----Original Message-----
> From: owner-argus-info <at> lists.andrew.cmu.edu 
> [mailto:owner-argus-info <at> lists.andrew.cmu.edu] On Behalf Of 
> Andrew Pollock
> Sent: Wednesday, June 04, 2003 9:40 PM
> To: argus-info <at> lists.andrew.cmu.edu
> Subject: Me and my usual questions
> 
> 
> Hi,
> 
> I'm the never ending purgatory that is my life, I'm trying to 
> make Argus 
> produce consistently sensible data, and as usual, failing miserably.
> 
> I'm going to go back to square one:
> 
> We have a network topology like this:
> 
>               LB        LB
>                |         |
>             +--------------+
>             |              |
(Continue reading)

Permalink | Reply | 2 comments |
headers
Andrew Pollock | 6 Jun 2003 11:04
Picon
Favicon

Re: Me and my usual questions

On Thu, Jun 05, 2003 at 11:23:08AM -0400, Carter Bullard wrote:
> Andrew,
>    Svc only counts udp and tcp traffic, as you have
> to have a port to do the svc counting.  Try your ramon
> calls with the "tcp or udp" filter to verify.

Carter,

If you look at the original message I am using "tcp or udp and host blah". 
Is this correct, or do I need to use some parenthesis?

Andrew
Permalink | Reply | 1 comment |
headers
Russell Fulton | 6 Jun 2003 11:44
Picon
Picon
Favicon

Re: Me and my usual questions

On Fri, 2003-06-06 at 21:04, Andrew Pollock wrote:
> On Thu, Jun 05, 2003 at 11:23:08AM -0400, Carter Bullard wrote:
> > Andrew,
> >    Svc only counts udp and tcp traffic, as you have
> > to have a port to do the svc counting.  Try your ramon
> > calls with the "tcp or udp" filter to verify.
> 
> Carter,
> 
> If you look at the original message I am using "tcp or udp and host blah". 
> Is this correct, or do I need to use some parenthesis?

MY golden rule if in doubt use parenthesis!  Most boolean
implementations have and binding tighter than or so the above is
probably tcp or (udp and host blah)

--

-- 
Russell Fulton, Network Security Officer, The University of Auckland,
New Zealand.
Permalink | Reply | 0 comments |
headers
Carter Bullard | 6 Jun 2003 14:59

RE: Me and my usual questions

Hey Andrew,
  But the totals are the same?  This is probably due to
the way that src and dst are assigned in the various
tools.  The ramon tools are answering the unique
question, "what metrics apply to this interface", which
has a different concept of source and destination than
what a flow represents.

   With the "ragator | ragator | rasort | ra" you still
have the concept that the source is the first sender.  So
when conversations are originated from an external
network to your host, the packets they send will be counted
in the same counter as the packets sent by your host when 
it initiates the conversation.  Seems confusing, but the
differentiation allows for very powerful accountability.

   In order to validate this, run your "rag | rag | ras | ra"
twice, one with filter, "and src host 10.11.2.243" and then
again with "and dst host 10.11.2.243" and see how these
counters compare.  You should find out if this is where
the discrepancies lie.

Carter

   I 

> -----Original Message-----
> From: Andrew Pollock [mailto:andrew-argus <at> andrew.net.au] 
> Sent: Friday, June 06, 2003 5:02 AM
> To: Carter Bullard
(Continue reading)

Permalink | Reply | 1 comment |
headers
Mahlon E. Smith | 7 Jun 2003 08:04
Picon
Favicon
Gravatar

Recommendations for monthly totals?


Hi everyone.

I'm new to argus, and new to the list.  I'm trying to get an automated
monthly report generated, that gives traffic accounting totals in
GB on a per IP basis.  Essentially, your run of the mill hosting
setup that needs to know when customers go over their alloted bandwidth.

I'm currently using the 'argusarchive' script to organize saved data files,
and running a cronned perl script once a month that pulls the totals for
the previous month, parses the goods, and sends the totals out.  It works
just fine - with the exception of one thing.  It takes a really, really
long time to run.  This isn't a slow machine that is doing the processing,
but doing 50 IP addresses takes almost 2 hours.  Where as that isn't
completely unacceptable, I can see that isn't going to scale in the long
run.

Here's what the perl script calls for each IP:
% racount -r .../argus/archive/YEAR/MONTH/*/*.bz2 - host IPADDRESS

The only other way I can think to do this is to write a separate daemon
that opens a pipe to an 'ra -c' client, and keeps it's own totals in dbm files
or somesuch... but I'd like to avoid that if there is a better way.

How are others doing this?

Thanks for any tips provided!

-Mahlon
(Continue reading)

Permalink | Reply | 1 comment |
headers
Mahlon E. Smith | 9 Jun 2003 07:11
Picon
Favicon
Gravatar

Re: Recommendations for monthly totals?

On Sat, 2003-06-07 at 06:04, Mahlon E. Smith wrote: 
> The only other way I can think to do this is to write a separate daemon
> that opens a pipe to an 'ra -c' client, and keeps it's own totals in dbm files
> or somesuch... but I'd like to avoid that if there is a better way.
> 
> How are others doing this?

It was suggested to me by Neil Long to simply take samples daily, tally
them to a separate file, and just poll that file at the end of the
month.  

I went ahead and implemented this - based on speed info gleaned from how
long it takes my current IPs in use - a full class C would take about 18
minutes of processing power per night.  The monthly report is
instantaneous.   This is much, much better than the almost 8 hours it
would have taken to generate a monthly report in my original
implementation.  :)

I'd still be interested to hear some other real world examples of how
others are doing this sort of thing - I'm sure this is one of those
instances where there is more than one good way to do it.

-Mahlon

Mahlon E. Smith                        jabber id: mahlon <at> chat.martini.nu
http://www.martini.nu/               get pgp key:  mahlon-pgp <at> martini.nu
........................................................................
  See the Stinkymeat; Oh sickening putrid mass; You are what you eat.
- Bob Noble
(Continue reading)

Permalink | Reply | 0 comments |
headers
Andrew Pollock | 11 Jun 2003 03:13
Picon
Favicon

Re: Me and my usual questions

Hi Carter,

to summarise, I've got the following:

[1] ramon -w - -M TopN -r 2003-05-03 - host 10.11.2.243 | racount
[2] ragator -w - -r 2003-05-03 - \(tcp or udp\) and host 10.11.2.243 | 
ragator -f /usr/local/etc/dport.conf -w - | rasort -M bytes -w - | ra -s 
dport pkts bytes

[3] ra -r 2003-05-03 -w - - host 10.11.2.243 | ramon -M Svc -r -

I have been using [1] to arrive at my totals

I'm now trying to break down the total arrived at by [1], and I've tried 
[2] and [3]

If I add up the output of [2] or [3], it doesn't match the output of [1]
Even when I add both directions together, I come up with a total that's 
slightly less than the total bytes in [1]. It happens that the totals from 
the output of [2] and [3] match exactly.

If I play with [2] and add "src host blah" or "dst host blah" to the 
filter, I obviously get different results. I'm not sure what I was 
supposed to figure out from doing that.

Andrew

On Fri, Jun 06, 2003 at 08:59:59AM -0400, Carter Bullard wrote:
> Hey Andrew,
>   But the totals are the same?  This is probably due to
(Continue reading)

Permalink | Reply | 0 comments |

Gmane