headers
Fabien COELHO | 1 Sep 2003 17:09
Picon
Favicon

ra stdin does not work ?


Hi folks,

With version 2.0.6.beta41 on debian

shell> cat /var/log/argus/argus.log | ra -r - -
<EMPTY>

although

shell> ra -r /var/log/argus/argus.log -
...
<OK>

It seems that the '-' for stdin processing of ra and racount does not
work.

--

-- 
Fabien Coelho - coelho <at> cri.ensmp.fr
Permalink | Reply | 1 comment |
headers
Peter Van Epp | 1 Sep 2003 20:02
Picon
Picon
Favicon

Re: ra stdin does not work ?

On Mon, Sep 01, 2003 at 05:09:30PM +0200, Fabien COELHO wrote:
> 
> Hi folks,
> 
> With version 2.0.6.beta41 on debian
> 
> shell> cat /var/log/argus/argus.log | ra -r - -
> <EMPTY>
> 
> although
> 
> shell> ra -r /var/log/argus/argus.log -
> ...
> <OK>
> 
> It seems that the '-' for stdin processing of ra and racount does not
> work.
> 
> -- 
> Fabien Coelho - coelho <at> cri.ensmp.fr

	You look to be correct you have uncovered a bug. On beta.13
/ clients.beta.44 (the latest versions) beta.13 ra works correctly but 
beta.44 ra doesn't for stdin:

ar2d2# pwd
/usr/local/src/argus-clients-2.0.6.beta.44/bin
r2d2# ./ra -r argus.out -n | more
18 Aug 03 20:56:22           man  229.97.122.203  v2.0                   1 0
 0        0         0            0           STA
(Continue reading)

Permalink | Reply | 0 comments |
headers
Carter Bullard | 2 Sep 2003 03:23

RE: ra stdin does not work ?

Hmmm mmm, just got back from a little vacation.  I'll
check this out tonight and get a fix out ASAP.

Carter

> -----Original Message-----
> From: owner-argus-info <at> lists.andrew.cmu.edu
> [mailto:owner-argus-info <at> lists.andrew.cmu.edu] On Behalf Of
> Peter Van Epp
> Sent: Monday, September 01, 2003 1:03 PM
> To: argus-info <at> lists.andrew.cmu.edu
> Subject: Re: ra stdin does not work ?
>
>
> On Mon, Sep 01, 2003 at 05:09:30PM +0200, Fabien COELHO wrote:
> >
> > Hi folks,
> >
> > With version 2.0.6.beta41 on debian
> >
> > shell> cat /var/log/argus/argus.log | ra -r - -
> > <EMPTY>
> >
> > although
> >
> > shell> ra -r /var/log/argus/argus.log -
> > ...
> > <OK>
> >
> > It seems that the '-' for stdin processing of ra and
(Continue reading)

Permalink | Reply | 0 comments |
headers
Carter Bullard | 2 Sep 2003 03:29

RE: ra dies off doing analysis

Hey Eric,
   There is a shell script called argusbug, and is very good
at collecting the information that you provided.  We all are
suppose to be using it, so give it a try.

   At this point, the best recourse is to try to find the file
that is causing the problem.  Once you've found that, then
run racount() with the same parameters to see if its all
the programs, or just ra().  Then the method of choice is
o whitling it down to a usable size, where it still
generates the bug, and then possibly sharing the file so
we can debug it on the list.

   If its in the filter, which is rather complex to debug,
one thing to do is to use the '-b' option to see what
the compiler is actually generating.  That really can
help.

Carter

> -----Original Message-----
> From: owner-argus-info <at> lists.andrew.cmu.edu
> [mailto:owner-argus-info <at> lists.andrew.cmu.edu] On Behalf Of Eric
> Sent: Thursday, August 28, 2003 8:45 PM
> To: argus-info <at> lists.andrew.cmu.edu
> Subject: ra dies off doing analysis
>
>
> I've got a core from ra.
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Geoff Powell | 2 Sep 2003 04:05
Picon

Processing speed of ra utilities

Hi all,

I'm using scripts to do a lot of similar processes on the same argus data
file (which is quite large), and I'm wondering if anyone knows of a way I 
can speed up the process, and reduce the time it takes for ra utilities 
to produce results.

Some examples of the commands I'm doing:
racount -n -r large-file.out - src net c.class.ip.1/32
racount -n -r large-file.out - dst net c.class.ip.2/32
racount -n -r large-file.out - src net c.class.ip.3/32
...all the way to 254/32.

After that I might look at specific ports/ip protocols for each IP address
in the c class.

I'm guessing racount has to process each transaction? When the argus 
data file size is 50Mb+, even though the computer doing the processing is 
reasonably fast (Dual Xeon 1.5Ghz with 2gb of ram), each racount command 
usually takes around 30sec-1min.

Is there way I can speed up the process, like running multiple racounts, 
using ragator or another application?

Thanks for any help

Regards,
Geoff (geoff <at> lanrex.net.au)
(Continue reading)

Permalink | Reply | 0 comments |
headers
Carter Bullard | 2 Sep 2003 04:48

RE: Processing speed of ra utilities

Hey Geoff,
   Looking at your samples, ragator() can definitely
do some bulk processing for you.  With a simple
ragator.conf such as:

 Flow  100 ip  *   *   *   *    *     200  0   0
 Model 200 ip  255.255.255.0  0.0.0.0   no no no

you can generate stats for all the source class-c nets.

 Flow  100 ip  *   *   *   *    *     200  0   0
 Model 200 ip  0.0.0.0 255.255.255.0    no no no

will get you all the dst class-c net stats, in one
pass of the data.  If you want the data sorted by
network, just pipe the output through rasort(),
and with our first example ragator.conf file, you won't
be interested in the dst addr, so give this a try:

ragator -f ragator.conf -r large-file.out -w - | \
     rasort -M saddr -s -dir -s -daddr

   All the ra* programs process argus data files
sequentially, reading each record and doing whatever
processing they are designed to perform, and yes
one performance bottle neck with the simple samples
provided in the argus-clients distribution is that
they really only do one thing for each pass
of the data.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Geoff Powell | 2 Sep 2003 06:32
Picon

RE: Processing speed of ra utilities

G'day Carter

I think I understand what you are saying about only doing one thing per
pass of the argus flows. If I understand correctly, it would be ideal
if racount suppressed the data before counting, and if it allowed 
the user to specify multiple nets (but then I guess memory usage becomes 
an issue)

Perhaps even a ragator utility that was able to supress a complete 
data file (which could be written to disk), then different racount 
commands could be used on that file.

I've done a bit of c programming before, I'll have a look at the existing
ra utility source code and see if I can make sense of it.

Thanks for the info

Regards,
Geoff

On Mon, 1 Sep 2003, Carter Bullard wrote:

> Hey Geoff,
>    Looking at your samples, ragator() can definitely
> do some bulk processing for you.  With a simple
> ragator.conf such as:
> 
>  Flow  100 ip  *   *   *   *    *     200  0   0
>  Model 200 ip  255.255.255.0  0.0.0.0   no no no
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Andrew Pollock | 2 Sep 2003 09:28
Picon
Favicon

ramon -M HostP

Hi Carter,

Hope you had a nice vacation.

I've only just gotten around to playing with the ramon -M HostP option you 
put in (and I'm currently playing with 2.0.6.beta.41 on this particular 
box).

I think you've interpretted "protocol" as "ip protocol" whereas I've been 
meaning "TCP or UDP protocol", so I'm getting a lovely breakdown that 
shows mostly TCP traffic. I'm interested in working out how much of the 
traffic total generated from running an "ramon -M TopN | racount" type 
thing is HTTP, SMTP etc etc.

Sorry if I wasn't clear on this in the first place, or am I just driving 
this new option wrong?

regards

Andrew
Permalink | Reply | 0 comments |
headers
Carter Bullard | 2 Sep 2003 15:29

RE: Processing speed of ra utilities

Hey Geoff,
   All ra* programs are stream processors, at least from the
perspective of the academic relational database community,
and they have to sequentially work on each datum in order to
do their thing.  I'm not sure what you mean by 'suppress' the
data, but in order to not process the data sequentially and
get things like total bytes, etc ... you have to aggregate
the data, and then you have to worry about what identifiers
you want to keep, which ones you can throw away etc.....

ragator() is our example of ra aggregation, and so you
definitely should give it a try first, as it will do what
you indicated in your first mail.

If you do start writing your own ra* clients, don't hesitate
to send mail!!!!!!

Carter

> -----Original Message-----
> From: Geoff Powell [mailto:geoff <at> lanrex.net.au]
> Sent: Monday, September 01, 2003 11:33 PM
> To: Carter Bullard
> Cc: argus-info <at> lists.andrew.cmu.edu
> Subject: RE: Processing speed of ra utilities
>
>
> G'day Carter
>
> I think I understand what you are saying about only doing one
(Continue reading)

Permalink | Reply | 0 comments |
headers
Carter Bullard | 2 Sep 2003 17:28

RE: ramon -M HostP

Hey Andrew,
No problem, you're just jumping up the stack rather than
moving up it ;o)  So, you're really interested in
host/proto/dport.   Lets call it HostSvc and change
the name of HostP to HostProto, to remove any ambiguity.
I'll have this new approach in the next release, which
should be tonight or tomorrow.

Carter

> -----Original Message-----
> From: owner-argus-info <at> lists.andrew.cmu.edu
> [mailto:owner-argus-info <at> lists.andrew.cmu.edu] On Behalf Of
> Andrew Pollock
> Sent: Tuesday, September 02, 2003 2:28 AM
> To: argus-info <at> lists.andrew.cmu.edu
> Subject: ramon -M HostP
>
>
> Hi Carter,
>
> Hope you had a nice vacation.
>
> I've only just gotten around to playing with the ramon -M
> HostP option you
> put in (and I'm currently playing with 2.0.6.beta.41 on this
> particular
> box).
>
> I think you've interpretted "protocol" as "ip protocol"
(Continue reading)

Permalink | Reply | 2 comments |

Gmane