carter | 1 Nov 04:03

Re: Re : Re: Re : Re: ramon question

Hey Real,
Try this:
   racluster -M rmon -m proto sport -w - - tcp or udp | rasort -um pkts -N 10 
-s stime proto sport spkts dpkts sbytes dbytes

Carter
Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: real.melancon <at> videotron.ca
Date: Tue, 31 Oct 2006 19:00:25 
To:Carter Bullard <carter <at> qosient.com>
Cc:argus-info <at> lists.andrew.cmu.edu
Subject: Re : Re: Re : Re: [ARGUS] ramon question

Merci Carter,

I ran a couple of tests and rasplit works like a charm. 

One last last question. To get top tcp/udp ports with ra 2.0.6 I was doing this:

/usr/local/bin/ra -r /var/log/argus/argus.out -w - | /usr/local/bin/ramon -u -M svc | egrep -v
"(arp|llc|decr)" | head -n 10
1162317564  udp 1985             1564     0         96676        0
1162319601  tcp ssh              151      115       13798        20119
(Continue reading)

Denton, Rick | 1 Nov 08:03

RE: racluster, wounded after battling a dragon...

 
the filter="" syntax goes filter="src net 192.168.0.0 mask 255.255.0.0" given the strtol() processing of the network length on the saddr/daddr you may also want to use
the [sd]addr/255.255.255.0 syntax instead temporarily ;)
 
yes.. i muttered about the strtol() in a previous diff..
it is actually more fundamentally incorrect than that..
 
the declaration isn't 'char *endptr = NULL;' it is 'char **endptr = NULL'!!
hence gcc doesn't bleat about the missing & in call to strtol()... :)
 
yes.. endptr is always going to be null going into strtol() and thus will never be set so the subsequent test for failure will always succeed..
 
'endptr' is passed directly into strtol you need &endptr but this will be a char *** if you cast htis back it'll work.. but also be very wrong  ;)
 
should read imho (as per one of my previous diffs):
 
char *endptr = NULL;
..
if ((len = strtol(ptr, &endptr, 10)) == 0)
 
the test for actual validity is also incorrect and doesn't check all cases..
because mptr != endptr doesn't mean mptr contained a real number it means at least the first char did..
now strtol() actually has a real endptr and endptr has memory strtol() will populate it properly the test for validity should go along the lines of:
 
if  ((endptr <= ptr) || (*endptr != '\0')) then error
 
or (endptr == ptr) if you like but it shouldn't be lower than it either.. :)
 
 
and yes, len would make sense :\ i guess i hadn't previously tested the /<netlen> variant enough to notice :/
 
error checking is quite lacking :(.. i started a racluster on some data friday afternoon before leaving.. seemingly i had stupidly fed it a ragator config instead of a racluster config :) it didn't notice and was happy to consume cpu cycles all weekend until i noticed it was going nowhere and killed it :) just need to be more careful..
 
 
it would only check over saddr/daddr mpls because it is looking through a mask list ie a mask to modify the value for the aggregate as per the model and masking on the ports doesn't make too much sense(although to do so would be interesting ;)).. aggregating on ports (well proto and dport tuple at least) appears to work correctly from my testing you will just need to change your filter syntax and optionally your daddr and saddr modifiers..
 

From: argus-info-bounces <at> lists.andrew.cmu.edu [mailto:argus-info-bounces <at> lists.andrew.cmu.edu] On Behalf Of Adrian Bool
Sent: Wednesday, 1 November 2006 03:24 ish
To: argus-info <at> lists.andrew.cmu.edu
Subject: [ARGUS] racluster, wounded after battling a dragon...


Hi,

I have previously used ragator to look a traffic flows from archived pcap data. Over the last couple of days I've been trying to do similar in Argus 3rc33 with racluster.

I'm trying for a filter of the form,

filter="src net 192.168.0.0/16" model="saddr/24 daddr/24 dport"

Whilst trying to get the data I want I have found a couple of bugs in the function argus_client.c::ArgusNewAggregator,

Mask length decoding,

Currently we have,
char * endptr = NULL ;

if ((len = strtol(mptr, endptr, 10)) == 0)
if (*endptr == mptr)
ArgusLog (LOG_ERR, "syntax error: %s %s", mptr, strerror(errno));

... but really need ...

char * endptr = NULL ;

if ((len = strtol(mptr, &endptr, 10)) == 0)
if (endptr == mptr)
ArgusLog (LOG_ERR, "syntax error: %s %s", mptr, strerror(errno));


... and possibly bomb out if we hit the syntax error. strtol stakes a char ** as its second argument - and if that is NULL it will not return any endptr information. So, we pass the address of the NULL ptr, it places the end of processing ptr in our end ptr and then we can just compare normal pointers.

Then, there is currently,

if (len <= 32)
mask.addr_un.ipv4 = (0xFFFFFFFF << (32 - retn->saddrlen));

The mask.addr_un.ipv4 is not a function of the len we just decoded! Using the following seems to be ok,

if (len <= 32)
mask.addr_un.ipv4 = (0xFFFFFFFF << (32 - len));

saddrlen (or daddrlen) seems to be set as appropriate lower down.

I'm actually looking at aggregating on destination port - which worked pretty well in ragator. Lower down this function iterates through members of ArgusMaskDefs but only seem to be picking up saddr, daddr, smpls and dmpls. sport and dport seem to be ignored. Is aggregation on ports not implemented as yet?

Regards,

aid


CS Lee | 1 Nov 09:21
Picon

rasort -N

Carter,

The -N option seems not working for me for rasort and it is not in rasort -h or man page either, I'm on rc33. I use head -10 for top 10 since when running rasort with -N 10 shows nothing.

Thanks.

--
Best Regards,

CS Lee<geekooL[at]gmail.com>

CS Lee | 1 Nov 09:39
Picon

Filtering Portrange

Hey all,

Is it possible to perform flow filtering on certain port range which I think very useful when we need to analyse passive ftp flow in place. Tcpdump on bsd platform offers portrange filter and I can't find any similar in argus or anyone knows the trick that can share. Thanks.

--
Best Regards,

CS Lee<geekooL[at]gmail.com>

carter | 1 Nov 12:21

Re: Filtering Portrange

Try something like:
   ra -r file - dst port \( gt 1024 and lt 2048 \)

Most filter objects that compare numerics can handle 'lt', 'gt', 'eq' like descriptors.

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: "CS Lee" <geek00l <at> gmail.com>
Date: Wed, 1 Nov 2006 16:39:16 
To:argus-info <at> lists.andrew.cmu.edu
Subject: [ARGUS] Filtering Portrange

Hey all,

 Is it possible to perform flow filtering on certain port range which I think very useful when we need to
analyse passive ftp flow in place. Tcpdump on bsd platform offers portrange filter and I can't find any
similar in argus or anyone knows the trick that can share. Thanks.

--

-- 
Best Regards,

CS Lee<geekooL[at]gmail.com> 

carter | 1 Nov 12:32

Re: rasort -N

Hmmmm, all ra* programs support the '-N x' option, and it controls how many input records will be processed. 
You need to read the ra.1 man page to see all the ra* options.

Hmmmmmm,  the right thing to do is:
    rasort -r file -m pkts -w - | ra -N x -w top.x.out

to get the top x pkt generators.  I would use ra() rather that head(), to create the binary output of your top x
list just in case you also wanted to know store it and process it later, like generating an hourly topx, and
then from all the hourly, generate a daily, monthly, etc.

Sorry if I generated any confusion.

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: "CS Lee" <geek00l <at> gmail.com>
Date: Wed, 1 Nov 2006 16:21:53 
To:argus-info <at> lists.andrew.cmu.edu
Subject: [ARGUS] rasort -N

Carter,

 The -N option seems not working for me for rasort and it is not in rasort -h or man page either, I'm on rc33. I use
head -10 for top 10 since when running rasort with -N 10 shows nothing. 

 Thanks.

--

-- 
Best Regards,

CS Lee<geekooL[at]gmail.com> 
CS Lee | 1 Nov 13:41
Picon

Re: rasort -N

Carter,

Apparently -N option is not shown in ra(1) man page either, while it can be used and of course I prefer if -N is available instead of using head(). However I have read that all the argus clients support ra options, but it is not working for rasort(racluster works pretty well with -N), I try to use -N with rasort and it doesn't work so I think piping to ra is what I can do for the moment. Thanks for the clue :)

On 11/1/06, carter <at> qosient.com < carter <at> qosient.com> wrote:
Hmmmm, all ra* programs support the '-N x' option, and it controls how many input records will be processed.  You need to read the ra.1 man page to see all the ra* options.

Hmmmmmm,  the right thing to do is:
    rasort -r file -m pkts -w - | ra -N x -w top.x.out

to get the top x pkt generators.  I would use ra() rather that head(), to create the binary output of your top x list just in case you also wanted to know store it and process it later, like generating an hourly topx, and then from all the hourly, generate a daily, monthly, etc.

Sorry if I generated any confusion.

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax

-----Original Message-----
From: "CS Lee" <geek00l <at> gmail.com>
Date: Wed, 1 Nov 2006 16:21:53
To:argus-info <at> lists.andrew.cmu.edu
Subject: [ARGUS] rasort -N

Carter,

The -N option seems not working for me for rasort and it is not in rasort -h or man page either, I'm on rc33. I use head -10 for top 10 since when running rasort with -N 10 shows nothing.

Thanks.

--
Best Regards,

CS Lee<geekooL[at]gmail.com>



--
Best Regards,

CS Lee<geekooL[at]gmail.com>
CS Lee | 1 Nov 13:52
Picon

Re: Filtering Portrange

Carter,

I were using the similar filter but messing out the grouping, it works now.

Thanks a lot.

On 11/1/06, carter <at> qosient.com <carter <at> qosient.com> wrote:
Try something like:
   ra -r file - dst port \( gt 1024 and lt 2048 \)

Most filter objects that compare numerics can handle 'lt', 'gt', 'eq' like descriptors.

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax

-----Original Message-----
From: "CS Lee" < geek00l <at> gmail.com>
Date: Wed, 1 Nov 2006 16:39:16
To:argus-info <at> lists.andrew.cmu.edu
Subject: [ARGUS] Filtering Portrange

Hey all,

Is it possible to perform flow filtering on certain port range which I think very useful when we need to analyse passive ftp flow in place. Tcpdump on bsd platform offers portrange filter and I can't find any similar in argus or anyone knows the trick that can share. Thanks.

--
Best Regards,

CS Lee<geekooL[at]gmail.com>



--
Best Regards,

CS Lee<geekooL[at]gmail.com>
Carter Bullard | 3 Nov 15:47

Re: rasort -N

Hey CS Lee,
My apologies, yes, I found a stupid bug in rasort, trying to be clever and
reusing the Nflag.  I have added the -N option to the ra.1 man page.
Hopefully I will get a new round of code up today.

So, this brings up a good question again.  The -N flag is very useful, but
somewhat primitive (sometimes that is a good thing).  Should we consider
extensions, like having an input -N and an output -N ?   Should the -N 
option accept a range, to facilitate web based paging viewers?

I think so, but , ....  The biggest hurdle is having a useful syntax for the
command line option.  So is this useful?

   ra -N 10         (first input 10, implied range starting at 1)
   ra -N 10-20   (range, input records 10-20)
   ra -N o10       (output the first 10 records)
   ra -N -10        (output the last 10 records ?)

What do you think?  I am not sold on this idea, except that we probably do
need to support a range expression.

Sorry for any inconvenience,  

Carter


On Nov 1, 2006, at 7:41 AM, CS Lee wrote:

Carter,

Apparently -N option is not shown in ra(1) man page either, while it can be used and of course I prefer if -N is available instead of using head(). However I have read that all the argus clients support ra options, but it is not working for rasort(racluster works pretty well with -N), I try to use -N with rasort and it doesn't work so I think piping to ra is what I can do for the moment. Thanks for the clue :)

On 11/1/06, carter <at> qosient.com < carter <at> qosient.com> wrote:
Hmmmm, all ra* programs support the '-N x' option, and it controls how many input records will be processed.  You need to read the ra.1 man page to see all the ra* options.

Hmmmmmm,  the right thing to do is:
    rasort -r file -m pkts -w - | ra -N x -w top.x.out

to get the top x pkt generators.  I would use ra() rather that head(), to create the binary output of your top x list just in case you also wanted to know store it and process it later, like generating an hourly topx, and then from all the hourly, generate a daily, monthly, etc.

Sorry if I generated any confusion.

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax

-----Original Message-----
From: "CS Lee" <geek00l <at> gmail.com>
Date: Wed, 1 Nov 2006 16:21:53
To:argus-info <at> lists.andrew.cmu.edu
Subject: [ARGUS] rasort -N

Carter,

The -N option seems not working for me for rasort and it is not in rasort -h or man page either, I'm on rc33. I use head -10 for top 10 since when running rasort with -N 10 shows nothing.

Thanks.

--
Best Regards,

CS Lee<geekooL[at]gmail.com>



--
Best Regards,

CS Lee<geekooL[at]gmail.com>




Carter Bullard | 3 Nov 16:05

Re: racluster, wounded after battling a dragon...

Hey Adrian,
Sorry for the delayed response.   Thanks for the email, and yes Rick is correct that
this has been an issue.  I thought I fixed them all but, alas, bugs abound.
'dport' as a aggregation object is supported, but it only makes sense for some
protocols.  Check to see that the record that is being aggregated  is an IPv[4,6]
record, and ICMP, TCP,  UDP/RTP, ESP etc....   There are a lot of protocols that
just don't have a port concept (there are some that have a dport, but no sport, like
ESP).

I'm hoping to have new code up on the server later today (Fri).
Thanks again for the email.

Carter


On Oct 31, 2006, at 11:24 AM, Adrian Bool wrote:


Hi,

I have previously used ragator to look a traffic flows from archived pcap data.  Over the last couple of days I've been trying to do similar in Argus 3rc33 with racluster.

I'm trying for a filter of the form,

filter="src net 192.168.0.0/16" model="saddr/24 daddr/24 dport"

Whilst trying to get the data I want I have found a couple of bugs in the function argus_client.c::ArgusNewAggregator,

Mask length decoding,

Currently we have,
char * endptr = NULL ;

       if ((len = strtol(mptr, endptr, 10)) == 0)
          if (*endptr == mptr)
              ArgusLog (LOG_ERR, "syntax error: %s %s", mptr, strerror(errno));

... but really need ...

char * endptr = NULL ;

       if ((len = strtol(mptr, &endptr, 10)) == 0)
            if (endptr == mptr)
               ArgusLog (LOG_ERR, "syntax error: %s %s", mptr, strerror(errno));


... and possibly bomb out if we hit the syntax error.  strtol stakes a char ** as its second argument - and if that is NULL it will not return any endptr information.  So, we pass the address of the NULL ptr, it places the end of processing ptr in our end ptr and then we can just compare normal pointers.

Then, there is currently,

       if (len <= 32)
                  mask.addr_un.ipv4 = (0xFFFFFFFF << (32 - retn->saddrlen));

The mask.addr_un.ipv4 is not a function of the len we just decoded!  Using the following seems to be ok,

       if (len <= 32)
                  mask.addr_un.ipv4 = (0xFFFFFFFF << (32 - len));

saddrlen (or daddrlen) seems to be set as appropriate lower down.

I'm actually looking at aggregating on destination port  - which worked pretty well in ragator.  Lower down this function iterates through members of ArgusMaskDefs but only seem to be picking up saddr, daddr, smpls and dmpls.  sport and dport seem to be ignored.  Is aggregation on ports not implemented as yet?

Regards,

aid





Gmane