argus 3 - direction determination algorithm

Having recently switched my sensor to argus3, I'm now trying to get my head around
some of the changes.  Today, one of my standard reports on IRC traffic lit up
indicating that dozens of our machines were now talking to servers with sketchy
reputations.

However, looking at the flows in detail with "ra -Zb ..." it seems what was really going on
was a remote SYN-ACK scan from those servers to which our machines were responding
with reset packets. However, argus claimed that our machines were the src IPs for the flow.

IIRC, argus3 changed the argus2 algorithm for determining flow direction. Now, I can
certainly imagine that SYN-ACK might be taken as an indication of a dst IP. If that's
true, I'm probably going to want to flip it back for the situation above.  I'm not
throwing a rock here. I understand direction determination is non-trivial.

Is there a description of the argus3 algorithm for direction determination?

--

-- 
John Gerth      gerth <at> cs.stanford.edu  Gates 378   (650) 725-3273  fax 723-0033

Re: argus 3 - direction determination algorithm

Hey John,
In argus-3.0, we removed most of the direction logic out of argus, and moved
it into the clients, just for this very situation.  argus-2.x would declare that the
originator of a SYN_ACK packet was the destination, regardless of what was
going on, and we put a lot of complex logic in the clients to try to deal with
that when it was a A.SYN_ACK / B.RESET volley.

You should be getting that A is the source, now, and that B is the destination.
If that is not correct, then, could you send some flow records that demonstrate
the problem, so I can debug?  Better yet, if you had a packet capture, then
I can debug the whole flow thread.  I'll look around for a packet trace, maybe
on 

Thanks!!!!

Carter

On Apr 6, 2010, at 3:22 PM, John Gerth wrote:

> Having recently switched my sensor to argus3, I'm now trying to get my head around
> some of the changes.  Today, one of my standard reports on IRC traffic lit up
> indicating that dozens of our machines were now talking to servers with sketchy
> reputations.
> 
> However, looking at the flows in detail with "ra -Zb ..." it seems what was really going on
> was a remote SYN-ACK scan from those servers to which our machines were responding
> with reset packets. However, argus claimed that our machines were the src IPs for the flow.
> 
> IIRC, argus3 changed the argus2 algorithm for determining flow direction. Now, I can
> certainly imagine that SYN-ACK might be taken as an indication of a dst IP. If that's

Re: argus 3 - direction determination algorithm

Hey John,
The reset is probably negating some other logic, so let me take a look, and
check it out.

All the logic is already in the client library, so you don't have to do anything,
at least that is the design ;o)

Carter

On Apr 6, 2010, at 4:11 PM, John Gerth wrote:

> On 4/6/2010 12:58 PM, Carter Bullard wrote:
>> Hey John,
>> In argus-3.0, we removed most of the direction logic out of argus, and moved
>> it into the clients, just for this very situation.  argus-2.x would declare that the
>> originator of a SYN_ACK packet was the destination, regardless of what was
>> going on, and we put a lot of complex logic in the clients to try to deal with
>> that when it was a A.SYN_ACK / B.RESET volley.
>> 
>> You should be getting that A is the source, now, and that B is the destination.
>> If that is not correct, then, could you send some flow records that demonstrate
>> the problem, so I can debug?  Better yet, if you had a packet capture, then
>> I can debug the whole flow thread.  I'll look around for a packet trace, maybe
>> on 
>> 
> Attached is an extract done with:
>      ra -r ar-2010-04-06.12 -w ~/win/synack.argus - host 69.16.172.40 and port 6667
> It shows that most of the S/A are src'ed remotely but whenever there's an R, ra
> shows it the other way.  It's conceivable that this is our fault if somehow the
> span ports are dumping things down incorrectly (I don't have bi-directional timing

Re: argus 3 - direction determination algorithm

Hey John,
I've uploaded a new argus-clients-3.0.3.6.tar.gz to the developers site
that fixes your direction problem.  The clients make all the "override"
decisions on direction, and I added some more tests for the TCP syn/synAck
decisions.  I also added better weights for the destination port number when
there is no Syn or SynAck, and at least one of the ports is IPPORT_RESERVED,
< 1024.

Give this a run, and if you're having problems, holler!!!!

Carter

On Apr 6, 2010, at 4:40 PM, Carter Bullard wrote:

> Hey John,
> The reset is probably negating some other logic, so let me take a look, and
> check it out.
> 
> All the logic is already in the client library, so you don't have to do anything,
> at least that is the design ;o)
> 
> Carter
> 
> On Apr 6, 2010, at 4:11 PM, John Gerth wrote:
> 
>> On 4/6/2010 12:58 PM, Carter Bullard wrote:
>>> Hey John,
>>> In argus-3.0, we removed most of the direction logic out of argus, and moved
>>> it into the clients, just for this very situation.  argus-2.x would declare that the
>>> originator of a SYN_ACK packet was the destination, regardless of what was

Re: argus 3 - direction determination algorithm - bug!!!! Fixed!!!!

Hey John,
The new clients I uploaded Tues had a very bad bug, where for some
TCP flows the timestamps would be removed.  I've fixed this and uploaded
a new argus-clients-3.0.3.6.tar.gz to the server.  Please get that for any
testing that you will be doing.

   http://qosient.com/argus/dev/argus-clients-3.0.3.6.tar.gz

Sorry for any inconvenience!!!

Carter

On Apr 6, 2010, at 8:23 PM, Carter Bullard wrote:

> Hey John,
> I've uploaded a new argus-clients-3.0.3.6.tar.gz to the developers site
> that fixes your direction problem.  The clients make all the "override"
> decisions on direction, and I added some more tests for the TCP syn/synAck
> decisions.  I also added better weights for the destination port number when
> there is no Syn or SynAck, and at least one of the ports is IPPORT_RESERVED,
> < 1024.
> 
> Give this a run, and if you're having problems, holler!!!!
> 
> Carter
> 
> On Apr 6, 2010, at 4:40 PM, Carter Bullard wrote:
> 
>> Hey John,
>> The reset is probably negating some other logic, so let me take a look, and

database indexing support additions

Gentle people,
As promised, I'm finishing up the first of the tools to improve archive
management and performance.  This first set of tools provides very fast time
based searching of argus data,  when the data is stored in a native file system.
We do this through MySQL based time indexes for all the files in the archive,
using the program, rasqltimeindex().  You access the argus data using
the program rasql() with a "-t timerange" command line option.

I've described them briefly here:

   http://qosient.com/archives.shtml

The methods used in rasqltimeindex() are pretty straightforward.  We process
a given file and calculate the byte offsets for each second in the file where an
argus record spans that second.    

   rasqltimeindex -r argus.file -w mysql://user <at> localhost/database

This creates a Seconds, Filename and Probes table, and inserts the derived
data of where in the flle the specific seconds start and stop.  To do the fast
lookups you use rasql()  with a "-t timerangespecifiction" option and a
database to read from:

   rasql -t -8h+1m -r mysql://user <at> localhost/database
   rasql -t 2010/02/12.10:14:30+60s -r mysql://user <at> localhost/database

If you are building native file system archives, you could use rastream() to
write argus data into the archive structure, and in your rastream.sh shell
script, call rasqltimeindex().

Re: database indexing support additions

Re: database indexing support additions

Nice...a couple of obvious questions come to mind

First, does rasqlindex detect an unsorted archive file?
I use a relatively large 60 second reporting interval
and frequently see flows out of order, but in principle
can't flows be reported out-of-order for any interval?

Although I've become a fan of databases, I'm still a
belt-and-suspenders guy for data and like having the
flat files as archive.  Given the simplicity of the
time index table, I was wondering why you chose to
only allow that as DB table since it would be
straightforward to allow a flat file for that too.

/J

Re: database indexing support additions

Hey John,
So, rasqltimeindex() doesn't care if the file is sorted or not, and generates
correct byte offsets for each second in the file, regardless.  When the file is
not time sorted, however, the index may not necessarily provide an improvement
over simple linear search for the time based data.  In your case, however,
the index will still help alot, as your file will not be grossly out of order.

I understand about DB vs file.  I don't necessarily have a apriori requirement
for MySQL to run an argus archive.  So, we can implement this as a file,
rather than in a database table, but a database has a bit better searching
capabilities than a simple sequential file.

One thing we could do is prepend the file with this type of index, to help
in generic file data processing.  Maybe have a minute index rather than
a second index?

Carter

On Apr 9, 2010, at 2:22 PM, John Gerth wrote:

> Nice...a couple of obvious questions come to mind
> 
> First, does rasqlindex detect an unsorted archive file?
> I use a relatively large 60 second reporting interval
> and frequently see flows out of order, but in principle
> can't flows be reported out-of-order for any interval?
> 
> Although I've become a fan of databases, I'm still a
> belt-and-suspenders guy for data and like having the
> flat files as archive.  Given the simplicity of the
> time index table, I was wondering why you chose to
> only allow that as DB table since it would be
> straightforward to allow a flat file for that too.
> 
> /J
> 

Flow loss metric

Hi,

I was using Argus to determine traffic loss and noticed odd results.  To
recreate the scenario a file was transfered trough the loopback and
captured with tshark.  Then the dump was analyzed with argus with the
following results:

> argus -X -r tsharkcapture-lo -w - | ra -n -L 1 -s saddr sport dir daddr dport pkts spkts dpkts loss sloss dloss ploss
>            SrcAddr  Sport   Dir            DstAddr  Dport  TotPkts  SrcPkts  DstPkts       Loss    SrcLoss    DstLoss    pLoss 
>          127.0.0.1.46929     ->          127.0.0.1.80           58       28       30         54         26         28 48.21428

I can't understand that 58 packets were captured and 54 of them were
lost.  There is something wrong?

Attached is the captured file.

Best regards,

Pablo J. Rebollo-Sosa

---
Argus Version 3.0.3.3
Ra Version 3.0.3.6

Serve the file
mini-httpd -d ARGUS/

Loopback capture
tshark -s 0 -i lo -w tsharkcapture-lo

Download file
wget http://127.0.0.1/argus-3.0.3.3.tar.gz