Re: Missing racount ragator rasort in RA 3.0.0.rc.39

Hey OlRoy OlRoy,
argus-3.0.0-clients provides a number of old and new programs.
   ra
   rabins
   racount
   racluster
   ragraph
   ragrep
   rahisto
   ramatrix
   rasplit
   rasort
   rastrip
   ratop

racluster replaces the two programs ragator and ramon.  The man page should
describe how to do what you need. 

Configuration files are in the ./support/Config directory of each 
distribution.
After you take a look, if you're having problems, definately send mail 
to the
list and we'll get you started in the right direction.   ra.conf is 
called rarc.

Carter

OlRoy OlRoy wrote:

> I've compiled Argus and RA 3.0.0.rc.39 on OpenBSD 4.0 which works, 

reading argus files using non-sequential access

Gentle people,
All ra* programs have the ability to read argus files using starting
and ending byte offsets.  If you have a list of offsets, this type
of feature can make processing large argus files very fast/efficient.

The syntax for reading files using offsets has been/is/will be/could be:
   "-r file::ostart:oend"

(or at least that is how I've implemented it in the past)
where ostart is the starting offset, and oend is the ending offset.

This is not a useful feature if you don't know where the record
boundaries are in the file, so I haven't 'exposed' this feature yet, but
I think that it is something that we can begin to work with, or at
least talk about how we could use it.

Anyone interested in this type of feature and would like to
talk about how we could use this?

Carter

RE: reading argus files using non-sequential access

How would byte offset be more valuable than time-offset?  Does it end up
being much faster?
Mark.

> -----Original Message-----
> From: argus-info-bounces <at> lists.andrew.cmu.edu [mailto:argus-info-
> bounces <at> lists.andrew.cmu.edu] On Behalf Of Carter Bullard
> Sent: Wednesday, January 31, 2007 9:36 PM
> To: Argus
> Subject: [ARGUS] reading argus files using non-sequential access
> 
> Gentle people,
> All ra* programs have the ability to read argus files using starting
> and ending byte offsets.  If you have a list of offsets, this type
> of feature can make processing large argus files very fast/efficient.
> 
> The syntax for reading files using offsets has been/is/will be/could be:
>    "-r file::ostart:oend"
> 
> (or at least that is how I've implemented it in the past)
> where ostart is the starting offset, and oend is the ending offset.
> 
> This is not a useful feature if you don't know where the record
> boundaries are in the file, so I haven't 'exposed' this feature yet, but
> I think that it is something that we can begin to work with, or at
> least talk about how we could use it.
> 
> Anyone interested in this type of feature and would like to
> talk about how we could use this?

Re: reading argus files using non-sequential access

Hey Mark,
Byte offset access causes the ra* program to fseek()  ostart bytes before it starts to process the records. 
Without this option, argus reads the entire file to find records to process.

The "-t" option isn't "time offset", its more like "time range".  With "-t" by itself, ra() will
sewuentially read the entire file testing each record's bounds.  You have to do this if the file isn't
sorted at all.  The byte offset option let's you skip (potentially huge) chunks of the file, if you know that
you can, and where to skip to.

So yes, if the file is large (» .5M) the speed ups can be significant!!

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: "Mark Poepping" <poepping <at> cmu.edu>
Date: Wed, 31 Jan 2007 23:56:40 
To:"'Carter Bullard'" <carter <at> qosient.com>
Cc:"'Argus'" <argus-info <at> lists.andrew.cmu.edu>
Subject: RE: [ARGUS] reading argus files using non-sequential access


How would byte offset be more valuable than time-offset?  Does it end up
being much faster?

ArgusEye - new version 0.0.6

Hi,

there is a new version of the GUI available:

 Download: http://www.datenspionage.de/arguseye/arguseye-0.0.6.tgz
Changelog: http://www.datenspionage.de/arguseye/Changelog

Live reading is now possible, and I've started a configuration manager.

In theory it should be able of managing 100's of Argi (as suggested last year by
Carter, when I asked what are possibly good features for a GUI..) In practice
this is a simple implementation with probably lots of horrible bugs, but at least
it's working for me...

I really would like to make this a helpful and stable feature. But this requires
feedback from other users. So if you have some spare time for playing with this,
please let me know what is working and what not...

Cheers, Philipp

Re: reading argus files using non-sequential access

Hey this is great!

this feature would make it possible to handle larger files in ArgusEye.

At moment it reads all available fields of each transaction into memory and
builds a complete view. This is stupid, time-consuming, eating up all the memory
and allows only a limited file size. To improve that I could try to build a list
of offsets and do the partial reading with ra only for the rows that actually
fit into the view, as you suggested some months ago. This seems realizable to
me, but this would only make scrolling of the rows
possible, but inspecting a large file by scrolling millions of rows doesn't seem
very inspiring to me...

So how to use that for sorting and filtering? For the display filter I can
imagine just applying a ra filter expression. That would be a good solution
anyway, because my current attempt to do filtering with an acceptable
performance in Perl is anything but successful.
As the filtering is done while scrolling the view, there would be no information
available
how many transactions are affected by the filter, but thats affordable.

And sorting? I like sorting of transactions in the view, as this is helpful for
finding patterns, and it should be possible for a filtered display as well, but
with partial reading there is no information on a transactions position in a
sorted context. I can imagine reading sort keys from the file when the list of
offsets is generated or to use rasort and generate a new list of offsets, but
both seems very time-consuming to me...

Thanks for revealing that feature, this will help for a better GUI!

Re: reading argus files using non-sequential access

  The files I generate are large (> 2GB) and I can see using the offsets
as a tremendous boon for reanalyzing data, but only if I have a way of
displaying the offset originally. Then when someone wants to look at my
data they can start at the point in the file where the actual data shows
up, shaving seconds off their seek time.

Karl

On Thu, 2007-02-01 at 11:20 +0000, carter <at> qosient.com wrote:
> Hey Mark,
> Byte offset access causes the ra* program to fseek()  ostart bytes before it starts to process the records. 
Without this option, argus reads the entire file to find records to process.
> 
> The "-t" option isn't "time offset", its more like "time range".  With "-t" by itself, ra() will
sewuentially read the entire file testing each record's bounds.  You have to do this if the file isn't
sorted at all.  The byte offset option let's you skip (potentially huge) chunks of the file, if you know that
you can, and where to skip to.
> 
> So yes, if the file is large (» .5M) the speed ups can be significant!!
> 
> Carter
> 
> Carter Bullard
> QoSient LLC
> 150 E. 57th Street Suite 12D
> New York, New York 10022
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax  
> 

Re: reading argus files using non-sequential access

On Thu, Feb 01, 2007 at 09:27:10AM -0600, Karl Tatgenhorst wrote:
> 
>   The files I generate are large (> 2GB) and I can see using the offsets
> as a tremendous boon for reanalyzing data, but only if I have a way of
> displaying the offset originally. Then when someone wants to look at my
> data they can start at the point in the file where the actual data shows
> up, shaving seconds off their seek time.
> 
> 
> Karl
> 

	That initial offset is of course the rub. I suspect the answer to that
is to (optionally?) have argus or more likely ra (since argus is likely 
writing to a socket on a high volume installation and ra is archiving) output 
the offset of a record in the output file every once in a while probably with 
a timestamp so as to create an offset index to the argus file by time (that
still leaves the problem of wanting to filter for some particular field other
than time though). 

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

rc.37 filter broken?

	This filter works as expected on 2.0.6:

ra -r /usr/local/argus/com_argus.archive/2007/02/01/* -nn host 209.190.24.98 and not host 142.58.195.106

	But doesn't seem to on rc.37 (I haven't upgraded yet):

/usr/local/bin/ra3 -r /archive/argus3/com_argus.archive/2007/001/* -n -- host 209.190.24.98 and
not host 142.58.197.106   
   23:59:58.461746             tcp      209.190.24.98.12969     ->     142.58.195.106.80            3        2         1404          120   CON
   00:00:03.212768             tcp      209.190.24.98.12553     ->     142.58.195.106.80            3        4          180         2644   FIN

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

Traffic Profiling