headers
Russell Fulton | 5 Jul 2000 05:19
Picon
Picon
Favicon

Argus 2.0 features

Hi All,
	My main interest in argus is two fold, firstly as a tool for 
detecting threatening or anomolous traffic and secondly as an audit 
tool for forensic investigations.

The current version of argus is an very good for that latter purpose 
but has significant weaknesses in the former role.  In particular it 
does not do a very good job of logging single packets that do not 
conform to the normal tcp state transitions.  Carter has done a great 
deal over the last couple of years to improve argus in this respect 
(Thanks!) but he has now run up against the storage limitations of the 
current audit record.

So what I would like to initiate here is a discussion amongst those of 
us involved in Misuse detection (for want of a better term).

The question is what new features do we want from Argus?

Here one idea of the top of my head:

1/ the ability to log tcp packets that are anomalous, e.g.
   packets with illegal combinations of flags.
2/ the logging of more complete information for such packets (perhaps 
   the best way to do this would be to have a new record class for such 
   packets.
3/ When such packets are detected the should be written out immediately.

There are some cases where it is not straight forward to decide if a 
packet is anomolous or not e.g.  a packet with ACK of FIN set where 
there is no established tcp stream.   It may be a tcp-ping or FIN scan
(Continue reading)

Permalink | Reply | 6 comments |
headers
Russell Fulton | 5 Jul 2000 06:56
Picon
Picon
Favicon

Re: RE: long jumps not supported

Hmmm.... I see Carter's Nortel address now bounces so I'll post this to 
the list...

On Fri, 2 Jun 2000 06:25:11 -0700 Carter Bullard 
<cbullard <at> nortelnetworks.com> wrote:

> Hey Lenny,
>    Argus used the libpcap compiler as a starting
> point for its own compiler many years ago.  Since
> then libpcap has made a few adjustments, and one
> of them has been to remove the "long jump" limitation.

>    Here are the new versions of the files that need
> to be changed to solve your problem, ./include/gencode.h
> and ./common/argus_util.c.  If you don't mind, could you
> make the changes and give it a test?

--------------------------------------------
Hi Carter,

I have just tried this out on my linux 6.1 sysetm but find that when I 
try ra with a large filter it dies with a core dump.  Do you know about 
this or should I investigate?

I also had trouble compiling argus_util.c -- gcc did not like the macro 
defs split over more than one line ???  

Cheers, Russell.
(Continue reading)

Permalink | Reply | 501 comments |
headers
Peter Van Epp | 5 Jul 2000 17:48
Picon
Picon
Favicon

Re: Argus 2.0 features

<snip>
> 
> Lastly an aside -- I am about to buy a new machine which will be used 
> to store and analyse argus records.  Anybody have any experience on 
> what factors limit the performance of ra and friends running on disk 
> based logs?  Oh, BTW money, is tight ;-)
> 
> I thought of a fast and wide scsi disk and 128MB memory 500Mhz 
> processor.  Not taking the standard IDE disks increases the price 
> considerably does anyone have a feeling for what difference this will 
> make to performance?  

	Well I can probably help here :-). At the moment I have my production
Argus server (P2 450, 256 Megs, dual 9 gig fast wide SCSI):

ids /kernel: CPU: Pentium II (quarter-micron) (451.02-MHz 686-class CPU)
ids /kernel: Origin = "GenuineIntel"  Id = 0x652  Stepping=2
ids /kernel: Features=0x183fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,C X8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,<b24>>
ids /kernel: real memory  = 268435456 (262144K bytes)
ids /kernel: ahc0: <Adaptec aic7890/91 Ultra2 SCSI adapter> rev 0x00 int a irq 10 on pci0.14.0
ids /kernel: ahc0: aic7890/91 Wide Channel A, SCSI Id=7, 16/255 SCBs
ids /kernel: fxp0: <Intel EtherExpress Pro 10/100B Ethernet> rev 0x05 int a irq 9 on pci0.18.0
ids /kernel: da1: <WDIGTL WDE9100 1.50> Fixed Direct Access SCSI 2 device
ids /kernel: da1: 40.0MB/s transfers (20.0MHz, offset 15, 16bit)
ids /kernel: da1: 8683MB (17783204 512 byte sectors: 255H 63S/T1106C)
ids /kernel: da0 at ahc0 bus 0 target 0 lun 0
ids /kernel: da0: <WDIGTL WDE9100 1.50> Fixed Direct Access SCSI2 device
ids /kernel: da0: 40.0MB/s transfers (20.0MHz, offset 15, 16bit)
ids /kernel: da0: 8683MB (17783204 512 byte sectors: 255H 63S/T1106C)
(Continue reading)

Permalink | Reply | 3 comments |
headers
Chas DiFatta | 5 Jul 2000 22:51

RE: Argus 2.0 features

It all depends on what your doing.  If you've written
clients that pre-process the data or are pulling LOTS of data
from multiple interfaces, you want to consider a quick machine.
Note, that's only in extreme cases when your talking about >1G
day logs, or LOTS of cpu time pre-processing.  I only have
experience with multiple processors on Solaris and it worked well.
Other OS's should work as well.

It also depends on the need to access  historical data.  If your
writing clients that throw data into a db and you need to go
back to the raw Argus files to extract more information frequently,
get some big disks and consider an archiving device that you don't
have to reload too frequently.

I'd consider cheap hosts for collection devices, then worry about
processing requirements later if you need it.  If your not doing
anything huge, don't worry about it.

	...Chas

> -----Original Message-----
> From: owner-argus <at> lists.andrew.cmu.edu
> [mailto:owner-argus <at> lists.andrew.cmu.edu]On Behalf Of Peter Van Epp
> Sent: Wednesday, July 05, 2000 8:48 AM
> To: argus <at> lists.andrew.cmu.edu
> Subject: Re: Argus 2.0 features
>
>
> <snip>
> >
(Continue reading)

Permalink | Reply | 0 comments |
headers
Russell Fulton | 5 Jul 2000 23:11
Picon
Picon
Favicon

Re: Argus 2.0 features


On Wed, 5 Jul 2000 08:48:28 -0700 (PDT) Peter Van Epp <vanepp <at> sfu.ca> 
wrote:
> 
> 	The UDMA33 IDE looks to be faster (and a lot cheaper) than the SCSI.
> I appear to be able to fill the wire from disk with UDMA 33 IDE and tcpreplay
> as well.

Wow, thanks Peter!  I did not expect to get such a clear cut answer.  
I'll save my money (or buy a bigger disk ;-)

> 
> > Is there anyway I can trade memory for disk reading performance on 
> > these OSes.
> 
> 	Run a filesystem in memory/swap (I do that with /tmp) but it is 
> volitile on a boot. 
> 

I was aware of that possibility, I was thinking along the lines of 
clever read  ahead caches or smart disk controllers -- I have seen some 
theoretical work on things like this ages ago and was wondering if 
there was anything available for your common or garden OSes.

Russell.
Permalink | Reply | 1 comment |
headers
Peter Van Epp | 5 Jul 2000 23:49
Picon
Picon
Favicon

Re: Argus 2.0 features

> 
> Wow, thanks Peter!  I did not expect to get such a clear cut answer.  
> I'll save my money (or buy a bigger disk ;-)

	Yes, the 40 gig UDMA 66 IDE I bought for a bit more than $500 a few
months ago is now down to around $350, its almost full of argus logs. Time
to buy another one.
	As well the original machine (admittably a Supermicro server class
machine) was > $6K, the replacement rackmount Intel was under $2K for a faster
machine (but a couple of years later).

> 
> > 
> > > Is there anyway I can trade memory for disk reading performance on 
> > > these OSes.
> > 
> > 	Run a filesystem in memory/swap (I do that with /tmp) but it is 
> > volitile on a boot. 
> > 
> 
> I was aware of that possibility, I was thinking along the lines of 
> clever read  ahead caches or smart disk controllers -- I have seen some 
> theoretical work on things like this ages ago and was wondering if 
> there was anything available for your common or garden OSes.

	There is FreeBSD support for various of the raid controller, I don't
know what the would do for performace (in theory good things if the access
pattern is correct) but I haven't used any of them.

Peter Van Epp / Operations and Technical Support
(Continue reading)

Permalink | Reply | 0 comments |
headers
Neil Long | 6 Jul 2000 18:21
Picon
Picon

racount

Hi

racount on Solaris is underestimating the total bytes, e.g.

for a 1 hour data file

../racount -nr argus.07.06.11:00.gz net xxx.yyy.zzz
racount: totrcds        101282  rcds      1110  pkts         2779525
bytes    1927726621

but

../racount -nr argus.07.06.11:00.gz
racount: totrcds        101282  rcds    101269  pkts       173961940
bytes    1067804319

i.e. the total bytes for the net is larger than the total for all nets

I increased the format number from 9 to 12 in racount.c just to be sure
there is not a simple format error. The data types are unisigned long.

Just to confirm this I simply added fields to output the total_src_bytes,
total_dst_bytes before the grand total ---

../racount -nr argus.07.06.11:00.gz
racount: totrcds        101282  rcds    101269  pkts       173961940
bytes    2710310111     bytes     2652461504    bytes     1067804319

so that was 5.1G .....
(Continue reading)

Permalink | Reply | 2 comments |
headers
Neil Long | 6 Jul 2000 20:00
Picon
Picon

Re: racount


>Hi
>
>racount on Solaris is underestimating the total bytes, e.g.
>
>for a 1 hour data file
>
>../racount -nr argus.07.06.11:00.gz net xxx.yyy.zzz
>racount: totrcds        101282  rcds      1110  pkts         2779525
>bytes    1927726621
>
>but
>
>../racount -nr argus.07.06.11:00.gz
>racount: totrcds        101282  rcds    101269  pkts       173961940
>bytes    1067804319
>
>i.e. the total bytes for the net is larger than the total for all nets
>
>I increased the format number from 9 to 12 in racount.c just to be sure
>there is not a simple format error. The data types are unisigned long.
>
>Just to confirm this I simply added fields to output the total_src_bytes,
>total_dst_bytes before the grand total ---
>
>../racount -nr argus.07.06.11:00.gz
>racount: totrcds        101282  rcds    101269  pkts       173961940
>bytes    2710310111     bytes     2652461504    bytes     1067804319
>
>so that was 5.1G .....
(Continue reading)

Permalink | Reply | 0 comments |
headers
Peter Van Epp | 7 Jul 2000 04:05
Picon
Picon
Favicon

Re: racount

> i.e. the total bytes for the net is larger than the total for all nets
> 
> I increased the format number from 9 to 12 in racount.c just to be sure
> there is not a simple format error. The data types are unisigned long.
> 
> Just to confirm this I simply added fields to output the total_src_bytes,
> total_dst_bytes before the grand total ---
> 
> ../racount -nr argus.07.06.11:00.gz
> racount: totrcds        101282  rcds    101269  pkts       173961940
> bytes    2710310111     bytes     2652461504    bytes     1067804319
> 
> so that was 5.1G .....

	And if a long is 32 bits as is standard (although perhaps not on one
of the 64 bit machines), you just overflowed, because 32 bits is around 4 gigs.
Which would leave you about a gig and a half after the over flow as you are
seeing. I get around this by feeding the raw data in to perl which happily does 
multiprecision something under the covers and keeps on chugging up to the 
60 to 80 gigabyte level (I ran a test once to see if it had problems while
I was chasing count descrepincies, I think I was up in the 100s of gigs
without problem). That is perhaps one good reason for at least some thought
of a perl interface from the data as Russell suggested (that and indexing
arrays by almost anything ...) because it protects the unwary (such as me).
One of the perl scripts that I posted long ago should do verify this for 
you if you feed it the raw ra data.

> 
> Anyone got a LART I can borrow?
(Continue reading)

Permalink | Reply | 1 comment |
headers
Lenny Zeltser | 10 Jul 2000 00:53

Reading argus-1.8.1 output in Perl

Hi all,

I have argus-1.8.1 in a nice and stable way, where it logs all traffic in
detailed mode. I would like to write some scripts for extracting data from
the program's output file. I know fullra is available as a sample program
for C programmers, but does anyone have any Perl scripts for data extraction
they can share?

Thanks,

-- Lenny
Permalink | Reply | 1 comment |

Gmane