Added printing country codes to ra* progs

Gentle people,
I've added printing country codes for IPv4 addresses to all the ra*  
programs.
The new printing fields are "sco" and "dco" for src country and dst  
country.
I'll add filtering, aggregation and sorting possibly before release, but
more than  likely, after.

ragraph() will aggregate based on country code, so you will be able to
graph based on this field.

This feature needs a configuration file that is constructed from a  
set of files
that you get from the internet number registries.  I will have a  
working file,
called ./support/Config/delegated-ipv4-latest, that was created on  
2007-10-01.
This file will be in the distribution, and will be installed by the  
package
Makefile.  I have included a shell script that creates new config  
file, by
fetching all the needed files, and consolidating the data into a single
file suitable for the ra* programs.  This script is called
"./support/Config/ragetcountrycodes.sh".   This program uses wget()  
to get
the files from ARIN, the American Registry for Internet Numbers, for all
the global registries and generates the composite file that the ra*  
programs
use.

Re: irritating arugs behaviour

Hey Russell,
I'm thinking about this and trying to come up with a reasonable
solution to your problem.  Understanding that there is not enough
diskspace is tricky unless you generate a filesystem full error, and
I'm wondering if there isn't something a little more elegant.

I'll ponder this until tomorrow, and lets figure out what we need to do.

I was very glad to see that the program "arugs" was irritating you
instead of my program.

Carter

On Sep 24, 2007, at 5:18 PM, Russell Fulton wrote:

> Hi
>
> When a disk write from argus fails (due to diskfull for example) argus
> keeps running but never retries the output file.   I have a daily job
> that clears space in the archive and occasionally the cleared space is
> not enough for the next day's logs so the system runs out of disk late
> at night.
>
> I've just got  back after a few days away to find that the monitor ran
> out of disk on the first evening, of course!  :)
>
> When argus closes an output process due to errors it would be nice  
> if it
> could recheck the file periodically to see if it can proceed again.
>

country code discussion and example

new clients rc.58 on the server

Gentle people,
ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.0.rc.58.tar.gz

is now ready for testing.  This release candidate has a large number of
modifications to fix bugs as well as to finalize the argus-3.0.0 record
enhancement support.  This primarily is the addition of packet size
reporting and country code printing.

Of the bug fixes, one in particular is an unreported bug where radium()
would/could eat as much CPU as it could get.

There are a number of fixes to ragraph().  It should use modern rrdtool
distributions correctly (switching from the \J label to the \l label  
for left
justification of legend items) and I've changed the color selections so
that we repeat colors only after 24 items in the graph.  I also added
support for new rrd_graph options, such as -units si and the -units- 
exponent
option.

If anyone finds any issues, regardless of wether its code, installation
problems, documentation errors, etc....   please send email!!!!!

Thanks for all the support!!

Carter

Re: irritating arugs behaviour

Hey Russell,
So we have mechanisms in argus to periodically check the filesystem.
We're running stat() against the output file only once every second to
see if the file has been renamed and there is a need to recreate it.
We can use the same mechanism to do a statfs() to see if there is
any space to write out to the disk.

So question is, should we have thresholds for the amount of free space
that needs to be there to write out, and free space to start writing
again after stopping?  Seems reasonable.  The system call statfs()
can differentiate free blocks available to non-super user and total
free blocks, so ..... seems that argus is running as root, so we'll
worry about total free blocks?

So how does this sound? When we reach the limit, or fail due to
a number of reasons, we close the output file.  Then every second
when we have records to write out, we'll test the filesystem for
space, and as soon as space becomes free, we'll reopen the file,
or recreate it, and start writing again.

We probably need to tally the number of records that weren't
written, so I'll have to figure out how to do that, possibly writing
the number in the initial MAR that is written to the new file.

Is that what you were thinking?

Carter

Russell Fulton wrote:
>> I was very glad to see that the program "arugs" was irritating you
>> instead of my program.
>>     
> :-P
>
> R
>
> Yes, I agree that deciding what the best way to handle this is not
> obvious and you certainly don't want to keep trying to write every
> record as you will almost certainly end up with corrupt files.
>
> Cheers, Russell
>
>
>   

Re: new clients rc.58 / radium segfaults

Hi Carter,

> Of the bug fixes, one in particular is an unreported bug where radium()
> would/could eat as much CPU as it could get.

Oh yeah, now it consumes no more CPU, but memory  (see below)

With version rc.58 I get a segfault:

 /usr/local/sbin/radium -f /etc/radium.conf

strace output:

...
mmap2(NULL, 397312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0xb7acf000
gettimeofday({1191482968, 288485}, NULL) = 0
read(3, "uivalent\n#\n  \n#RADIUM_FILTER=\"\"\n"..., 4096) = 3402
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 5530 detached

radium in the foreground works, so the debbuging output is not really
helpful...

compiled with .debug:

% ./radium/radium -D8 -f /etc/radium.conf

... very long output ...

radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning
0xbfe4b008
radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning
0xbfeac008
radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning
0xbff0d008
radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning
0xbff6e008
radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning 0x0
radium[15272]: 2007-10-04 10:06:57 ArgusAddHostList((null)) ArgusCalloc
Cannot allocate memory
^^^^^^^^^^^^^^^^^^^^^^^^
radium[15272]: 2007-10-04 10:06:57 ArgusShutDown (3)
radium[15272]: 2007-10-04 10:06:57 ArgusFree (0x81fa080)
radium[15272]: 2007-10-04 10:06:57 ArgusDeleteQueue (0x81fa080) returning
radium[15272]: 2007-10-04 10:06:57 RaParseComplete(0)
radium[15272]: 2007-10-04 10:06:57 ArgusCloseInput(0xb7cfc008) closing
radium[15272]: 2007-10-04 10:06:57 ArgusWriteConnection(0xb7cfc008,
0x80a6689, 6) returning 6
radium[15272]: 2007-10-04 10:06:57 ArgusCloseInput(0xb7cfc008) done
radium[15272]: 2007-10-04 10:06:57 ArgusFree (0xb7cfc008)
radium[15272]: 2007-10-04 10:06:57 ArgusCloseInput(0xb7c88008) closing
radium[15272]: 2007-10-04 10:06:57 ArgusWriteConnection(0xb7c88008,
0x80a6689, 6) returning 6
radium[15272]: 2007-10-04 10:06:57 ArgusCloseInput(0xb7c88008) done
*** glibc detected *** double free or corruption (fasttop): 0x081fa780 ***
Abort

Wolfgang
--

-- 
<wob (at) swobspace de> * http://www.swobspace.de

Re: new clients rc.58 / radium segfaults

On Thu, Oct 04, 2007 at 10:10:16AM +0200, wob wrote:

> Oh yeah, now it consumes no more CPU, but memory  (see below)

Additional info: radium rc.56 works, rc.57 and rc.58 fails.

Wolfgang
--

-- 
<wob (at) swobspace de> * http://www.swobspace.de

Re: new clients rc.58 on the server

Carter Bullard wrote:
> Gentle people,
> ftp://qosient.com/dev/argus-3.0/argus-clients-3.0.0.rc.58.tar.gz

> If anyone finds any issues, regardless of wether its code, installation
> problems, documentation errors, etc....   please send email!!!!!

It would seem that rastrip is still broken

We collect argusdata on two nodes with two NICs each (IN and OUT).
We collect 12 bytes of user DATA.
We merge the data from the two nodes with racluster

Because of policy we need to clean out userdata after a period of time,
but right now we don't trust rastrip since it looks as if it does things
with the data that it shouldn't

Here's a representation of the problem:

#Sort the indata (since it comes from more than one collector)
rasort -r /var/log/argus/2007/10/03-0000 -w 100300.rasort
#Strip userdata from logfiles
rastrip -M -suser -M -duser -r 100300.rasort -w 100300.rasort.strip1
#Strip userdata again (should do nothing)
rastrip -M -suser -M -duser -r 100300.rasort.strip1 -w 100300.rasort.strip2
#Plaintext the stripped files
ra -r 100300.rasort.strip1 >strip1
ra -r 100300.rasort.strip2 >strip2

# I would expect the second run of rastrip to do nothing to the data since the
# only thing it's supposed to do is remove suser and duser and that should
# already be gone. But a simple comparison of the text output shows that
# something funky is happening to some of the flows.

#Compare the two plaintext files. There should be no difference.
diff strip1 strip2
10703c10703
<    23:55:11.467660            tcp     1.2.3.4.63212     ->       2.3.4.5.smtp          1        2           68          198   RST
---
>    23:54:59.085949            tcp     1.2.3.4.63212     ->       2.3.4.5.smtp          1        2           68          198   RST
10760c10760
<    23:55:11.579806    d       tcp      3.4.5.6.55954     ->      2.3.4.6.smtp          1        1           60           60   RST
---
>    23:54:58.833879    d       tcp      3.4.5.6.55954     ->      2.3.4.6.smtp          1        1           60           60   RST
19540c19540
<    23:55:29.428842    d       tcp    4.5.6.7.3014      ->       2.3.4.7.http          7       12          426        15047   FIN
---
>    23:54:57.897868    d       tcp    4.5.6.7.3014      ->       2.3.4.7.http          7       12          426        15047   FIN
23914c23914
<    23:55:37.793078  e         tcp    5.6.7.8.42565     ->      2.3.4.8.http          3        4          192          264   RST
---
>    23:55:09.038700  e         tcp    5.6.7.8.42565     ->      2.3.4.8.http          3        4          192          264   RST

<REST OF DATA STRIPPED>

Only 24 out of 121773 flows are affected, but that's bad enough.

The collector nodes are running argus-3.0.0 64-bit on a RHEL-4ES 2.6.9-55.ELsmp
racluster is argus-3.0.0.rc.44 32bit on a RHEL-3AS 2.4.21-47.ELsmp
rasort,rastrip and ra above are argus-3.0.0.rc.58 32bit on a RHEL-3AS 2.4.21-47.ELsmp

Regards,

/Patrick

Re: new clients rc.58 / radium segfaults

Man, ...,  Wolfgang,
You sure do find the bugs!!!   Now, that is a very good thing, except
that there aren't suppose to be any bugs    :o)

OK, so why would you be failing adding to the host list?  I suspect
the error message is bogus, but we'll see.

How many remote sources are you collecting from?  Are any of
them actually live? reachable?  radium should not fail here, of
course, but I suspect it maybe either a bogus return code, or
a radium.conf problem???!!!

Carter

On Oct 4, 2007, at 4:10 AM, Wolfgang Barth wrote:

> Hi Carter,
>
>> Of the bug fixes, one in particular is an unreported bug where  
>> radium()
>> would/could eat as much CPU as it could get.
>
> Oh yeah, now it consumes no more CPU, but memory  (see below)
>
> With version rc.58 I get a segfault:
>
>  /usr/local/sbin/radium -f /etc/radium.conf
>
> strace output:
>
> ...
> mmap2(NULL, 397312, PROT_READ|PROT_WRITE, MAP_PRIVATE| 
> MAP_ANONYMOUS, -1, 0)
> = 0xb7acf000
> gettimeofday({1191482968, 288485}, NULL) = 0
> read(3, "uivalent\n#\n  \n#RADIUM_FILTER=\"\"\n"..., 4096) = 3402
> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> +++ killed by SIGSEGV +++
> Process 5530 detached
>
> radium in the foreground works, so the debbuging output is not really
> helpful...
>
> compiled with .debug:
>
> % ./radium/radium -D8 -f /etc/radium.conf
>
> ... very long output ...
>
> radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning
> 0xbfe4b008
> radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning
> 0xbfeac008
> radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning
> 0xbff0d008
> radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576) returning
> 0xbff6e008
> radium[15272]: 2007-10-04 10:06:57 ArgusCalloc (1, 394576)  
> returning 0x0
> radium[15272]: 2007-10-04 10:06:57 ArgusAddHostList((null))  
> ArgusCalloc
> Cannot allocate memory
> ^^^^^^^^^^^^^^^^^^^^^^^^
> radium[15272]: 2007-10-04 10:06:57 ArgusShutDown (3)
> radium[15272]: 2007-10-04 10:06:57 ArgusFree (0x81fa080)
> radium[15272]: 2007-10-04 10:06:57 ArgusDeleteQueue (0x81fa080)  
> returning
> radium[15272]: 2007-10-04 10:06:57 RaParseComplete(0)
> radium[15272]: 2007-10-04 10:06:57 ArgusCloseInput(0xb7cfc008) closing
> radium[15272]: 2007-10-04 10:06:57 ArgusWriteConnection(0xb7cfc008,
> 0x80a6689, 6) returning 6
> radium[15272]: 2007-10-04 10:06:57 ArgusCloseInput(0xb7cfc008) done
> radium[15272]: 2007-10-04 10:06:57 ArgusFree (0xb7cfc008)
> radium[15272]: 2007-10-04 10:06:57 ArgusCloseInput(0xb7c88008) closing
> radium[15272]: 2007-10-04 10:06:57 ArgusWriteConnection(0xb7c88008,
> 0x80a6689, 6) returning 6
> radium[15272]: 2007-10-04 10:06:57 ArgusCloseInput(0xb7c88008) done
> *** glibc detected *** double free or corruption (fasttop):  
> 0x081fa780 ***
> Abort
>
> Wolfgang
> -- 
> <wob (at) swobspace de> * http://www.swobspace.de
>

Re: new clients rc.58 / radium segfaults

Hey Wolfgang,
Could you run  your failing radium() using the "-D3" option?  Preferably
the first command line option?  Do you have any RA_ARGUS_SERVER
directives in your .rarc, and ..., can I get a copy of your  
radium.conf file?
You can corrupt anything you're not comfortable sharing, like making
IP addresses bogus, etc......!!!

Carter

On Oct 4, 2007, at 4:17 AM, Wolfgang Barth wrote:

> On Thu, Oct 04, 2007 at 10:10:16AM +0200, wob wrote:
>
>> Oh yeah, now it consumes no more CPU, but memory  (see below)
>
> Additional info: radium rc.56 works, rc.57 and rc.58 fails.
>
> Wolfgang
> -- 
> <wob (at) swobspace de> * http://www.swobspace.de
>