headers
Ken Anderson | 8 Dec 04:07
Favicon

best way to process data from multiple hosts?

Hello, 
I'm new to argus. ragraph and racluster are very cool!

Currently, I have argus running on 7 or 8 machines listening on port
561. I would like to monitor these on a single machine and keep each
server's argus logs separate. 

Is radium capable of opening 1 log for each RADIUM_ARGUS_SERVER? That
would be nice, I think. I could run rasplit -d for each stream, or run
multiple instances of radium, I suppose. 
Is there a better way I've overlooked?

Thanks for any ideas,
Ken

--

-- 
Ken Anderson
Pacific.Net
Permalink | Reply | 2 comments |
headers
Carter Bullard | 8 Dec 16:38

Re: best way to process data from multiple hosts?

Hey Ken,
Try using these two programs, rasplit() and/or rastream().
Both programs split an incoming argus stream, and you can
set it up to split based on the argus data source id, which should
give you your separation.

Focusing on rasplit(),  this is what I do on all my collection sinks.

    rasplit -S radii:561 -M time 5m -w /path/to/archive/\$srcid/%Y/%m/ 
%d/argus.%Y.%m.%d.%H.%M.%S

This will take in a generic stream, and split the data into
an "argus source id" rooted, time based file structure, where
each file represents the data in a given 5 minute time span.
As time goes on, rasplit() creates new files, so you're archive
grows, as needed.

Rasplit() will break records across time boundaries, so that
the stats are preserved correctly when graphing, processing,
analyzing, whatever.

Because rasplit() can connect to up to, what is it, 64 remote sources,
and because they can all be argi, or radii, or a mix of the two, you  
don't
have to worry about the collection tree structure so much.

So I recommend, at first, one radium() to connect to all your sources,
and one rasplit() to connect to your radium().  Where the radium
resides is not important, as long as you have resources.   But having
radium() and rasplit() on the same machine has its advantages, in
(Continue reading)

Permalink | Reply | 1 comment |
headers
Ken A | 8 Dec 17:18
Favicon

Re: best way to process data from multiple hosts?

Carter Bullard wrote:
> Hey Ken,
> Try using these two programs, rasplit() and/or rastream().
> Both programs split an incoming argus stream, and you can
> set it up to split based on the argus data source id, which should
> give you your separation.
> 
> Focusing on rasplit(),  this is what I do on all my collection sinks.
> 
>    rasplit -S radii:561 -M time 5m -w 
> /path/to/archive/\$srcid/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
> 
> This will take in a generic stream, and split the data into
> an "argus source id" rooted, time based file structure, where
> each file represents the data in a given 5 minute time span.
> As time goes on, rasplit() creates new files, so you're archive
> grows, as needed.
> 
> Rasplit() will break records across time boundaries, so that
> the stats are preserved correctly when graphing, processing,
> analyzing, whatever.
> 
> Because rasplit() can connect to up to, what is it, 64 remote sources,
> and because they can all be argi, or radii, or a mix of the two, you don't
> have to worry about the collection tree structure so much.
> 
> So I recommend, at first, one radium() to connect to all your sources,
> and one rasplit() to connect to your radium().  Where the radium
> resides is not important, as long as you have resources.   But having
> radium() and rasplit() on the same machine has its advantages, in
(Continue reading)

Permalink | Reply | 0 comments |
headers
Darren Spruell | 9 Dec 02:07
Picon
Gravatar

Argus daemon (3.0.0, 3.0.1b2) dies after time on OpenBSD 4.x

Hi,

I have an Argus installation on an OpenBSD 4.4 sensor for which I've
noted the daemon running for a while (several days at a time) and then
dying. Typical uptimes seem to be vary between 6 and 10 days. No core
files are found, and I don't seem to have anything in syslog
indicating an error.

I've had the same experience on both argus-3.0.0 and
argus-3.0.1.beta.2. Started as:

/usr/local/sbin/argus -F /etc/argus/argus.conf

Config file:

ARGUS_FLOW_TYPE="Bidirectional"
ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"
ARGUS_DAEMON=yes
ARGUS_MONITOR_ID=quagmire
ARGUS_ACCESS_PORT=561
ARGUS_BIND_IP="x.y.241.103"
ARGUS_INTERFACE=em1
ARGUS_SETUSER_ID=argus
ARGUS_SETGROUP_ID=argus
ARGUS_OUTPUT_FILE=/var/log/argus/argus-quagmire.out
ARGUS_SET_PID=yes
ARGUS_PID_PATH="/var/run/argus"
ARGUS_FLOW_STATUS_INTERVAL=5
ARGUS_MAR_STATUS_INTERVAL=60
ARGUS_DEBUG_LEVEL=0
(Continue reading)

Permalink | Reply | 5 comments |
headers
Peter Van Epp | 9 Dec 03:22
Picon
Picon
Favicon
Gravatar

Re: Argus daemon (3.0.0, 3.0.1b2) dies after time on OpenBSD 4.x

	How much memory is there on the sensor and how fast is the network it
is connected to? I have some recollection that when it runs out of memory 
the daemon dies silently (but memory problems have been thought to be fixed
for a long time too :-)). Doing your ps and watching memory usage over time 
is probably a good bet. If it grows towards the amount of memory available
in the days before a crash that would be one indication. 
	Otherwise you probably need to touch .devel and ./debug in the root
of the argus directory, then run ./configure and recompile with debugging 
support enabled. At that point changing 

ARGUS_DAEMON=yes
  to no (debug won't work in daemon mode I don't believe)
and

ARGUS_DEBUG_LEVEL=0

to 1 or 2 would likely be a good start although you can't run in daemon mode
anymore and need to redirect standard out to a file (which can grow very 
large on a busy link).

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

On Mon, Dec 08, 2008 at 06:07:59PM -0700, Darren Spruell wrote:
> Hi,
> 
> I have an Argus installation on an OpenBSD 4.4 sensor for which I've
> noted the daemon running for a while (several days at a time) and then
> dying. Typical uptimes seem to be vary between 6 and 10 days. No core
> files are found, and I don't seem to have anything in syslog
(Continue reading)

Permalink | Reply | 1 comment |
headers
Carter Bullard | 9 Dec 04:12

Re: Argus daemon (3.0.0, 3.0.1b2) dies after time on OpenBSD 4.x

Hey Darren,
No output in /var/log/messages?  Does seem odd.
Try removing the ./.threads file, and reconfiguring and rebuilding.
    % rm .threads
    % ./configure;make clean;make

And see if that doesn't help.  I've noticed that on OpenBSD that the
threads package is a bit persnickity.  I have worked it a bit on
argus-3.0.1.beta.3 and may have a change or 2 if this helps you.

Carter

On Dec 8, 2008, at 8:07 PM, Darren Spruell wrote:

> Hi,
>
> I have an Argus installation on an OpenBSD 4.4 sensor for which I've
> noted the daemon running for a while (several days at a time) and then
> dying. Typical uptimes seem to be vary between 6 and 10 days. No core
> files are found, and I don't seem to have anything in syslog
> indicating an error.
>
> I've had the same experience on both argus-3.0.0 and
> argus-3.0.1.beta.2. Started as:
>
> /usr/local/sbin/argus -F /etc/argus/argus.conf
>
> Config file:
>
> ARGUS_FLOW_TYPE="Bidirectional"
(Continue reading)

Permalink | Reply | 2 comments |
headers
Darren Spruell | 9 Dec 08:09
Picon
Gravatar

Re: Argus daemon (3.0.0, 3.0.1b2) dies after time on OpenBSD 4.x

On Mon, Dec 8, 2008 at 7:22 PM, Peter Van Epp <vanepp <at> sfu.ca> wrote:
>        How much memory is there on the sensor and how fast is the network it
> is connected to? I have some recollection that when it runs out of memory
> the daemon dies silently (but memory problems have been thought to be fixed
> for a long time too :-)). Doing your ps and watching memory usage over time
> is probably a good bet. If it grows towards the amount of memory available
> in the days before a crash that would be one indication.

2 GB memory:

cpu0: Intel(R) Pentium(R) 4 CPU 3.06GHz ("GenuineIntel" 686-class) 3.07 GHz
real mem  = 2146992128 (2047MB)
avail mem = 2067632128 (1971MB)

There's a few apps running on the host but I have over 1 GB free
whenever I check, like this:

Memory: Real: 279M/530M act/tot  Free: 1479M  Swap: 0K/1028M used/tot

The link speed is 1000Mbps, but the traffic it is monitoring is a
pathetic trickle of < 15Kbps and < 10Kpps. It's only monitoring
traffic that has been sinkholed via DNS redirection.

>        Otherwise you probably need to touch .devel and ./debug in the root
> of the argus directory, then run ./configure and recompile with debugging
> support enabled. At that point changing
[...]

I'll give this a go.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Darren Spruell | 9 Dec 08:36
Picon
Gravatar

Re: Argus daemon (3.0.0, 3.0.1b2) dies after time on OpenBSD 4.x

On Mon, Dec 8, 2008 at 8:12 PM, Carter Bullard <carter <at> qosient.com> wrote:
> Hey Darren,
> No output in /var/log/messages?  Does seem odd.

I thought so too. Verified that logs are devoid of any messages from
'argus' at all. What are the default facility/priority for syslog
output? Maybe I'll have to adjust syslog.conf (Default:
http://www.openbsd.org/cgi-bin/cvsweb/src/etc/syslog.conf?rev=1.17;content-type=text%2Fplain).

> Try removing the ./.threads file, and reconfiguring and rebuilding.
>   % rm .threads
>   % ./configure;make clean;make
>
> And see if that doesn't help.  I've noticed that on OpenBSD that the
> threads package is a bit persnickity.  I have worked it a bit on
> argus-3.0.1.beta.3 and may have a change or 2 if this helps you.

No .threads file by default and none created, no libpthread linking:

$ ldd /usr/local/sbin/argus
/usr/local/sbin/argus:
        Start    End      Type Open Ref GrpRef Name
        1c000000 3c0aa000 exe  1    0   0      /usr/local/sbin/argus
        06f58000 26f68000 rlib 0    1   0      /usr/lib/libpcap.so.6.0
        06178000 26180000 rlib 0    1   0      /usr/lib/libm.so.3.0
        08e0c000 28e42000 rlib 0    1   0      /usr/lib/libc.so.48.0
        08ef9000 08ef9000 rtld 0    1   0      /usr/libexec/ld.so

Any other suggestions? I figure I'll give a shot to Peter's debug
instructions and try to capture more crash details.
(Continue reading)

Permalink | Reply | 1 comment |
headers
carter | 9 Dec 14:48

Re: Argus daemon (3.0.0, 3.0.1b2) dies after time on OpenBSD 4.x

Hey Darren,
Sounds like it could be a specific packet type.
Try writing packets out to a file, and seeing if we
can catch the little bugger. 

Carter
Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: "Darren Spruell" <phatbuckett <at> gmail.com>

Date: Tue, 9 Dec 2008 00:36:31 
To: Argus<argus-info <at> lists.andrew.cmu.edu>
Subject: Re: [ARGUS] Argus daemon (3.0.0,
	3.0.1b2) dies after time on OpenBSD 4.x


On Mon, Dec 8, 2008 at 8:12 PM, Carter Bullard <carter <at> qosient.com> wrote:
> Hey Darren,
> No output in /var/log/messages?  Does seem odd.

I thought so too. Verified that logs are devoid of any messages from
'argus' at all. What are the default facility/priority for syslog
output? Maybe I'll have to adjust syslog.conf (Default:
http://www.openbsd.org/cgi-bin/cvsweb/src/etc/syslog.conf?rev=1.17;content-type=text%2Fplain).

> Try removing the ./.threads file, and reconfiguring and rebuilding.
>   % rm .threads
>   % ./configure;make clean;make
>
(Continue reading)

Permalink | Reply | 0 comments |
headers
Ken A | 24 Dec 18:21
Favicon

ragraph w/large files

Hey all,

I'm writing a php script to webify using ragraph, but I've run into a 
problem. Giving ragraph a lot of data sometimes results in rabins eating 
nearly all system memory (2gb in this case), or ragraph generating a 
very huge but empty, one color graph image. This happens when I tell 
ragraph to read (-R) and process ("sbytes dbytes dport") log directories 
that total in size ~200mb or more.

I've hacked in a 'max-ports-to-graph' command line argument with 2 
additional lines in ragraph around line 918 and 960:
if($i > $max_ports_to_graph) { last; }
This forces ragraph out of it's processing after it's finished a certain 
number of ports and reduces the size of the image generated.

Is this a dumb thing to do, or is there a better way? Typically, when I 
want to look at larger time periods, I am interested in ports that will 
be in the top 100 ports.

Thanks,

Ken

--

-- 
Ken Anderson
http://www.pacific.net/
Permalink | Reply | 4 comments |

Gmane