Markku Parviainen | 1 Mar 2012 15:24
Picon

Re: Misc problems with argus-clients v. 3.0.5.34

Hi,

2012/2/29 Carter Bullard <carter <at> qosient.com>:
> I think the problems you're getting from rabins.1 may not be bugs, but
> its definitely not the right results.   When processing time based bins,
> rabins.1 will reject data if its not within its working time range.  If you
> don't tell rabins.1 what time range to use, with the -t option, rabins.1 will
> have to figure out what the time range is.  It does this when reading
> from a file by reading the file twice.  Once to figure out the time ranges,
> then another run to process the data into its preallocated bins.
>
> When you pipe data into rabins.1, it has to guess what the time range
> is going to be, and so there is potential to throw data away if its all over
> the calendar, so to speak.    This doesn't solve your problem, but it just
> explains why I think you're getting unpredicted results.

Actually on my original script where I noticed this problem, I already
was using -t option. It doesn't affect to the results. But thanks for
pointing out that rabins will read the data twice without it.

Related to this problem, rabins also seems to generate new records.
Look at the port 0 (and 3074) below.

This is what the data looks like:
# racluster -m proto dport -r test.arg.gz -w - - udp | rasort -m pkts
-s proto dport pkts -n  | head -5
   udp 161      353877
   udp 137      133430
   udp 3074     108009
   udp 53        89020
(Continue reading)

Carter Bullard | 4 Mar 2012 19:22

Re: argus-clients failing to listen using argus-udp after 3.0.5.11

Hey Terry,
Any additional information on this issue?
Hope all is most excellent,
Carter


On Feb 20, 2012, at 12:24 PM, Terry Burton wrote:

On 20 February 2012 17:00, Carter Bullard <carter <at> qosient.com> wrote:
If you were to use "any" instead of 0.0.0.0, does that work?

Hi,

It turns out that I'm wrong about the client not binding to the UDP
socket. The ra process does in fact bind, however the socket is closed
as soon as data arrives at the port. I will now have to resume my
investigations tomorrow.

For reference, any vs 0.0.0.0 both appear to be equivalent and work as
far as binding is concerned but thanks for the pointer.


Thanks,

Terry


On Feb 20, 2012, at 11:20 AM, Terry Burton <tez <at> terryburton.co.uk> wrote:

Hi,

The netflow-related changes introduced in argus-clients between
3.0.5.10 and 3.0.5.11 appear to have broken argus-udp functionality by
causing them to fail to bind to a local UDP socket when invoked as
follows:

./ra -X -S argus-udp://0.0.0.0:1234


Thanks again,

Terry

Attachment (smime.p7s): application/pkcs7-signature, 4367 bytes
Paul Schmehl | 5 Mar 2012 22:55
Picon

Argus doesn't collect packet data - only session data

I'm setting up a new server, and I've run into a problem with argus.  No 
matter what I do, I can't seem to get it to collect user data.  It works 
fine on the old server.

This is on FreeBSD 8.2.  Argus was built from ports and is version 3.0.4 
without SASL.

Here's the argus.conf file:

ARGUS_DAEMON=yes
ARGUS_INTERFACE="bce1"
ARGUS_OUTPUT_FILE=/var/data/nsm/argus/argus.log
ARGUS_SET_PID=yes
ARGUS_GO_PROMISCUOUS=yes
ARGUS_GENERATE_RESPONSE_TIME_DATA=yes
ARGUS_GENERATE_MAC_DATA=yes
ARGUS_FILTER="ip and not icmp"
ARGUS_CAPTURE_DATA_LEN=800

And here's the process running:

root    94930 18.1  1.1 210416 189484  ??  Rs    8:48PM   0:41.79 
/usr/local/sbin/argus -F /usr/local/etc/argus.conf -F 
/usr/local/etc/argus.conf

No matter what CAPTURE_DATA_LEN I use, this is what I get in the log:

20:53:03.717210  e         udp         10.21.1.49.59178    <-> 
74.200.191.77.domain        2        408   CON
   20:53:03.718211  e         tcp     10.110.143.147.64634    <?> 
205.203.132.65.http         13      10133   CON
   20:53:03.718618  e         tcp     10.160.122.248.55405     -> 
69.116.12.102.28550         3        213   CON
   20:53:03.722132  e         tcp       10.21.21.177.56017     -> 
218.6.12.180.http          3        180   RST
   20:53:03.720056  e         tcp       71.71.229.73.12283     -> 
129.110.94.160.http         11       1633   FIN
   20:53:03.720769  e d       tcp       10.21.17.168.58421     -> 
121.11.151.47.http        351     380622   CON
   20:53:03.721078  e i       tcp       10.40.128.53.55652    <?> 
10.110.47.178.ssh         171     108674   CON

As you can see, no packet data at all.

I've even tried using -U 800 on the commandline, but same results. Googling 
didn't help.

--

-- 
Paul Schmehl (pauls <at> utdallas.edu)

Carter Bullard | 5 Mar 2012 23:29

Re: Argus doesn't collect packet data - only session data

Hey Paul,
You need to specify the "suser" and/or "duser" fields to print using the command line option "-s +suser +duser"
or add these fields to your .rarc file in the RA_FIELD_SPECIFIER variable definition.
Many options changed in the 3.x releases. Check out the man page, if in doubt.

You may find that the newest code, argus-clients-3.0.5.34 works as well:

Hope this helps,
Carter



On Mar 5, 2012, at 4:55 PM, Paul Schmehl wrote:

I'm setting up a new server, and I've run into a problem with argus.  No matter what I do, I can't seem to get it to collect user data.  It works fine on the old server.

This is on FreeBSD 8.2.  Argus was built from ports and is version 3.0.4 without SASL.

Here's the argus.conf file:

ARGUS_DAEMON=yes
ARGUS_INTERFACE="bce1"
ARGUS_OUTPUT_FILE=/var/data/nsm/argus/argus.log
ARGUS_SET_PID=yes
ARGUS_GO_PROMISCUOUS=yes
ARGUS_GENERATE_RESPONSE_TIME_DATA=yes
ARGUS_GENERATE_MAC_DATA=yes
ARGUS_FILTER="ip and not icmp"
ARGUS_CAPTURE_DATA_LEN=800

And here's the process running:

root    94930 18.1  1.1 210416 189484  ??  Rs    8:48PM   0:41.79 /usr/local/sbin/argus -F /usr/local/etc/argus.conf -F /usr/local/etc/argus.conf

No matter what CAPTURE_DATA_LEN I use, this is what I get in the log:

20:53:03.717210  e         udp         10.21.1.49.59178    <-> 74.200.191.77.domain        2        408   CON
 20:53:03.718211  e         tcp     10.110.143.147.64634    <?> 205.203.132.65.http         13      10133   CON
 20:53:03.718618  e         tcp     10.160.122.248.55405     -> 69.116.12.102.28550         3        213   CON
 20:53:03.722132  e         tcp       10.21.21.177.56017     -> 218.6.12.180.http          3        180   RST
 20:53:03.720056  e         tcp       71.71.229.73.12283     -> 129.110.94.160.http         11       1633   FIN
 20:53:03.720769  e d       tcp       10.21.17.168.58421     -> 121.11.151.47.http        351     380622   CON
 20:53:03.721078  e i       tcp       10.40.128.53.55652    <?> 10.110.47.178.ssh         171     108674   CON

As you can see, no packet data at all.

I've even tried using -U 800 on the commandline, but same results. Googling didn't help.

--
Paul Schmehl (pauls <at> utdallas.edu)

Attachment (smime.p7s): application/pkcs7-signature, 4367 bytes
Paul Schmehl | 6 Mar 2012 00:01
Picon

Re: Argus doesn't collect packet data - only session data

Arghh..knew there was something I was missing.....thanks, Carter.

--On March 5, 2012 5:29:17 PM -0500 Carter Bullard <carter <at> qosient.com> 
wrote:

> Hey Paul,
>
> You need to specify the "suser" and/or "duser" fields to print using the
> command line option "-s +suser +duser"
> or add these fields to your .rarc file in the RA_FIELD_SPECIFIER variable
> definition.
> Many options changed in the 3.x releases. Check out the man page, if in
> doubt.
>
>
> You may find that the newest code, argus-clients-3.0.5.34 works as well:
>    http://qosient.com/argus/dev/argus-clients-latest-tar.gz
>
>
> Hope this helps,
> Carter
>
>
>
>
>
>
>
> On Mar 5, 2012, at 4:55 PM, Paul Schmehl wrote:
>
>
> I'm setting up a new server, and I've run into a problem with argus.  No
> matter what I do, I can't seem to get it to collect user data.  It works
> fine on the old server.
>
> This is on FreeBSD 8.2.  Argus was built from ports and is version 3.0.4
> without SASL.
>
> Here's the argus.conf file:
>
> ARGUS_DAEMON=yes
> ARGUS_INTERFACE="bce1"
> ARGUS_OUTPUT_FILE=/var/data/nsm/argus/argus.log
> ARGUS_SET_PID=yes
> ARGUS_GO_PROMISCUOUS=yes
> ARGUS_GENERATE_RESPONSE_TIME_DATA=yes
> ARGUS_GENERATE_MAC_DATA=yes
> ARGUS_FILTER="ip and not icmp"
> ARGUS_CAPTURE_DATA_LEN=800
>
> And here's the process running:
>
> root    94930 18.1  1.1 210416 189484  ??  Rs    8:48PM   0:41.79
> /usr/local/sbin/argus -F /usr/local/etc/argus.conf -F
> /usr/local/etc/argus.conf
>
> No matter what CAPTURE_DATA_LEN I use, this is what I get in the log:
>
> 20:53:03.717210  e         udp         10.21.1.49.59178    <->
> 74.200.191.77.domain        2        408   CON
>   20:53:03.718211  e         tcp     10.110.143.147.64634    <?>
> 205.203.132.65.http         13      10133   CON
>   20:53:03.718618  e         tcp     10.160.122.248.55405     ->
> 69.116.12.102.28550         3        213   CON
>   20:53:03.722132  e         tcp       10.21.21.177.56017     ->
> 218.6.12.180.http          3        180   RST
>   20:53:03.720056  e         tcp       71.71.229.73.12283     ->
> 129.110.94.160.http         11       1633   FIN
>   20:53:03.720769  e d       tcp       10.21.17.168.58421     ->
> 121.11.151.47.http        351     380622   CON
>   20:53:03.721078  e i       tcp       10.40.128.53.55652    <?>
> 10.110.47.178.ssh         171     108674   CON
>
> As you can see, no packet data at all.
>
> I've even tried using -U 800 on the commandline, but same results.
> Googling didn't help.

--

-- 
Paul Schmehl, Senior Infosec Analyst
As if it wasn't already obvious, my opinions
are my own and not those of my employer.
*******************************************
"It is as useless to argue with those who have
renounced the use of reason as to administer
medication to the dead." Thomas Jefferson
"There are some ideas so wrong that only a very
intelligent person could believe in them." George Orwell

Russell Fulton | 6 Mar 2012 23:41
Picon
Picon
Favicon

debian packages for argus 3.0.x

Hi Folks

i am moving to use puppet to manage my sensors and it really want to deal with binary packages (in my case .deb
for Debian).  The debian packages support seems to have stopped at 2.0.6.

If need be I'll do the packaging myself but I'd rather not reinvent the wheel.

Anyone doing anything or have anything to get me started?

Russell
Rafael Barbosa | 7 Mar 2012 17:08
Picon

Time Filters

Hi Carter,

I am having problems with time filters. I can't even make a simple query for a specific day in my dump file:

$> ra -t 2010/10/20 -r anon.argus
$> ra -t 2010/10/20+10h -r anon.argus
$> ra -t 2010/10/20.10 -r anon.argus
$> ra -t ****/**/20 -r anon.argus

All return nothing, while the dump file definitely has some date in the range:

$> ra -r anon.argus -F time.conf | grep 20101020 | wc
120

time.conf has a single line: RA_TIME_FORMAT="%Y%m%d"

Am I missing something or is this a bug?

I am using Ra() Version 3.0.5.26, and I sent the dump file to the ftp.

Best regards,
Rafael Barbosa
Carter Bullard | 7 Mar 2012 17:20

Re: Time Filters

Hey Rafael,
There have been some bug fixes in this area.  I'll be uploading argus-clients-3.0.5.35
later today, so please grab that to see if there are any improvements.

For debugging, I have found that it is useful to run ra.1 with the -D2 option, as this
will print out the time range ra.1 will use for the filter.   The bugs caused a filter of:
   " -t 2010/10/20 "

to look at only the one second range:
   " -t 2010/10/20.00:00:00-00:00:01"

Also timezone modifications in the .rarc were not being applied uniformly to
all timestamps comparisons, so that also should be fixed now.
I'll grab your data files and make sure that the new code at least finds some records.

Carter

On Mar 7, 2012, at 11:08 AM, Rafael Barbosa wrote:

Hi Carter,

I am having problems with time filters. I can't even make a simple query for a specific day in my dump file:

$> ra -t 2010/10/20 -r anon.argus
$> ra -t 2010/10/20+10h -r anon.argus
$> ra -t 2010/10/20.10 -r anon.argus
$> ra -t ****/**/20 -r anon.argus

All return nothing, while the dump file definitely has some date in the range:

$> ra -r anon.argus -F time.conf | grep 20101020 | wc
120

time.conf has a single line: RA_TIME_FORMAT="%Y%m%d"

Am I missing something or is this a bug?

I am using Ra() Version 3.0.5.26, and I sent the dump file to the ftp.

Best regards,
Rafael Barbosa

Attachment (smime.p7s): application/pkcs7-signature, 4367 bytes
Carter Bullard | 8 Mar 2012 01:17

Re: Misc problems with argus-clients v. 3.0.5.34

Hey Markku,
I found the bug, where rabins would generate results inconsistent with racluster. The bug
originated from rabins applying flow correction logic when aggregating with a modified
flow key (incorrect), and racluster not doing the flow correction (the correct behavior).

I have fixed this in argus-clients-3.0.5.35, which I will upload very late tonight.
Thanks for the sample data, I wouldn't not have been able to debug without it.
I believe the new code fixes all the bugs that you reported.
Thanks for all the help !!!!!

Carter



On Mar 1, 2012, at 9:24 AM, Markku Parviainen wrote:

Hi,

2012/2/29 Carter Bullard <carter <at> qosient.com>:
I think the problems you're getting from rabins.1 may not be bugs, but
its definitely not the right results.   When processing time based bins,
rabins.1 will reject data if its not within its working time range.  If you
don't tell rabins.1 what time range to use, with the -t option, rabins.1 will
have to figure out what the time range is.  It does this when reading
from a file by reading the file twice.  Once to figure out the time ranges,
then another run to process the data into its preallocated bins.

When you pipe data into rabins.1, it has to guess what the time range
is going to be, and so there is potential to throw data away if its all over
the calendar, so to speak.    This doesn't solve your problem, but it just
explains why I think you're getting unpredicted results.

Actually on my original script where I noticed this problem, I already
was using -t option. It doesn't affect to the results. But thanks for
pointing out that rabins will read the data twice without it.

Related to this problem, rabins also seems to generate new records.
Look at the port 0 (and 3074) below.

This is what the data looks like:
# racluster -m proto dport -r test.arg.gz -w - - udp | rasort -m pkts
-s proto dport pkts -n  | head -5
  udp 161      353877
  udp 137      133430
  udp 3074     108009
  udp 53        89020
  udp 8612      69988

Rabins produces port 0 and changes the packet counter of port 3074:
# rasort -m stime -r test.arg.gz -w - - udp | time rabins -M hard time
5s -m proto dport -w - | racluster -m proto dport -w - | rasort -m
pkts -s proto dport pkts -n  | head -5
  udp 161      353877
  udp 137      133430
  udp 3074     105026
  udp 0        103673
  udp 53        89020

But the individual search for 3074 produces the correct result:
# rasort -m stime -r test.arg.gz -w - - udp and dst port 3074 | rabins
-M hard time 5s -m proto dport -w - | racluster -m proto dport -w - |
rasort -m pkts -s proto dport pkts -n  | head -5
  udp 3074     108009

# racluster -m srcid -r test.arg.gz -s pkts - udp and dst port 3074
 108009

There are no really many port 0 packets there:
# rasort -m stime -r test.arg.gz -w - - udp and dst port 0 | rabins -M
hard time 5s -m proto dport -w - | racluster -m proto dport -w - |
rasort -m pkts -s proto dport pkts -n  | head -5
  udp 0           107
# racluster -m srcid -r test.arg.gz -s pkts - udp and dst port 0
    107

# ratimerange -r test.arg.gz -u
1327489126.869327 - 1327492741.717686

I tried all commands also with -t 1327489126-1327492741 (with every
rasort, rabins and racluster). It didn't change the results, nor did
it make those commands any faster.

I also noticed this:

# ranonymize -r test.arg.gz -w anon.arg
# racluster -m srcid -r anon.arg -s pkts -  udp and dst port 3074
      1
# racluster -m srcid -r anon.arg -s pkts - udp and dst port 0
    107
# racluster -m srcid -r anon.arg -s pkts
4046512
# racluster -m srcid -r test.arg.gz -s pkts
4046512

There are packets in the anonymized file, but the filter does not find
all of them?

I'll send the original packet file separately for your analysis.

Attachment (smime.p7s): application/pkcs7-signature, 4367 bytes
Carter Bullard | 8 Mar 2012 07:08

new argus and clients on dev server

Gentle people,
I've uploaded bug fix releases for the argus-3.0.6 candidates.  argus-3.0.5.11 and argus-clients-3.0.5.35.
These fix a number of bugs relating to filtering, date filtering and date processing.
Fixes for rabins.1 and racluster.1 are also included, to fix problems with
aggregation when using subsets of the 5-tuple key.  Man pages have been
updated for all core problems (ra* programs in the clients directory).


Please do give these releases a test run.  All bug reports from the mailing list should
now be considered fixed.  If there is a problem that you think is in need of attention,
please give a holler.

And for John, this version of the clients should fix your esoteric rasplit.1 problems.
Fingers are crossed.

Carter

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax



Attachment (smime.p7s): application/pkcs7-signature, 4367 bytes

Gmane