headers
elof2 | 20 May 2013 13:41
Picon
Favicon

Ra - filter syntax error


Carter and I have discussed a very unusual error message; "filter syntax 
error", which may show up if the machine is HEAVILY burdened, like 
swapping a lot and/or receiving tons of interrupts, while feeding a long 
and complex filter expression to a ra* process.

> I haven't seen this problem before, but there maybe a simple fix.
> In order to get around a problem with using flex and bison to compile
> multiple filters at a time, we fork a process to compile a filter, and wait
> for that process to write the filter back to argus through a pipe.  We
> have specific error messages for the pipe creation, but we use a
> generic syntax error message if the forked process doesn't
> return a binary filter back to us in a timeout period, which is now set
> at 200 milliseconds.

I suggest that Carter add a specific error message for this 
particular scenario, logging the message "filter compilation timeout" 
instead of the generic "filter syntax error".

> In the argus-clients file ./common/argus_code.c, in the routine ArgusFilterCompile(),
> on line 389, we set the timer value.  Increase the time to, what, a second ?
> Here is a patch for the 3.0.6.2 code…..
> thoth:common carter$ diff argus_code.c.orig argus_code.c
> 389,390c389,390
> <       wait.tv_sec  = 0;
> <       wait.tv_usec = 200000;
> ---
> >       wait.tv_sec  = 1;
> >       wait.tv_usec = 0;
(Continue reading)

Permalink | Reply | 0 comments |
headers
Craig Merchant | 16 May 2013 08:29
Favicon

Anomaly detection

Carter,

 

Thank you so much for your analysis of the APT1 threats.  Those emails were extremely educational.

 

I wanted to pick your brain about a couple of things related to anomaly detection…

 

We backhaul all remote offices through a central network that Argus can monitor.  Since those remote offices use DHCP, it’s hard for Argus to build a reliable model of “normal” behavior by IP address.   And it can’t see the MAC addresses of flows from those remote offices.  What’s the best approach for anomaly detection in that kind of scenario?  Do you look at the producer/consumer metrics of the whole DHCP subnet and then compare individual flows against that baseline?

 

What kind of anomaly detection strategy do you use for environments where you have farms of different functional roles – web, MTA, database, etc.?  Do you recommend building a behavioral model by individual host or would you compare individual hosts against a baseline for that class of system?

 

Thanks.

 

Craig

Permalink | Reply | 0 comments |
headers
Matt Brown | 15 May 2013 23:55
Picon
Gravatar

raservices crashes when processing

Hello all,

 

I took a day's worth of argus data and, as suggested on http://thread.gmane.org/gmane.network.argus/6228/focus=6234, I analyzed it with rauserdata as follows:

 

#racluster -r * -w day.cache

#rauserdata -r day.cache > /tmp/raservices.conf

 

 

I then inspected /tmp/raservices.conf and it's messy (lots of single lines with arbirary ports, likely sport maybe rpc?), but I figured why not give raservices a shot:

 

#racluster -r * -w - | raservices -f raservices.conf

 

I receive the following error:

raservices[21315]: 16:51:00.727719 RaCreateSrvEntry: format error Service: http

 

 

I straced the process, and I see no occurances of "http" in the output (other than the writev()); the data appears to be read correctly until a blank line is read [read(3, "", 4096)                       = 0]:

 

read(3, "\"  \n\nService: 48956             "..., 4096) = 4096

read(3, "...xxxxxx"  dst ="..., 4096) = 4096

read(3, "xxxx"..., 4096) = 689

read(3, "", 4096)                       = 0

close(3)                                = 0

munmap(0xb766e000, 4096)                = 0

gettimeofday({1368651683, 272271}, NULL) = 0

time(NULL)                              = 1368651683

writev(2, [{"raservices[21523]: 17:01:23.2722"..., 79}, {"\n", 1}], 2raservices[21523]: 17:01:23.272271 RaCreateSrvEntry: format error Service: http

) = 80

 

 

Any idea on why this would be?  Is my data processing flow incorrect?

 

 

Both clients are 3.0.7.8.

 

 

Thanks,

 

Matt

Permalink | Reply | 11 comments |
headers
Matt Brown | 14 May 2013 16:50
Picon
Gravatar

rastream 3.0.7.8, no suser duser

Hello all/Carter,

I am using rastream to write argus data to files.

When I query these files using ra or racluster, suser and duser are
not returning any data.

I'm guessing it isn't being written by rastream which has been started
as follows:

rastream -S 127.0.0.1:561 -B 15s -M time 1h -w
/var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh

How do I use rastream to record N bytes of suser and duser?

Thanks,

Matt
Permalink | Reply | 5 comments |
headers
Dave Edelman | 14 May 2013 13:08
Picon
Favicon

Additional rasqlinsert information

Carter,

It looks like changing the CIDR notation parameter may be masking the
problem rather than fixing it. I found a bunch of MySQL error messages in
the system message log so I restarted rasqlinsert with -D 3. 

For some reason redirection of STDERR prevented the program from running so
I just used cut and paste into a file.

--Dave

[root <at> monolith ~]# /usr/local/bin/rasqlinsert -M time 1d -M cache -S localhost:9603 -w
mysql://argus:argus <at> localhost/argus/matrix_%Y_%m_%d -m srcid matrix proto -s ltime dur srcid saddr
daddr proto bytes -D3 
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.340 RaTopNewProcess(0x718dc010)
returns 0x1d557d0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.398 RaMySQLInit () RaSource (null)
RaArchive (null) RaFormat (null)
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 ArgusInitAddrtoname
(0x7f41718dc010, 0x0, 0x0)
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 ArgusParseInit(0x7f41718dc010, NULL)
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 main: reading files completed
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 Trying 127.0.0.1 port 9603
Expecting Argus records
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 connected
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 ArgusGetServerSocket
(0x7f41715ec010) returning 4
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.413 ArgusReadConnection() read 16 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.414 ArgusParseInit(0x7f41718dc010 0x7f41715ec010
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.414
ArgusWriteConnection(0x715ec010, 0xb9eb56e0, 7) returning 7
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.414 ArgusReadConnection(0x715ec010,
2) returning 1
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.414 ArgusReadStream(0x7f41718dc010) starting
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.674
RaProcessSplitOptions(matrix_2013_05_14, 4096, 0x715ec620): returns 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.685 ArgusCreateSQLSaveTable
(matrix_2013_05_14) returning
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.686 ArgusProcessThisRecord () sql
query SELECT record FROM matrix_2013_05_14 WHERE srcid="69.113.13.218" and saddr="10.1.1.45" and
daddr="10.1.1.60" and proto="tcp"
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.686 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.686 ArgusClientTimeout
ArgusTotalSearches 1 ArgusTotalSQLUpdates 1 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.686 ArgusSQLQuery (UPDATE
matrix_2013_05_14 SET
ltime="1368528902.272",dur="39244.605",bytes="1204124179",record="..." WHERE
srcid="69.113.13.218" and saddr="0.0.0.0" and daddr="0.0.0.0" and proto="ip")
rasqlinsert[12009]: 2013-05-14-10:55:02.687 mysql_real_query error You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near
'"69.113.13.21' at line 1
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.837 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.837 ArgusClientTimeout
ArgusTotalSearches 1 ArgusTotalSQLUpdates 1 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.987 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.987 ArgusClientTimeout
ArgusTotalSearches 1 ArgusTotalSQLUpdates 1 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.120 ArgusProcessThisRecord () sql
query SELECT record FROM matrix_2013_05_14 WHERE srcid="69.113.13.218" and saddr="10.1.1.50" and
daddr="209.177.156.55" and proto="tcp"
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.121 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.121 ArgusClientTimeout
ArgusTotalSearches 2 ArgusTotalSQLUpdates 1 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.159 ArgusProcessThisRecord () sql
query SELECT record FROM matrix_2013_05_14 WHERE srcid="69.113.13.218" and saddr="10.1.1.10" and
daddr="10.1.1.50" and proto="tcp"
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.160 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.160 ArgusClientTimeout
ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.160 ArgusSQLQuery (UPDATE
matrix_2013_05_14 SET
ltime="1368528898.093",dur="39240.430",bytes="1204115615",record="..." WHERE
srcid="69.113.13.218" and saddr="0.0.0.0" and daddr="0.0.0.0" and proto="ip")
rasqlinsert[12009]: 2013-05-14-10:55:03.160 mysql_real_query error You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near
'"69.113.13.218' at line 1
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.160 ArgusSQLQuery (UPDATE
matrix_2013_05_14 SET
ltime="1368528901.670",dur="39244.008",bytes="1204183953",record="..." WHERE
srcid="69.113.13.218" and saddr="0.0.0.0" and daddr="0.0.0.0" and proto="ip")
rasqlinsert[12009]: 2013-05-14-10:55:03.160 mysql_real_query error You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near
'"69.113.13.' at line 1
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.310 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.310 ArgusClientTimeout
ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.460 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.460 ArgusClientTimeout
ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.611 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.611 ArgusClientTimeout
ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.761 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.761 ArgusClientTimeout
ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.911 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.911 ArgusClientTimeout
ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.061 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.061 ArgusClientTimeout
ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.138 ArgusProcessThisRecord () sql
query SELECT record FROM matrix_2013_05_14 WHERE srcid="69.113.13.218" and saddr="10.1.1.45" and
daddr="10.1.1.68" and proto="tcp"
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.139 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.139 ArgusClientTimeout
ArgusTotalSearches 4 ArgusTotalSQLUpdates 3 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.289 ArgusRefreshDisplay (0x718dc010)
screen 0 display 0
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.289 ArgusClientTimeout
ArgusTotalSearches 4 ArgusTotalSQLUpdates 4 written 0 bytes
rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.289 ArgusSQLQuery (UPDATE
matrix_2013_05_14 SET
ltime="1368528899.219",dur="39241.555",bytes="1204115753",record="..." WHERE
srcid="69.113.13.218" and saddr="0.0.0.0" and daddr="0.0.0.0" and proto="ip")
rasqlinsert[12009]: 2013-05-14-10:55:04.289 mysql_real_query error You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near
'"69.113.13.' at line 1
Permalink | Reply | 3 comments |
headers
Dave Edelman | 10 May 2013 05:14
Picon
Favicon

RA_CIDR_ADDRESS_FORMAT="yes" and rasqlinsert -S [radium source] may be a problem

I have a single instance of argus that has been running for years and
creating hourly files of the flow data. On a daily basis, I copy a day's
worth of the flow files to a second system where I run rqasqlinsert in
various flavors to create several different tables.

I finally decided to use radium and rastream to do this the right way (but I
didn't stop the local file creation, just to be safe.) 

The details follow but it looks like there is a toxic interaction between
RA_CIDR_ADDRESS_FORMAT="yes" in ~/.rarc and rasqlinsert using -S from a
radium instance in client version 3.0.7.8 and possibly earlier.

--------- Danger beyond this point are the gory details
--------------------------------

The original method worked very well:
A typical MySQL table queried for all 10.1.1.10 activity gave reasonable
results (I learned not to select the record blob :-) )

mysql> select ltime, dur,  saddr, daddr, proto, bytes from matrix_2013_04_09
where saddr = '10.1.1.10' or daddr = '10.1.1.10';
+-------------------+--------------+------------+-----------------+-------+-
------------+
| ltime             | dur          | saddr      | daddr           | proto |
bytes       |
+-------------------+--------------+------------+-----------------+-------+-
------------+
| 1365551889.364000 | 86287.953000 | 10.1.1.10  | 10.1.1.60       | tcp   |
843282 |
| 1365551948.803000 | 86347.195000 | 10.1.1.10  | 10.1.1.46       | tcp   |
783536 |
| 1365551993.490000 | 86391.336000 | 10.1.1.10  | 10.1.1.45       | tcp   |
16050742461 |
| 1365551986.978000 | 86381.195000 | 10.1.1.10  | 10.1.1.50       | tcp   |
3962458654 |
| 1365551957.703000 | 86340.031000 | 10.1.1.10  | 255.255.255.255 | udp   |
1346514 |
| 1365551992.462000 | 86374.555000 | 10.1.1.10  | 10.1.1.50       | udp   |
1667967 |
| 1365551992.462000 | 86369.734000 | 10.1.1.10  | 10.1.1.50       | icmp  |
1426656 |
| 1365551938.461000 | 86280.000000 | 10.1.1.45  | 10.1.1.10       | arp   |
157312 |
| 1365551964.909000 | 86302.922000 | 10.1.1.50  | 10.1.1.10       | arp   |
152960 |
| 1365551953.810000 | 86274.445000 | 10.1.1.46  | 10.1.1.10       | arp   |
107904 |
| 1365551889.353000 | 86206.047000 | 10.1.1.60  | 10.1.1.10       | arp   |
108544 |
| 1365551868.423000 | 86055.141000 | 10.1.1.10  | 10.1.1.12       | udp   |
30672 |
| 1365551873.421000 | 86055.141000 | 10.1.1.10  | 10.1.1.12       | arp   |
36352 |
| 1365551639.982000 | 85736.523000 | 10.1.1.10  | 10.1.1.127      | udp   |
61560 |
| 1365551920.751000 | 85978.414000 | 10.1.1.10  | 10.1.1.50       | arp   |
49408 |
| 1365551560.220000 | 77317.164000 | 10.1.1.10  | 10.1.1.101      | udp   |
115032 |
| 1365551560.220000 | 77310.164000 | 10.1.1.10  | 10.1.1.101      | arp   |
29696 |
| 1365550944.567000 | 76398.391000 | 10.1.1.10  | 224.0.0.251     | udp   |
3703 |
| 1365529023.282000 | 54396.770000 | 10.1.1.10  | 10.1.1.101      | tcp   |
1032139 |
| 1365475483.795000 |   786.242000 | 10.1.1.101 | 10.1.1.10       | arp   |
896 |
| 1365501012.119000 | 24844.955000 | 10.1.1.10  | 10.1.1.45       | udp   |
2130 |
| 1365501012.119000 | 24844.953000 | 10.1.1.10  | 10.1.1.45       | icmp  |
2718 |
| 1365501007.827000 | 24840.254000 | 10.1.1.10  | 10.1.1.126      | arp   |
384 |
| 1365501012.354000 | 24844.787000 | 10.1.1.10  | 167.206.245.130 | udp   |
5766 |
| 1365501008.507000 | 24840.812000 | 10.1.1.10  | 113.37.91.61    | icmp  |
612 |
| 1365501012.895000 | 24844.477000 | 10.1.1.10  | 113.37.91.61    | tcp   |
37293 |
| 1365502197.518000 | 24840.180000 | 10.1.1.126 | 10.1.1.10       | arp   |
384 |
+-------------------+--------------+------------+-----------------+-------+-
------------+
27 rows in set (0.00 sec)

rasql gave happy results:

rasql -u -r mysql://argus:argus <at> localhost/argus/matrix_2013_04_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"  
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1365551889.364  86287.953      69.113.13.218          10.1.1.60
10.1.1.10    tcp     843282
         1365551948.803  86347.195      69.113.13.218          10.1.1.46
10.1.1.10    tcp     783536
         1365551993.490  86391.336      69.113.13.218          10.1.1.45
10.1.1.10    tcp 160507424*
         1365551986.978  86381.195      69.113.13.218          10.1.1.50
10.1.1.10    tcp 3962458654
         1365551957.703  86340.031      69.113.13.218          10.1.1.10
255.255.255.255    udp    1346514
         1365551992.462  86374.555      69.113.13.218          10.1.1.10
10.1.1.50    udp    1667967
         1365551992.462  86369.734      69.113.13.218          10.1.1.10
10.1.1.50   icmp    1426656
         1365551938.461  86280.000      69.113.13.218          10.1.1.45
10.1.1.10    arp     157312
         1365551964.909  86302.922      69.113.13.218          10.1.1.50
10.1.1.10    arp     152960
         1365551953.810  86274.445      69.113.13.218          10.1.1.46
10.1.1.10    arp     107904
         1365551889.353  86206.047      69.113.13.218          10.1.1.60
10.1.1.10    arp     108544
         1365551868.423  86055.141      69.113.13.218          10.1.1.10
10.1.1.12    udp      30672
         1365551873.421  86055.141      69.113.13.218          10.1.1.10
10.1.1.12    arp      36352
         1365551639.982  85736.523      69.113.13.218          10.1.1.10
10.1.1.127    udp      61560
         1365551920.751  85978.414      69.113.13.218          10.1.1.10
10.1.1.50    arp      49408
         1365551560.220  77317.164      69.113.13.218          10.1.1.10
10.1.1.101    udp     115032
         1365551560.220  77310.164      69.113.13.218          10.1.1.10
10.1.1.101    arp      29696
         1365550944.567  76398.391      69.113.13.218          10.1.1.10
224.0.0.251    udp       3703
         1365529023.282  54396.770      69.113.13.218         10.1.1.101
10.1.1.10    tcp    1032139
         1365475483.795    786.242      69.113.13.218         10.1.1.101
10.1.1.10    arp        896
         1365501012.119  24844.955      69.113.13.218          10.1.1.10
10.1.1.45    udp       2130
         1365501012.119  24844.953      69.113.13.218          10.1.1.45
10.1.1.10   icmp       2718
         1365501007.827  24840.254      69.113.13.218          10.1.1.10
10.1.1.126    arp        384
         1365501012.354  24844.787      69.113.13.218          10.1.1.10
167.206.245.130    udp       5766
         1365501008.507  24840.812      69.113.13.218          10.1.1.10
113.37.91.61   icmp        612
         1365501012.895  24844.477      69.113.13.218          10.1.1.10
113.37.91.61    tcp      37293
         1365502197.518  24840.180      69.113.13.218         10.1.1.126
10.1.1.10    arp        384

And a confirmation from the original flow files checked out well

racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr
daddr proto bytes - host 10.1.1.10
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1365551992.462  86374.555      69.113.13.218          10.1.1.10
10.1.1.50    udp    1667967
         1365551560.220  77317.164      69.113.13.218          10.1.1.10
10.1.1.101    udp     115032
         1365551957.703  86340.031      69.113.13.218          10.1.1.10
255.255.255.255    udp    1346514
         1365501012.354  24844.787      69.113.13.218          10.1.1.10
167.206.245.130    udp       5766
         1365551868.423  86055.141      69.113.13.218          10.1.1.10
10.1.1.12    udp      30672
         1365551639.982  85736.523      69.113.13.218          10.1.1.10
10.1.1.127    udp      61560
         1365501012.119  24844.955      69.113.13.218          10.1.1.10
10.1.1.45    udp       2130
         1365550944.567  76398.391      69.113.13.218          10.1.1.10
224.0.0.251    udp       3703
         1365551986.978  86389.391      69.113.13.218          10.1.1.10
10.1.1.50    tcp 3962486779
         1365529023.282  54396.770      69.113.13.218          10.1.1.10
10.1.1.101    tcp    1032139
         1365501012.895  24844.477      69.113.13.218          10.1.1.10
113.37.91.61    tcp      37293
         1365551993.490  86391.336      69.113.13.218          10.1.1.10
10.1.1.45    tcp 160507424*
         1365551948.803  86347.195      69.113.13.218          10.1.1.10
10.1.1.46    tcp     783536
         1365551889.364  86287.953      69.113.13.218          10.1.1.10
10.1.1.60    tcp     843282
         1365501012.119  24844.953      69.113.13.218          10.1.1.10
10.1.1.45   icmp       2718
         1365551992.462  86369.734      69.113.13.218          10.1.1.10
10.1.1.50   icmp    1426656
         1365501008.507  24840.812      69.113.13.218          10.1.1.10
113.37.91.61   icmp        612
         1365551873.421  86055.141      69.113.13.218          10.1.1.10
10.1.1.12    arp      36352
         1365551920.751  85978.414      69.113.13.218          10.1.1.10
10.1.1.50    arp      49408
         1365551560.220  77310.164      69.113.13.218          10.1.1.10
10.1.1.101    arp      29696
         1365501007.827  24840.254      69.113.13.218          10.1.1.10
10.1.1.126    arp        384
         1365551938.461  86340.000      69.113.13.218          10.1.1.45
10.1.1.10    arp     157440
         1365551953.810  86274.445      69.113.13.218          10.1.1.46
10.1.1.10    arp     107904
         1365551964.909  86302.922      69.113.13.218          10.1.1.50
10.1.1.10    arp     152960
         1365551889.353  86206.047      69.113.13.218          10.1.1.60
10.1.1.10    arp     108544
         1365475483.795    786.242      69.113.13.218         10.1.1.101
10.1.1.10    arp        896
         1365502197.518  24840.180      69.113.13.218         10.1.1.126
10.1.1.10    arp        384

Then I set these three running on the machine with the database
(argus-clients-3.0.7.8)
/usr/local/bin/radium -f /usr/local/argus/SNKradium.conf -d
/usr/local/bin/rastream -S localhost:9603 -f /usr/local/argus/SNKstream.sh
-M time 1h -B 15 -w /data/argus/%Y/%m/%d/argus.%Y.%m.%d.%H -d
/usr/local/bin/rasqlinsert -M time 1d -M cache -S localhost:9603 -w
mysql://argus <at> localhost/argus/matrix_%Y_%m_%d -m srcid matrix proto -s ltime
dur srcid saddr daddr proto bytes -d

# cat /usr/local/argus/SNKradium.conf 
RADIUM_DAEMON=no
RADIUM_CLASSIFIER_FILE=/usr/local/argus/SNKlabel.conf
RADIUM_ACCESS_PORT=9603
RADIUM_ARGUS_SERVER=rodnel-new:561

The SNKstream.sh file doesn't do anything but gzip the file.

Now I get these results:
The MySQL table is a bit unusual but not absolutely awful:

mysql> select ltime,dur,srcid,saddr, daddr, proto, bytes from
matrix_2013_05_09 where saddr = '10.1.1.10' or daddr = '10.1.1.10';
+-------------------+--------------+---------------+------------+-----------
------+-------+------------+
| ltime             | dur          | srcid         | saddr      | daddr
| proto | bytes      |
+-------------------+--------------+---------------+------------+-----------
------+-------+------------+
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.50
| udp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.50
| icmp  | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.50  | 10.1.1.10
| arp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.50
| tcp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  |
255.255.255.255 | udp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.60
| tcp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.45
| tcp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.60  | 10.1.1.10
| arp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.45  | 10.1.1.10
| arp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.50
| arp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.127
| udp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.45
| udp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.45
| icmp  | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.126
| arp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  |
167.206.245.130 | udp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  |
113.37.91.61    | icmp  | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  |
113.37.91.61    | tcp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.126 | 10.1.1.10
| arp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10  | 10.1.1.71
| udp   | 4609938798 |
| 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.71  | 10.1.1.10
| arp   | 4609938798 |
+-------------------+--------------+---------------+------------+-----------
------+-------+------------+
20 rows in set (0.00 sec)

The files created by rastream look correct:

racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr
daddr proto bytes - host 10.1.1.10
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1368143927.352  86340.555      69.113.13.218          10.1.1.10
255.255.255.255    udp    1346514
         1368143957.087  86335.609      69.113.13.218          10.1.1.10
10.1.1.50    udp    1674059
         1368106993.711  44648.406      69.113.13.218          10.1.1.10
167.206.245.130    udp       5766
         1368143651.470  85679.195      69.113.13.218          10.1.1.10
10.1.1.127    udp      61560
         1368106993.205  44648.055      69.113.13.218          10.1.1.10
10.1.1.45    udp       2130
         1368110304.637      0.001      69.113.13.218          10.1.1.10
10.1.1.71    udp        188
         1368106995.036  44648.059      69.113.13.218          10.1.1.10
113.37.91.61    tcp      35811
         1368143944.782  86342.789      69.113.13.218          10.1.1.10
10.1.1.45    tcp 173423905*
         1368143942.334  86354.383      69.113.13.218          10.1.1.10
10.1.1.50    tcp 3945119719
         1368143931.073  86329.680      69.113.13.218          10.1.1.10
10.1.1.60    tcp     429295
         1368106993.205  44648.055      69.113.13.218          10.1.1.10
10.1.1.45   icmp       2718
         1368143957.087  86335.609      69.113.13.218          10.1.1.10
10.1.1.50   icmp    1439680
         1368106987.571  44641.789      69.113.13.218          10.1.1.10
113.37.91.61   icmp        612
         1368143870.011  85965.906      69.113.13.218          10.1.1.10
10.1.1.50    arp      60032
         1368106986.534  44640.969      69.113.13.218          10.1.1.10
10.1.1.126    arp        384
         1368143949.756  86280.000      69.113.13.218          10.1.1.45
10.1.1.10    arp     160128
         1368143942.065  86316.039      69.113.13.218          10.1.1.50
10.1.1.10    arp     158848
         1368143875.921  86209.273      69.113.13.218          10.1.1.60
10.1.1.10    arp     109440
         1368110304.635      0.000      69.113.13.218          10.1.1.71
10.1.1.10    arp        128
         1368108176.467  44641.297      69.113.13.218         10.1.1.126
10.1.1.10    arp        384

Then we come to the output of rasql which for some reason informs me way too
many times that something on my network (10.1.1.0/25) sent a bunch of
traffic to somewhere NB: this is the one place where I see CIDR notation and
that might be is  a clue.

rasql -u -r mysql://argus <at> localhost/argus/matrix_2013_05_09 -M sql=" saddr =
'10.1.1.10' or daddr = '10.1.1.10'"  
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798
         1368144006.774  86377.883      69.113.13.218        10.1.1.0/25
0.0.0.0/4     ip 4609938798

So I copied one day's files from the system running argus to a clean
directory on the MySQL machine and ran the rasqlinsert  incantation that I
always used to use:

rasqlinsert -M time 1d -r * -w
mysql://argus <at> localhost/argus/testMatrix_%Y_%m_%d -m srcid matrix proto -s
ltime dur srcid saddr daddr proto bytes

and got results like the ones I used to get:

rasql -u -r mysql://argus <at> localhost/argus/testMatrix_2013_05_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1368143991.233  86389.844      69.113.13.218          10.1.1.60
10.1.1.10    tcp     429589
         1368143944.782  86342.789      69.113.13.218          10.1.1.45
10.1.1.10    tcp 173086923*
         1368143957.087  86335.609      69.113.13.218          10.1.1.50
10.1.1.10    udp    1674059
         1368143957.087  86335.609      69.113.13.218          10.1.1.10
10.1.1.50   icmp    1439680
         1368143942.065  86316.039      69.113.13.218          10.1.1.50
10.1.1.10    arp     158848
         1368143942.334  86313.445      69.113.13.218          10.1.1.50
10.1.1.10    tcp 3945115585
         1368143987.364  86340.555      69.113.13.218          10.1.1.10
255.255.255.255    udp    1346514
         1368143996.241  86329.594      69.113.13.218          10.1.1.60
10.1.1.10    arp     109568
         1368143949.756  86280.000      69.113.13.218          10.1.1.45
10.1.1.10    arp     160128
         1368143870.011  85965.906      69.113.13.218          10.1.1.10
10.1.1.50    arp      60032
         1368143651.470  85679.195      69.113.13.218          10.1.1.10
10.1.1.127    udp      61560
         1368106993.205  44648.055      69.113.13.218          10.1.1.10
10.1.1.45    udp       2130
         1368106993.205  44648.055      69.113.13.218          10.1.1.45
10.1.1.10   icmp       2718
         1368106986.534  44640.969      69.113.13.218          10.1.1.10
10.1.1.126    arp        384
         1368106993.711  44648.406      69.113.13.218          10.1.1.10
167.206.245.130    udp       5766
         1368106987.571  44641.789      69.113.13.218          10.1.1.10
113.37.91.61   icmp        612
         1368106995.036  44648.059      69.113.13.218          10.1.1.10
113.37.91.61    tcp      35811
         1368108176.467  44641.297      69.113.13.218         10.1.1.126
10.1.1.10    arp        384
         1368110304.635      0.000      69.113.13.218          10.1.1.71
10.1.1.10    arp        128
         1368110304.637      0.001      69.113.13.218          10.1.1.71
10.1.1.10    udp        188

Then I used  the files created by rastream to do the same thing (remember
that these files came from the same radium feed as fed the rasqlinsert that
wasn't so good)
cd /data/argus/2013/05/09
rasqlinsert -M time 1d -r * -w
mysql://argus <at> localhost/argus/test2Matrix_%Y_%m_%d -m srcid matrix proto -s
ltime dur srcid saddr daddr proto bytes

and got the results that I expected:

rasql -u -r mysql://argus <at> localhost/argus/test2Matrix_2013_05_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1368143931.073  86329.680      69.113.13.218          10.1.1.60
10.1.1.10    tcp     429295
         1368143944.782  86342.789      69.113.13.218          10.1.1.45
10.1.1.10    tcp 173423905*
         1368143957.087  86335.609      69.113.13.218          10.1.1.50
10.1.1.10    udp    1674059
         1368143957.087  86335.609      69.113.13.218          10.1.1.10
10.1.1.50   icmp    1439680
         1368143942.065  86316.039      69.113.13.218          10.1.1.50
10.1.1.10    arp     158848
         1368143942.334  86313.445      69.113.13.218          10.1.1.50
10.1.1.10    tcp 3945115585
         1368143927.352  86280.539      69.113.13.218          10.1.1.10
255.255.255.255    udp    1345782
         1368143875.921  86209.273      69.113.13.218          10.1.1.60
10.1.1.10    arp     109440
         1368143949.756  86280.000      69.113.13.218          10.1.1.45
10.1.1.10    arp     160128
         1368143870.011  85965.906      69.113.13.218          10.1.1.10
10.1.1.50    arp      60032
         1368143651.470  85679.195      69.113.13.218          10.1.1.10
10.1.1.127    udp      61560
         1368106993.205  44648.055      69.113.13.218          10.1.1.10
10.1.1.45    udp       2130
         1368106993.205  44648.055      69.113.13.218          10.1.1.45
10.1.1.10   icmp       2718
         1368106986.534  44640.969      69.113.13.218          10.1.1.10
10.1.1.126    arp        384
         1368106993.711  44648.406      69.113.13.218          10.1.1.10
167.206.245.130    udp       5766
         1368106987.571  44641.789      69.113.13.218          10.1.1.10
113.37.91.61   icmp        612
         1368106995.036  44648.059      69.113.13.218          10.1.1.10
113.37.91.61    tcp      35811
         1368108176.467  44641.297      69.113.13.218         10.1.1.126
10.1.1.10    arp        384
         1368110304.635      0.000      69.113.13.218          10.1.1.71
10.1.1.10    arp        128
         1368110304.637      0.001      69.113.13.218          10.1.1.71
10.1.1.10    udp        188

Just in case the -M cache is making a difference, I included it in a test
and it didn't break anything:

rasqlinsert -M time 1d -r * -M cache  -w
mysql://argus <at> localhost/argus/test3Matrix_%Y_%m_%d -m srcid matrix proto -s
ltime dur srcid saddr daddr proto bytes
rasql -u -r mysql://argus <at> localhost/argus/test3Matrix_2013_05_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'"
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1368143991.233  86389.844      69.113.13.218          10.1.1.60
10.1.1.10    tcp     429589
         1368143944.782  86342.789      69.113.13.218          10.1.1.45
10.1.1.10    tcp 173086923*
         1368143957.087  86335.609      69.113.13.218          10.1.1.50
10.1.1.10    udp    1674059
         1368143957.087  86335.609      69.113.13.218          10.1.1.10
10.1.1.50   icmp    1439680
         1368143942.065  86316.039      69.113.13.218          10.1.1.50
10.1.1.10    arp     158848
         1368143942.334  86313.445      69.113.13.218          10.1.1.50
10.1.1.10    tcp 3945115585
         1368143987.364  86340.555      69.113.13.218          10.1.1.10
255.255.255.255    udp    1346514
         1368143996.241  86329.594      69.113.13.218          10.1.1.60
10.1.1.10    arp     109568
         1368143949.756  86280.000      69.113.13.218          10.1.1.45
10.1.1.10    arp     160128
         1368143870.011  85965.906      69.113.13.218          10.1.1.10
10.1.1.50    arp      60032
         1368143651.470  85679.195      69.113.13.218          10.1.1.10
10.1.1.127    udp      61560
         1368106993.205  44648.055      69.113.13.218          10.1.1.10
10.1.1.45    udp       2130
         1368106993.205  44648.055      69.113.13.218          10.1.1.45
10.1.1.10   icmp       2718
         1368106986.534  44640.969      69.113.13.218          10.1.1.10
10.1.1.126    arp        384
         1368106993.711  44648.406      69.113.13.218          10.1.1.10
167.206.245.130    udp       5766
         1368106987.571  44641.789      69.113.13.218          10.1.1.10
113.37.91.61   icmp        612
         1368106995.036  44648.059      69.113.13.218          10.1.1.10
113.37.91.61    tcp      35811
         1368108176.467  44641.297      69.113.13.218         10.1.1.126
10.1.1.10    arp        384
         1368110304.635      0.000      69.113.13.218          10.1.1.71
10.1.1.10    arp        128
         1368110304.637      0.001      69.113.13.218          10.1.1.71
10.1.1.10    udp        188

I kill CIDR notation in my ~/.rarc file to see what happens (I dropped the
current table and restarted the clients) and it is looking much better

rasql -u -r mysql://argus:argus <at> localhost/argus/matrix_2013_05_10 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'" 
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1368153616.837     60.160      69.113.13.218          10.1.1.60
10.1.1.10    tcp        588
         1368154260.248    700.975      69.113.13.218          10.1.1.50
10.1.1.10    udp      13899
         1368154260.248    700.975      69.113.13.218          10.1.1.10
10.1.1.50   icmp      10688
         1368153868.734    307.102      69.113.13.218          10.1.1.50
10.1.1.10    tcp     425755
         1368154283.605    721.920      69.113.13.218          10.1.1.60
10.1.1.10    arp       1408
         1368154247.544    675.886      69.113.13.218          10.1.1.10
255.255.255.255    udp      12078
         1368153964.784    360.043      69.113.13.218          10.1.1.45
10.1.1.10    tcp      10108
         1368154172.306    566.982      69.113.13.218          10.1.1.50
10.1.1.10    arp       1280
         1368154209.756    600.000      69.113.13.218          10.1.1.45
10.1.1.10    arp       1408
         1368153742.552      0.000      69.113.13.218          10.1.1.10
10.1.1.127    udp        513
         1368153908.043     67.891      69.113.13.218          10.1.1.10
10.1.1.50    arp        256

The fix is not retroactive, NB: the testMatrix, test2Matrix, and test3Matrix
tables were all generated by rasqlinsert with the .rarc containing
RA_CIDR_ADDRESS_FORMAT="yes" and they were fine so it looks like an
interaction between CIDR notation and rasqlinsert -S from a radium source

rasql -u -r mysql://argus:argus <at> localhost/argus/matrix_2013_05_09 -M sql="
saddr = '10.1.1.10' or daddr = '10.1.1.10'" 
               LastTime        Dur              SrcId            SrcAddr
DstAddr  Proto   TotBytes 
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
         1368144006.774  86377.883      69.113.13.218           10.1.1.0
0.0.0.0     ip 4609938798
Permalink | Reply | 3 comments |
headers
Matt Brown | 9 May 2013 20:39
Picon
Gravatar

Quick questions about rasort output

Hello,

How do I stop the concatenation of daddr and dport (for instance)?

How do I stop the resolution of service names when using dport (for instance)?

Thanks,

Matt
Permalink | Reply | 1 comment |
headers
Matt Brown | 9 May 2013 16:34
Picon
Gravatar

Re: rastream and %T in -w

Re-looped the list.

I had grabbed the *-latest.tar.gz off the site the other day.

Upgrading to 3.0.7.8 does not solve the problem with the file naming scheme.

I hadn't previously answered, but the times are the local TZ in the records and are not UTC in the filenames; simply always 01:00:00 regardless of when the file is created/process is started.


strace shows nothing useful at all.  I did verify that the md5 of /etc/localtime and the correct timezone file are the same.
...

Ah hah... `-M 1d`, so resolution of the time format strings is to the day, not the given time format string.  I do find this a little odd.  Carter, what do you think?

"Problem" resolved.


Thanks,

Matt




On May 8, 2013, at 11:14 PM, Carter Bullard <carter <at> qosient.com> wrote:

Hey Matt,
Are the times off by a fixed amount of time each time, like the offset to GMT ?
We did have a bug in time processing that should be fixed in argus-clients-3.0.7.8
which is on the server at:

If you grab that and the problem goes away, I would think that it was the tm_gmoffset
bug that is biting.

Carter 

On May 8, 2013, at 9:21 PM, Matt Brown <matthewbrown <at> gmail.com> wrote:

Sorry Carter.  I did use `-B 15s`

I will attempt to dump something with `strace` with harder evidence of the problem.

%Y-%m-%d do equate to the strftime() values, but %T, %H, %M, and %S do not.

Do you have any suggestions for further troubleshoot the problem?




On Wed, May 8, 2013 at 7:37 PM, Carter Bullard <carter <at> qosient.com> wrote:
Hey Matt,
Don't forget you need a "-B secs" option for rastream() to work.
If you are reading argus data, set it to 2x your flow status interval.
I haven't used a "-", but it shouldn't make a difference.  I use '.'
Carter

On May 8, 2013, at 7:25 PM, Matt Brown <matthewbrown <at> gmail.com> wrote:

Hello Carter,

Thanks for writing back quickly.


If I start rastream as follows:
rastream -S 127.0.0.1:561 -M time 1d -w /var/opt/argus//%Y-%m-%d/argus_%T

the generated file is named:
/var/opt/argus//%Y-%m-%d/argus_01:00:00


As is the case with %H %M and %S == 01 00 and 00

I pulled these variables from the man page of strftime() http://linux.die.net/man/3/strftime


I've finally got around to implementing argus in a real way to complement the project flow-inspector, which presents flow data via a web interface using a few d3.js visualizations.  The commit that extends support for the data source of an "rasqlinserted" argus DB can be reviewed: https://github.com/constcast/flow-inspector/commit/e800598c3481c8ec6a44b103d98906668f612546.  It would be great to have an ra* client that would BLPOP() data into a redis queue.  A python script takes in a few IPFIX information elements about the flows and inserts them into a backend DB (mysql, oracle, or mongo).  I've been going back and forth with Lothar Braun who has been quite responsive.


Thanks again for your help,

Matt Brown



On Wed, May 8, 2013 at 3:49 PM, Carter Bullard <carter <at> qosient.com> wrote:
Hey Matt,
Not sure, from your description, what is up.
So, your calling rastream() against a file or a stream?
Parameters ?

Since rastream() gets its time from the records, are those correct?

Carter



On May 8, 2013, at 1:52 PM, Matt Brown <matthewbrown <at> gmail.com> wrote:

> Hello all,
>
> With 3.0.6.2 I am seeing something odd with rastream's -w.
>
> It appears to not deal with %T %H %M or %S properly, not returning
> now(), but starting with 01:00:00 and 01 00 00 respectively.
>
> Why is this?
>
>
> Unfortunately gmane's search function seems to not be functioning.
>
>
> Any assistance is appreciated.
>
>
> Thanks,
>
> Matt Brown
>





Permalink | Reply | 3 comments |
headers
Matt Brown | 8 May 2013 19:52
Picon
Gravatar

rastream and %T in -w

Hello all,

With 3.0.6.2 I am seeing something odd with rastream's -w.

It appears to not deal with %T %H %M or %S properly, not returning
now(), but starting with 01:00:00 and 01 00 00 respectively.

Why is this?

Unfortunately gmane's search function seems to not be functioning.

Any assistance is appreciated.

Thanks,

Matt Brown
Permalink | Reply | 3 comments |
headers
Carter Bullard | 8 May 2013 05:31

Re: normalized appbyte ratio for producer/consumer relationship

Hey John,
Here is a run to try out the abr metric for basic producer/consumer anomaly detection.
Picked a bunch of hosts that are involved in a service that runs on a specific port.
There is a data transfer function, and there is a command and control function.
The abr metric picks out the producer and consumers for the data transfer, and it points to those that are involved in command and control, here you go.

thoth:backups root# racluster -M rmon -m saddr -r monthly.data -w - | \
                    rasort -m abr -s stime dur:16 proto saddr spkts:12 dpkts:12 abr
                 StartTime              Dur  Proto            SrcAddr      SrcPkts      DstPkts    ABRatio 
2013/02/05.14:03:48.304265   2022912.500000    tcp       192.168.1.31    118813516     85735688   0.999937
2013/02/06.16:19:37.099642   1895160.500000    tcp       192.168.2.75         3621         1899   0.997145
2013/02/07.12:06:09.973606     27554.181641    tcp       192.168.2.34          732          650   0.915472
2013/02/27.11:40:47.093087         4.957941    tcp       192.168.4.35           13           12   0.551477
2013/02/18.14:17:01.287529    758586.750000    tcp      192.168.7.166           23           18   0.422487
2013/02/06.10:10:26.473381        30.864573    tcp     192.168.12.149            7            5   0.295455
2013/02/13.16:45:55.793151        11.732343    tcp      192.168.7.155           15           13   0.250169
2013/02/06.09:31:55.244962   1228359.500000    tcp       192.168.2.54           32           35   0.241102
2013/02/08.10:09:14.794703   1727145.250000    tcp      192.168.0.125           21           18   0.213155
2013/02/06.13:17:45.550931   1819270.875000    tcp      192.168.1.138         1227         1231   0.135222
2013/02/15.11:28:36.104691        50.191151    tcp       192.168.2.71           75           59   0.100396
2013/02/07.14:00:35.770555        83.157898    tcp       192.168.0.70          894          890   0.057422
2013/02/12.14:19:09.720183         1.038289    tcp      192.168.1.125            7            8   0.029588
2013/02/21.03:07:04.628043        32.868526    tcp       192.168.0.72            7            5   0.023810
2013/02/11.08:39:44.024865   1478973.500000    tcp       192.168.2.45          268          157  -0.015058
2013/02/19.14:25:03.376258        28.092548    tcp      192.168.7.153            7            5  -0.043860
2013/02/06.14:05:03.101059        65.313805    tcp       192.168.2.58            6            7  -0.055469
2013/02/06.13:17:45.550931   1772498.375000    tcp       192.168.2.57           13           16  -0.164201
2013/02/06.12:20:05.543201   1913705.250000    tcp      192.168.2.138          256          317  -0.173220
2013/02/20.12:43:35.083127    424459.562500    tcp      192.168.2.102         1147         1160  -0.343431
2013/02/20.13:46:25.772704       940.071899    tcp       192.168.12.2           16           17  -0.392946
2013/02/07.12:55:49.369160   1822594.375000    tcp      192.168.1.110          296          400  -0.456629
2013/02/06.12:51:40.056876   1901223.250000    tcp      192.168.1.123          165          222  -0.601214
2013/02/06.09:37:40.607721   1900794.750000    tcp      192.168.1.111          681          995  -0.612489
2013/02/05.14:44:23.517669   1818851.000000    tcp      192.168.1.117           79          102  -0.627868
2013/02/05.14:05:30.070586   1889467.750000    tcp      192.168.1.127          143          175  -0.669750
2013/02/06.09:31:55.244962    697157.125000    tcp      192.168.1.115           51           56  -0.683094
2013/02/13.16:07:23.915251   1291093.750000    tcp      192.168.1.130           61           75  -0.694824
2013/02/07.13:33:41.019318   1557935.000000    tcp      192.168.1.113          100          133  -0.706807
2013/02/06.12:06:43.001669   1721456.875000    tcp      192.168.1.112          618         1010  -0.740505
2013/02/23.17:46:07.768913         0.588042    tcp       192.168.9.84           12           15  -0.758408
2013/02/06.11:31:22.894660   1906436.625000    tcp      192.168.1.122           76          101  -0.762083
2013/02/05.15:07:18.072928   1977283.125000    tcp      192.168.1.114           90          120  -0.781786
2013/02/08.09:07:55.007936   1140006.125000    tcp      192.168.1.118          112          136  -0.787711
2013/02/05.14:04:02.548134   2022898.250000    tcp       192.168.2.47       327011       243367  -0.795821
2013/02/06.14:30:10.891759   1905899.875000    tcp      192.168.1.116          343          588  -0.934336
2013/02/07.12:06:09.973606   1807785.625000    tcp      192.168.1.121         2579         4398  -0.990442
2013/02/05.14:03:48.304265   1465134.125000    tcp       192.168.2.29     85407678    118569168  -0.999999                                                                                                                                     


So we take a some records, in this case a complete month's worth of traffic involved in a specific application,
involving a specific subnet.  We want to know what hosts are producers and consumers for this app.
We need to get the bi-directional flow data into a single object statistic, so we'll aggregate the data for RMON
data processing (one object, in and out stats), and merge for the " saddr ", then just rasort() on the abr field.

We get a list from Producers to Consumers, and the guys in the middle where the abr approaches 0, and we have
balanced communications, we see the complete spectrum of data push agents (producers) where ( ABRation > 0.75 )
on top, and we have the pure data sinks, where the ( ABRatio < -0.75 ), and we've got maybe command
and control in the ( -0.5 < ABRatio < 0.5 ) range ?  Probably need to add a threshold for the amount of
data sent and received, to weed out the announcers in the command and control network...

I'd go for that set of rules for this specific application, in this observation domain…..

Carter



On May 6, 2013, at 6:34 PM, John Gerth <gerth <at> graphics.stanford.edu> wrote:

Yes, well an abr of 0.0, even without using -0.0 can have multiple meanings
since you will get 0 as long as s=d even if they are large.  The use of -0.0
is perhaps a bit cute, and, as you point out, since IEEE 754 requires 0.0 == -0.0
in all relational tests, one has to use signbit() to disambiguate in C.

Even so, I think your example shows that abr has good potential as a metric.

John Gerth      gerth <at> graphics.stanford.edu  Gates 378

On 5/6/2013 2:09 PM, Carter Bullard wrote:
Hey John,
OK, so there is a problem, in that IEEE floating point has ( -0.0 == 0.0 )
by definition.  So, I fixed it in my compiler, but others are going to have
issues when needing to discriminate between 0.0 and -0.0.

So, with the new argus-clients, you can filter for any value for abr, using
any of the ra* programs.  Still have to work on graphing abr, but I'll get
to that later tonight.

Here is the abr behavior for every DNS request here at QoSient World HQ
for 2013, that weren't ServFail errors:

thoth:tmp carter$ rahisto -H abr 10:-1.0-1.0 -r argus*domain* -s mean stddev - src pkts 1 and dst pkts 1 and not abr 0.0
N = 1009841  mean = -0.739084  stddev =  0.102829  max = -0.162791  min = -0.909605
          median = -0.749129     95% = -0.653846
            mode = -0.782609
Class      Interval         Freq    Rel.Freq     Cum.Freq          Mean     StdDev
    1   -1.000000e+00     225379    22.3183%     22.3183%     -0.815238   0.000650
    2   -8.000000e-01     738148    73.0955%     95.4137%     -0.740887   0.043837
    3   -6.000000e-01      10553     1.0450%     96.4588%     -0.534672   0.048717
    4   -4.000000e-01      35511     3.5165%     99.9752%     -0.283067   0.021374
    5   -2.000000e-01        250     0.0248%    100.0000%     -0.162791   0.000051
    6    0.000000e+00          0     0.0000%    100.0000%    
    7    2.000000e-01          0     0.0000%    100.0000%    
    8    4.000000e-01          0     0.0000%    100.0000%    
    9    6.000000e-01          0     0.0000%    100.0000%    
   10    8.000000e-01          0     0.0000%    100.0000%


So, anything positive would be a behavioral anomaly, from the perspective of this host.

  ra -r file.from.the.host - udp and port domain and src pkts 1 and dst pkts 1 and abr gt 0.0

and this would pick out candidates for DNS server availability errors:

  ra -r file.from.the.host - udp and port domain and src pkts 1 and dst pkts 1 and abr eq 0.0

Of course, you can do an analysis of every service, and get a rather interesting set of
" what is normal " behaviors, using this simple type of metric.  For those services where
the abr is always positive or always negative, seeing a shift to the other side, can
indicate events that should be of interest.

Hope this is helpful,

Carter


On May 6, 2013, at 2:41 PM, Carter Bullard <carter <at> qosient.com <mailto:carter <at> qosient.com>> wrote:

Hey John,
If no appbytes, currently we return -0.0, but the library knows if there are
appbytes or not, so we can return nada, when printing out the values.
Right now, when using xml format, you won't get a value.

Having problems getting my compiler to tell the difference between 0.0 and -0.0,
but should hopefully have this working by this afternoon.

Carter


On May 6, 2013, at 2:37 PM, John Gerth <gerth <at> graphics.stanford.edu <mailto:gerth <at> graphics.stanford.edu>> wrote:

Nice example. I'm looking forward to using this.

As your example shows, this metric is available for any existing argus
files that were created containing appbyte values. I'm assuming that if
the sensor wasn't configured to capture those, 'abr' is not available.


--
John Gerth      gerth <at> graphics.stanford.edu <mailto:gerth <at> graphics.stanford.edu>  Gates 378   (650) 725-3273 fax 725-6949

On 5/6/2013 11:19 AM, Carter Bullard wrote:
Hey John,
OK, so I've implemented " abr " as a new metric, using our normalized equation:

abr = (sappbytes - dappbytes)/(sappbytes + dappbytes)

This generates values between +1.0 - -1.0.  +1.0 means that all the app bytes
were from the source, indicating that the source is a pure PRODUCER, and the
destination is a pure CONSUMER.  You see this in FTP PUT file transfers,
as an example.  The sign bit reverses this relationship.

-0.0 denotes the special case, when there are no appbytes seen.

In the new argus-clients that I'll put up later today, you can print this out using:

ra -r argus.data -s +abr

You can also do operations using this metric, such as filter and generate histograms.
Here is a run that I did to show how this maybe used in an anomaly detection
application.  Here is the simple frequency distribution for all the internal DNS
requests made to my local DNS server from a specific client, for all of 2013:

thoth:06 carter$ pwd
/Volumes/Data/Archive/QoSient/192.168.0.68/2013
thoth:tmp carter$ rahisto -H abr 10:-1.0-1.0 -R . -s mean stddev - udp port domain and src pkts 1 and dst pkts 1
N = 1027764  mean = -0.726195  stddev =  0.140532  max = 0.000000  min = -0.909605
        median = -0.749129     95% = -0.292517
          mode = -0.782609
Class      Interval         Freq    Rel.Freq     Cum.Freq          Mean     StdDev
  1   -1.000000e+00     225379    21.9291%     21.9291%     -0.815238   0.000650
  2   -8.000000e-01     738148    71.8208%     93.7498%     -0.740887   0.043837
  3   -6.000000e-01      10553     1.0268%     94.7766%     -0.534672   0.048717
  4   -4.000000e-01      35511     3.4552%     98.2318%     -0.283067   0.021374
  5   -2.000000e-01        250     0.0243%     98.2561%     -0.162791   0.000051
  6    0.000000e+00      17923     1.7439%    100.0000%      0.000000   0.000000
  7    2.000000e-01          0     0.0000%    100.0000%    
  8    4.000000e-01          0     0.0000%    100.0000%    
  9    6.000000e-01          0     0.0000%    100.0000%    
 10    8.000000e-01          0     0.0000%    100.0000%    


OK, should be very clear, that my host is a net CONSUMER of DNS data, not a net PRODUCER
because the " abr <= 0 ".  The corollary holds true, the local DNS service is a net PRODUCER of
data, and not a net CONSUMER of data, from the prospective of this particular end system.
So testing filters like this:
ra -r daily.file - abr gt 0 and port domain and src pkts 1 and dst pkts 1

Should reveal flows that deserve a closer look.

OK, there were a lot of flows where the ( abr == 0 ), which was surprising.
When DNS experiences a ServFail, the response is the same as the request, just with an error bit
set in the DNS header.  QoSient had a big issue in Jan, 2013, when 17923 DNS ServFail failures
occurred, so that is where the ( abr == 0 ) flows occured.  Important to know this when evaluating
DNS as a channel for CONSUMER to PRODUCER conversion.

But for DNS health and operability, looking for flows where the ( sappbytes == dappbytes ) is
also a pretty interesting thing to look for.

Hope this is helpful,

Carter

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax






Attachment (smime.p7s): application/pkcs7-signature, 6837 bytes
Permalink | Reply | 0 comments |
headers
Harry Hoffman | 7 May 2013 20:17
Favicon

filter syntax error

Hi All,

Running the latest (i think) argus-clients 3.0.7.7.

I've got a setup where I have multiple src ids configured. I'm trying to
just pull records for one src id but am getting a syntax error.

[root <at> usher ~]# ra -nnr /var/log/argus/argus.out - srcid eth0
ra[31322]: 14:14:38.492897 srcid eth0 unknown
ra[31321]: 14:14:38.890573 srcid eth0 filter syntax error

I believe that the srcids are correct as ra can print them out.

[root <at> usher ~]# ra -nnr /var/log/argus/argus.out -s
saddr,sport,daddr,dport,srcid -N 3
           SrcAddr  Sport            DstAddr  Dport              SrcId
    172.16.255.170.123          24.124.0.251.123                  eth1
      72.94.xx.xxx.123          24.124.0.251.123                  eth0
    172.16.255.254            172.16.255.176                      eth1

Is this a bug or am I doing something wrong?

Cheers,
Harry
Permalink | Reply | 2 comments |

Gmane