Stéphane Peters | 3 May 2010 17:14
Picon

radump() shows only the first request [Re: Re: Question about payload

Hello,

one thing to note, there is a slight difference between ra() and radump() when several packets are aggregated in one flow.
ra() shows the first bytes of the flow, while radump() only dumps the first request.

For example, let's take a DNS argus record.
The output shown by ra()  is concatenated in one flow (some lines have been split for readability):
-bash-3.2$ ra -nr example.ra  -s +suser:200 +duser:200
  StartTime    Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  SrcPkts  DstPkts   TotBytes  State \
    srcUdata      dstUdata
30/04 15:01  e         udp            1.0.2.1.49055    <->          100.0.1.1.53            3        2        640   CON \
    s[115]=V............\
        dnl-07.geo.kaspersky.com......\
        A...........\
        meteo.fr.be.msn.com..................\
        quiz.fr.be.msn.com..... \
    \
    d[128]=V............\
        dnl-07.geo.kaspersky.com..............\
        <.....).............\
        geons4.kaspersky-labs.!.............\
        geons1.M.............
The DNS requests done from the anonymized address 1.0.2.1 during this DNS flow are unrelated as usual :
    dnl-07.geo.kaspersky.com
    meteo.fr.be.msn.com
    quiz.fr.be.msn.com
radump() shows only the first transaction; once decoded, it displays 45 bytes for the source and 33 bytes for the destination:
-bash-3.2$ radump -nr example.ra  -s +suser:200 +duser:200
30/04 15:01  e         udp            1.0.2.1.49055    <->          100.0.1.1.53            3        2        640   CON   \
    s[45]="22176+ [_] A? dnl-07.geo.kaspersky.com. (115)"   \
    d[33]="22176 1/4/4 A 195.222.17.41 (128)"   
I don't know if it is desirable or possible to show the remaining requests, ie meteo.fr.be.msn.com and quiz.fr.be.msn.com.

I have searched some clues reading the source of the 3.0.3.5 radump files, here they are:
s[45]          is the string length that radump writes to stdout for the src payload (use wc() to check )
22176        matches the DNS Transaction ID ("print-domain.c" line 619 )
+                means recursion desired ("print-domain.c" line 620) ("nameser.h" line 224 )
1/4/4          means 1 answer record, 4 authority records, 4 additional records ("print-domain.c" line 574);
                    specify -vv to enhance verbosity  ("print-domain.c" line 586)
(115)         indicates the length of the buffer before conversion ("print-domain.c" line 706 )
[|domain]   means something was truncated ("print-domain.c" line 710)

In fact, you could find nearly the same info launching tcpdump() on the matching packets:
-bash-3.2$  tcpdump -nr example.pcap | MyAnonymize
reading from file example.pcap, link-type EN10MB (Ethernet)
15:01:00.823204 IP 1.0.2.1.49055 > 100.0.1.1.domain: 22176+ A? dnl-07.geo.kaspersky.com. (42)
15:01:00.851334 IP 100.0.1.1.domain > 1.0.2.1.49055: 22176 1/4/4 A 195.222.17.41 (221)
15:01:05.650905 IP 1.0.2.1.49055 > 100.0.1.1.domain: 2113+ AAAA? meteo.fr.be.msn.com. (37)
15:01:05.848604 IP 100.0.1.1.domain > 1.0.2.1.49055: 2113 1/0/0 CNAME weather.db2.cb3.glbdns.microsoft.com. (84)
15:01:06.698140 IP 1.0.2.1.49055 > 100.0.1.1.domain: 2772+ AAAA? quiz.fr.be.msn.com. (36)
15:01:06.741270 IP 100.0.1.1.domain > 1.0.2.1.49055: 2772 1/0/0 CNAME quiz.fr.be.msn.com.itm.asp.msn.com.nsa4c.net. (94)
The src buffer length was (115) in argus; in tcpdump you have: 42 + 37 + 36 = 115,
another proof that the three packets were concatenated in one flow.
The dst buffer was filled to the configured size of 128 bytes.

Let's try radump() with the newly discovered option -vv :
-bash-3.2$ radump -nr /tmp/example.ra  -s +suser:55 +duser:300 -vvv
30/04 15:01  e         udp            1.0.2.1.49055    <->          100.0.1.1.53            3        2        640   CON  \
s[45]="22176+ [_] A? dnl-07.geo.kaspersky.com. (115)"    \
d[220]="22176 q: A? dnl-07.geo.kaspersky.com. \
   1/4/4 dnl-07.geo.kaspersky.com. \
   A 195.222.17.41 \
   ns: geo.kaspersky.com. \
   NS geons4.kaspersky-labs.com., geo.kaspersky.com. \
   NS geons1.kaspersky-labs.com., geo.kaspersky.com. \
   NS[|domain]"



Carter Bullard a écrit :
Hey Paul, s[51] means that this is the source user buffer and that its 51 bytes long. this is the payload of the packet sent from the entity that initiated the flow. d[135] means that this is the destination user buffer and its 135 bytes long. this is the payload of the packet sent from the entity that responded to the initiator. Using ra(), you are asking to print the raw buffer. the default is to print out as if its an ascii buffer, the '.'s are unprintable chars. For some protocols, like http, the buffers are just ascii strings. For others like DNS, they are a mix of binary and ascii. Definately looks like DNS traffic to me. radump() will attempt to parse out the user buffers and print out what it understands the buffer contents to be. Using radump(), you would get an output that looked like tcpdump's output for DNS. radump -r file -s +suser:128 +duser:135 - port domain the field lengths (:xxx) are whatever it takes to decode the buffer and print it out in ascii successfully. Carter On Apr 27, 2010, at 4:43 PM, Paul Schmehl wrote:
We're using argus to capture partial payloads. The output is quite a bit different from tcpdump, and there's some parts I don't understand. I'm hoping the experts here can enlighten me. What does s[51]= mean? What does d[135]= mean? I took the first one to mean the payload, but then the second seems unclear to me. Here's the packet I'm referring to: 27 Apr 10 18:21:39.137180 M udp 129.110.31.40.18677 <-> 92.241.190.252.domain CON s[51]=d............sandra.prichaonica.com.......)........ d[135]=d............sandra.prichaonica.com.................\................ns2...............ns1...V..........\....D..........\.....)........ Clearly it's a DNS lookup, but I don't get what the s[51]= and d[135]= refer to. -- Paul Schmehl, Senior Infosec Analyst As if it wasn't already obvious, my opinions are my own and not those of my employer. ******************************************* "It is as useless to argue with those who have renounced the use of reason as to administer medication to the dead." Thomas Jefferson
Carter Bullard CEO/President QoSient, LLC 150 E 57th Street Suite 12D New York, New York 10022 +1 212 588-9133 Phone +1 212 588-9134 Fax
Regards,
-- Stephane.Peters <at> forem.be
Carter Bullard | 4 May 2010 06:37

Re: argus-3.0.3.7 some Gentoo patches

Hey Peter,
I've added your changes to argus-3.0.3.8, which I should upload tomorrow.
If you see that I messed something up, just holler!!!!
Thanks!!!

Carter

On Apr 29, 2010, at 5:07 AM, Peter Volkov wrote:

> Hi, Carter. In Gentoo we bundle some patches for argus and I think it's
> good idea to include them upstream. Please, review and apply if
> appropriate:
> 
> 1. disable-tcp-wrappers-automagic just adds ./configure switch to allow
> users avoid linking with libwrap even if it is installed on system. (by
> Jeroen Roovers)
> 2. autoconf-2.65 is a patch to make argus buildable with autoconf-2.65.
> In other case it fails like reported in bug: bugs.gentoo.org/297943
> 3. override-CFLAGS.patch is a patch to allow user set desired
> optimization level. In other case argus forces -O3.
> 
> With best regards,
> -- 
> Peter.
> <argus-3.0.2-disable-tcp-wrappers-automagic.patch><argus-3.0.3.7-autoconf-2.65.patch><argus-3.0.3.7-override-CFLAGS.patch>

Attachment (smime.p7s): application/pkcs7-signature, 3815 bytes
Peter Volkov | 6 May 2010 11:03
Picon
Favicon

Re: argus-3.0.3.7 some Gentoo patches

В Втр, 04/05/2010 в 00:37 -0400, Carter Bullard пишет:
> I've added your changes to argus-3.0.3.8, which I should upload tomorrow.
> If you see that I messed something up, just holler!!!!

Hi Carter. It looks like one patch was applied somehow differently as
argus-3.0.3.8 still fails to build with autoconf-2.65:

configure: loading site script /usr/share/config.site
configure: loading site script /usr/share/crossdev/include/site/linux
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
./configure: line 2432: syntax error near unexpected token `newline'
./configure: line 2432: `yes:'

Updated patch is in attachment. Please, apply.

--

-- 
Peter.
Carter Bullard | 7 May 2010 22:31

Re: argus-3.0.3.7 some Gentoo patches

Thanks, I'll add it this weekend.
Carter

On May 6, 2010, at 5:03 AM, Peter Volkov wrote:

> В Втр, 04/05/2010 в 00:37 -0400, Carter Bullard пишет:
>> I've added your changes to argus-3.0.3.8, which I should upload tomorrow.
>> If you see that I messed something up, just holler!!!!
> 
> Hi Carter. It looks like one patch was applied somehow differently as
> argus-3.0.3.8 still fails to build with autoconf-2.65:
> 
> configure: loading site script /usr/share/config.site
> configure: loading site script /usr/share/crossdev/include/site/linux
> checking build system type... x86_64-pc-linux-gnu
> checking host system type... x86_64-pc-linux-gnu
> checking target system type... x86_64-pc-linux-gnu
> ./configure: line 2432: syntax error near unexpected token `newline'
> ./configure: line 2432: `yes:'
> 
> 
> Updated patch is in attachment. Please, apply.
> 
> -- 
> Peter.
> <argus-3.0.3.8-autoconf-2.65.patch>

Attachment (smime.p7s): application/pkcs7-signature, 3681 bytes
George Jones | 14 May 2010 16:32
Picon
Favicon

Python bindings ?

Has anyone created python bindings for argus?  e.g. something that would
read ra ouput (or argus binary format)  and parse into python class
structures ?

Thanks,
---George Jones

Carter Bullard | 14 May 2010 17:27

new threaded argus packet input and data output engines

Gentle people,
As promised, new features abound in argus-3.0.3.9

I've uploaded a new argus that has a new threaded model for packet
input processing and new transport strategies for argus data on 
the output side.

There will be a .threads tag file in the root directory, which turns on the
threaded support for the new features.  If you find that there are issues,
remove this tag file, and re-configure, make clean and make.

The significance of the new ARGUS_INTERFACE strategies are that you
get multiple threads for multiple interfaces, allowing for much better
performance.   Also, you can now have a single argus that monitors
multiple interfaces "independently", working as if you had independent
argi monitoring either interface.   This is important for things like laptops
that have wired and wireless interfaces, and you want to monitor both
at the same time, with separate monitor ids.  Now a single argus, can
support multiple observation domains, concurrently.

If you monitor multiple interfaces at a time on multi-core platforms, do
test this version of argus.  Please send email if you have any
problems at all  !!!!

Here is a snip from the new argus.conf file found in ./support/Config.

# Argus can track packets from any or all interfaces, concurrently.
# The interfaces can be tracked as:
#   1.  independant - this is where argus tracks flows from each
#          interface independant from the packets seen on any
#          other interface.  This is useful for hosts/routers that
#          have full-duplex interfaces, and you want to distinguish
#          flows based on their interface. There is an option to specify
#          a distinct srcid to each independant modeler.
#
#   2.  duplex - where argus tracks packets from 2 interfaces
#          as if they were two half duplex streams of the same link.
#          Because there is a single modeler tracking the 2
#          interfaces, there is a single srcid that can be passed as
#          an option.
# 
#   3.  bonded - where argus tracks packets from multiple interfaces
#          as if they were from the same stream.  Because there is a
#          single modeler tracking the 2 interfaces, there is a single
#          srcid that can be passed as an option.
# 
#  Interfaces can be specified as groups using '[',']' notation, to build
#  flexible definitions of packet sources.  However, each interface
#  should be referenced only once (this is due to performance and OS
#  limitations, so if your OS has no problem with this, go ahead).
#
#  The lo (loopback) interface will be included only if it is specifically
#  indicated in the option.
#
#  The syntax for specifying this either on the command line or in this file:
#     -i ind:all
#     -i dup:en0,en1/srcid
#     -i bond:en0,en1/srcid
#     -i dup:[bond:en0,en1],en2/srcid
#     -i en0/srcid -i en1/srcid  (equivalent '-i ind:en0/srcid,en1/srcid')
#     -i en0 en1                 (equivalent '-i bond:en0,en1')

For output, argus can now "push" records via AF_INET UDP based sockets,
so that you can multicast records (one use of the new feature) to multiple
listeners.  This "push" method is in addition to the "pull" methods already
available.  Here is the additional documentation in the sample ./support/Config/argus.conf file:

# Argus can write its output to one or a number of remote hosts.
# The default limit is 5 concurrent output streams, each with their
# own independant filters.
#
# The format is:
#      ARGUS_OUTPUT_STREAM="URI [filter]"
#      ARGUS_OUTPUT_STREAN="argus-udp://host:port 'tcp and not udp'"
#
# Most sites will have argus listen() for remote sites to request
# argus data, but for some sites and applications sending records without
# registration is desired.  This option will cause argus to transmit records
# that match the optional filter, to the configured targets using UDP as the
# transport mechanism.
#
# Commandline equivalent   -w argus-udp://host:port
#

#ARGUS_OUTPUT_STREAM=argus-udp://224.0.20.21:561

argus clients can currently read from these udp streams using:

   ra -S argus-udp://host:port

Again, if you have any problems with these features, send email!!!!

Carter

Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York  10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax

Attachment (smime.p7s): application/pkcs7-signature, 3815 bytes
Carter Bullard | 14 May 2010 17:36

Re: Python bindings ?

Hey George,
I don't do any python, but it couldn't be hard.
Are there guides for doing this type of thing?

Carter

On May 14, 2010, at 10:32 AM, George Jones wrote:

> Has anyone created python bindings for argus?  e.g. something that would
> read ra ouput (or argus binary format)  and parse into python class
> structures ?
> 
> Thanks,
> ---George Jones
> 
> 
> 

Attachment (smime.p7s): application/pkcs7-signature, 3815 bytes
George Jones | 14 May 2010 17:44
Picon
Favicon

Re: Python bindings ?

On Fri, 2010-05-14 at 11:36 -0400, Carter Bullard wrote:
> Hey George,
> I don't do any python, but it couldn't be hard.
> Are there guides for doing this type of thing?

google python class

PySiLK is the one thing I'm really missing from SiLK.   I think I can
roll a parser pretty quickly for reading ra output and shoving it into
classes.

---George

> 
> Carter
> 
> On May 14, 2010, at 10:32 AM, George Jones wrote:
> 
> > Has anyone created python bindings for argus?  e.g. something that would
> > read ra ouput (or argus binary format)  and parse into python class
> > structures ?
> > 
> > Thanks,
> > ---George Jones
> > 
> > 
> > 
> 
> 
> 
> 

John Gerth | 14 May 2010 18:21
Picon
Favicon

Re: Python bindings ?

ra can output XML and there are beaucoup python parsers for that.

On 5/14/2010 8:44 AM, George Jones wrote:
> On Fri, 2010-05-14 at 11:36 -0400, Carter Bullard wrote:
>> Hey George,
>> I don't do any python, but it couldn't be hard.
>> Are there guides for doing this type of thing?
> 
> google python class
> 
> PySiLK is the one thing I'm really missing from SiLK.   I think I can
> roll a parser pretty quickly for reading ra output and shoving it into
> classes.
> 
> ---George
> 
>> Carter
>>
>> On May 14, 2010, at 10:32 AM, George Jones wrote:
>>
>>> Has anyone created python bindings for argus?  e.g. something that would
>>> read ra ouput (or argus binary format)  and parse into python class
>>> structures ?
>>>
>>> Thanks,
>>> ---George Jones
>>>

Carter Bullard | 14 May 2010 18:22

Re: Python bindings ?

How would you describe what you get by having python in the mix?
Not being a python person, I really don't know why python :o)

Carter

On May 14, 2010, at 11:44 AM, George Jones wrote:

On Fri, 2010-05-14 at 11:36 -0400, Carter Bullard wrote:
Hey George,
I don't do any python, but it couldn't be hard.
Are there guides for doing this type of thing?

google python class

PySiLK is the one thing I'm really missing from SiLK.   I think I can
roll a parser pretty quickly for reading ra output and shoving it into
classes.

---George


Carter

On May 14, 2010, at 10:32 AM, George Jones wrote:

Has anyone created python bindings for argus?  e.g. something that would
read ra ouput (or argus binary format)  and parse into python class
structures ?

Thanks,
---George Jones






Attachment (smime.p7s): application/pkcs7-signature, 3815 bytes

Gmane