Harry Bock | 4 Jun 04:31

Re: Developing an Argus client

Hi all,

I've published my current work to our public cgit repository.
The current periscope source code can be found at http://projects.oshean.org/index.cgi/periscope-argus/.  I hope eventually that this work will be of some use to the community! :)

On Fri, May 29, 2009 at 3:58 PM, Harry Bock <harry <at> oshean.org> wrote:
Hey Carter,

Two quick questions:
- What is the difference between records with net subtypes ARGUS_TCP_PERF, ARGUS_TCP_INIT, and ARGUS_TCP_STATUS? Is it possible to receive all three for the same flow, in three different RaProcessRecord calls?

- I'm currently testing things using PCAP captures processed by the argus server program, but none of the TCP flows I've seen have the ARGUS_NORMAL_CLOSE flag set - should I assume it's a normal close if none of the ARGUS_RESET etc. flags are set?

Thanks,
Harry



--
Harry Bock
Software Developer, Package Maintainer
OSHEAN, Inc.
Email: harry <at> oshean.org
PGP Key ID: 546CC353
Harry Bock | 4 Jun 04:29

Application-specific context in ArgusParserStruct

Hi Carter,

I've come across a little snag in developing Periscope; I'd like to store a pointer to a mutable context relevant only to my application, without adding more fields to ArgusParserStruct and without using global variables.  I've noticed that a lot of the client-specific data structures are all stored within ArgusParserStruct, but there is no "agnostic" field where I could simply link in my own.

I've worked around this by saving my context in parser->RaFlowModel, which I've verified does not get modified by any Argus client function that I'm using, directly or indirectly.  In the future, however, I'd suggest adding a simple field like so to ArgusParserStruct:

struct ArgusParserStruct {
  int status, ...;
  ...
+  void *context;
};

As Argus clients are currently linked statically to the common/event/parser libraries, binary compatibility shouldn't be an issue, but you obviously know the direction of Argus far better than I could, so I don't want to change anything sensitive.  If the change is acceptable, could it be included in the final release of argus-clients-3.0.1?

This change would also greatly help reduce clutter in ArgusParserStruct, as you could move client-specific data (such as what is needed by radium, rabins, rahisto, etc) out of common code and straight into the application.  I'd be more than happy to contribute the patches necessary to all of the Argus clients, given a week or two.

--
Harry Bock
Software Developer, Package Maintainer
OSHEAN, Inc.
Email: harry <at> oshean.org
PGP Key ID: 546CC353

Carter Bullard | 5 Jun 16:17

Re: Application-specific context in ArgusParserStruct

Hey Harry,
I wouldn't "take over" any field that is in the ArgusParserStruct.  Argus is a subset
of a larger system.  At anytime, I may bring over code that uses these variables,
and then you would really not be in a good position at all.

Adding a variable is a no brainer, but the name needs a little work.
I've added "void *ArgusClientContext" to the ArgusParserStruct, and it will
be in the next upload.

A few concepts that you need to consider.  Don't assume that there is only
one ArgusParser.  The design is that within the context of the callout routines, 
there is only one valid ArgusParser, and its the one that we are passing.
For argus-clients, there is only one, but for other systems, there are multiple
ArgusParser's, so just take that into consideration, if you can.

Be sure and deallocate the ArgusClientContext in your RaParseComplete()
routine.

Carter





On Jun 3, 2009, at 10:29 PM, Harry Bock wrote:

Hi Carter,

I've come across a little snag in developing Periscope; I'd like to store a pointer to a mutable context relevant only to my application, without adding more fields to ArgusParserStruct and without using global variables.  I've noticed that a lot of the client-specific data structures are all stored within ArgusParserStruct, but there is no "agnostic" field where I could simply link in my own.

I've worked around this by saving my context in parser->RaFlowModel, which I've verified does not get modified by any Argus client function that I'm using, directly or indirectly.  In the future, however, I'd suggest adding a simple field like so to ArgusParserStruct:

struct ArgusParserStruct {
  int status, ...;
  ...
+  void *context;
};

As Argus clients are currently linked statically to the common/event/parser libraries, binary compatibility shouldn't be an issue, but you obviously know the direction of Argus far better than I could, so I don't want to change anything sensitive.  If the change is acceptable, could it be included in the final release of argus-clients-3.0.1?

This change would also greatly help reduce clutter in ArgusParserStruct, as you could move client-specific data (such as what is needed by radium, rabins, rahisto, etc) out of common code and straight into the application.  I'd be more than happy to contribute the patches necessary to all of the Argus clients, given a week or two.

--
Harry Bock
Software Developer, Package Maintainer
OSHEAN, Inc.
Email: harry <at> oshean.org
PGP Key ID: 546CC353


Attachment (smime.p7s): application/pkcs7-signature, 3815 bytes
Carter Bullard | 5 Jun 16:36

Re: Developing an Argus client

Hey Harry,
I don't think I answered your earlier questions?
Question #1.  Yes, they represent different metrics for the different phases of a TCP connection,
and can be controlled using the argus.conf variables.  ARGUS_TCP_PERF DSR's, are very big,
so we don't want to transport them if we don't have any PERF data to report.  The
ARGUS_TCP_STATUS, struct is the smallest and contains only the flags fields.  You will
see these more likely when the clients are converting Netflow records to Argus records.

The ARGUS_TCP_INIT struct will be seen when you see failed TCP connections.  I.e., 
SYN -> RST volleys, TCP RST scans, etc..... or when argus() is configured to not generate
the ARGUS_TCP_PERF dsrs.

 If you have configured argus() to generate ARGUS_TCP_PERF dsrs,  you can get either
ARGUS_TCP_INIT or ARGUS_TCP_PERF dsrs for the same flow, depending on many
conditions.


Question #2. I took out ARGUS_NORMAL_CLOSE a while back, because most Windows
machines weren't closing normally (using fast close strategies), and so it was confusing
to use the metric.  Tcp normal close was just this filter:

   "tcp and fin and finack"

So you can add that kind of logic to your system, if you like, but just because a system
sends a RST, doesn't mean it wasn't a healthy, normal TCP connection, so thats why that
is not there.

Carter

On Jun 3, 2009, at 10:31 PM, Harry Bock wrote:

Hi all,

I've published my current work to our public cgit repository.
The current periscope source code can be found at http://projects.oshean.org/index.cgi/periscope-argus/.  I hope eventually that this work will be of some use to the community! :)

On Fri, May 29, 2009 at 3:58 PM, Harry Bock <harry <at> oshean.org> wrote:
Hey Carter,

Two quick questions:
- What is the difference between records with net subtypes ARGUS_TCP_PERF, ARGUS_TCP_INIT, and ARGUS_TCP_STATUS? Is it possible to receive all three for the same flow, in three different RaProcessRecord calls?

- I'm currently testing things using PCAP captures processed by the argus server program, but none of the TCP flows I've seen have the ARGUS_NORMAL_CLOSE flag set - should I assume it's a normal close if none of the ARGUS_RESET etc. flags are set?

Thanks,
Harry



Attachment (smime.p7s): application/pkcs7-signature, 3815 bytes
Harry Bock | 5 Jun 16:44

Re: Application-specific context in ArgusParserStruct

Hey Carter,

Thanks for adding that field, that will be immensely helpful.  I know taking over a random field is a (very) bad idea, but I need it for the time being, until I rebase against the latest version of argus-clients.

In terms of multiple Argus parsers, I was actually going to ask a question about that.  The global instance ArgusParser is used in quite a bit of code, including routines that take an ArgusParserStruct argument; how does this play into thread safety of the library? As you've written clients before with multiple ArgusParsers, is there anything I need to do to ensure there won't be any race conditions?

Thanks,
Harry

On Fri, Jun 5, 2009 at 10:17 AM, Carter Bullard <carter <at> qosient.com> wrote:
Hey Harry,
I wouldn't "take over" any field that is in the ArgusParserStruct.  Argus is a subset
of a larger system.  At anytime, I may bring over code that uses these variables,
and then you would really not be in a good position at all.

Adding a variable is a no brainer, but the name needs a little work.
I've added "void *ArgusClientContext" to the ArgusParserStruct, and it will
be in the next upload.

A few concepts that you need to consider.  Don't assume that there is only
one ArgusParser.  The design is that within the context of the callout routines, 
there is only one valid ArgusParser, and its the one that we are passing.
For argus-clients, there is only one, but for other systems, there are multiple
ArgusParser's, so just take that into consideration, if you can.

Be sure and deallocate the ArgusClientContext in your RaParseComplete()
routine.

Carter





On Jun 3, 2009, at 10:29 PM, Harry Bock wrote:

Hi Carter,

I've come across a little snag in developing Periscope; I'd like to store a pointer to a mutable context relevant only to my application, without adding more fields to ArgusParserStruct and without using global variables.  I've noticed that a lot of the client-specific data structures are all stored within ArgusParserStruct, but there is no "agnostic" field where I could simply link in my own.

I've worked around this by saving my context in parser->RaFlowModel, which I've verified does not get modified by any Argus client function that I'm using, directly or indirectly.  In the future, however, I'd suggest adding a simple field like so to ArgusParserStruct:

struct ArgusParserStruct {
  int status, ...;
  ...
+  void *context;
};

As Argus clients are currently linked statically to the common/event/parser libraries, binary compatibility shouldn't be an issue, but you obviously know the direction of Argus far better than I could, so I don't want to change anything sensitive.  If the change is acceptable, could it be included in the final release of argus-clients-3.0.1?

This change would also greatly help reduce clutter in ArgusParserStruct, as you could move client-specific data (such as what is needed by radium, rabins, rahisto, etc) out of common code and straight into the application.  I'd be more than happy to contribute the patches necessary to all of the Argus clients, given a week or two.

--
Harry Bock
Software Developer, Package Maintainer
OSHEAN, Inc.
Email: harry <at> oshean.org
PGP Key ID: 546CC353





--
Harry Bock
Software Developer, Package Maintainer
OSHEAN, Inc.
Email: harry <at> oshean.org
PGP Key ID: 546CC353
John Kennedy | 6 Jun 00:35
Picon

Removing Internal flows from output

Extrusion Detection:
How would I filter out internal to internal traffic, but still be able to see traffic from an internal address going to an internet address?  The egress traffic could potentially be over any port.  I.E. I want to be able to ignore internal traffic (e.g. 10.0.0.1:11223 -> 10.2.3.5:80) and focus on any traffic bound to an internet IP address. (e.g. 10.0.0.1:11223 -> 121.10.114.137:80|21|443|6667|whatever).  Is there a way to get what I am asking for using an argus-clent and without using Perl/Bash/Ruby/Tcl etc.

In using some filters e.g. ra -L0 -n -r argus.log - not host 10.0.0.1 will filter all traffic for 10.0.0.1 even the egress traffic.

Thanks

John

John Kennedy | 6 Jun 05:52
Picon

Re: Removing Internal flows from output

I think I figured it out... :D Yah for me!!!! 

On Fri, Jun 5, 2009 at 4:35 PM, John Kennedy <wilson.amajohn <at> gmail.com> wrote:
Extrusion Detection:
How would I filter out internIal to internal traffic, but still be able to see traffic from an internal address going to an internet address?  The egress traffic could potentially be over any port.  I.E. I want to be able to ignore internal traffic (e.g. 10.0.0.1:11223 -> 10.2.3.5:80) and focus on any traffic bound o an internet IP address. (e.g. 10.0.0.1:11223 -> 121.10.114.137:80|21|443|6667|whatever).  Is there a way to get what I am asking for using an argus-clent and without using Perl/Bash/Ruby/Tcl etc.

In using some filters e.g. ra -L0 -n -r argus.log - not host 10.0.0.1 will filter all traffic for 10.0.0.1 even the egress traffic.

Thanks

John

Carter Bullard | 8 Jun 15:31

Re: Removing Internal flows from output

Hey John,
Glad you're figuring it out ;o)
Did you're filter look anything like this?

   ra - ip and not \(src net 10.0.0.0/8 and dst net 10.0.0.0/8\)

Carter

On Jun 5, 2009, at 11:52 PM, John Kennedy wrote:

I think I figured it out... :D Yah for me!!!! 

On Fri, Jun 5, 2009 at 4:35 PM, John Kennedy <wilson.amajohn <at> gmail.com> wrote:
Extrusion Detection:
How would I filter out internIal to internal traffic, but still be able to see traffic from an internal address going to an internet address?  The egress traffic could potentially be over any port.  I.E. I want to be able to ignore internal traffic (e.g. 10.0.0.1:11223 -> 10.2.3.5:80) and focus on any traffic bound o an internet IP address. (e.g. 10.0.0.1:11223 -> 121.10.114.137:80|21|443|6667|whatever).  Is there a way to get what I am asking for using an argus-clent and without using Perl/Bash/Ruby/Tcl etc.

In using some filters e.g. ra -L0 -n -r argus.log - not host 10.0.0.1 will filter all traffic for 10.0.0.1 even the egress traffic.

Thanks

John





Attachment (smime.p7s): application/pkcs7-signature, 3815 bytes
John Kennedy | 8 Jun 19:49
Picon

Re: Removing Internal flows from output

Very similar..

racluster -M norep -m matrix proto saddr daddr dport -r /var/log/argus/argus.log -w - - ip and not dst net 192.168.0.0/16 and not dst net 10.0.0.0/8 and not dst net 172.16.0.0/12 | rasort -m bytes -w - | ra -LO -n -N 10 -s proto saddr daddr dport pkts bytes state - not ip proto eigrp

At this point I don't want to know about internal Lan to Lan connections. I want to know what private address is making requests to the outside world. e.g. Port 80/443 traffic not using the proxy.  DNS Requests to known bad domains etc.  P2P Traffic indications etc.

Basically I am building a report similar to Mr. Van Epp's Perl Script to provide indications that a host may be compromised.

Regards,

John


On Mon, Jun 8, 2009 at 7:31 AM, Carter Bullard <carter <at> qosient.com> wrote:
Hey John,
Glad you're figuring it out ;o)
Did you're filter look anything like this?

   ra - ip and not \(src net 10.0.0.0/8 and dst net 10.0.0.0/8\)

Carter

On Jun 5, 2009, at 11:52 PM, John Kennedy wrote:

I think I figured it out... :D Yah for me!!!! 

On Fri, Jun 5, 2009 at 4:35 PM, John Kennedy <wilson.amajohn <at> gmail.com> wrote:
Extrusion Detection:
How would I filter out internIal to internal traffic, but still be able to see traffic from an internal address going to an internet address?  The egress traffic could potentially be over any port.  I.E. I want to be able to ignore internal traffic (e.g. 10.0.0.1:11223 -> 10.2.3.5:80) and focus on any traffic bound o an internet IP address. (e.g. 10.0.0.1:11223 -> 121.10.114.137:80|21|443|6667|whatever).  Is there a way to get what I am asking for using an argus-clent and without using Perl/Bash/Ruby/Tcl etc.

In using some filters e.g. ra -L0 -n -r argus.log - not host 10.0.0.1 will filter all traffic for 10.0.0.1 even the egress traffic.

Thanks

John






Carter Bullard | 8 Jun 21:26

Re: Removing Internal flows from output

Hey John,
Also, if you're argus is well positioned and you know what L2 addresses are
involved in the external traffic, you can use a simple "ether host xx:xx:xx:xx:xx:xx"
filter on the input.

The reason the L2 filters are more preferable, is that you can find errant traffic with
a L2 filter, but L3 filters may exclude some traffic that you would want to be interested
in, like traffic with spoofed source addresses and the like.

BUT, for most purposes, the L3 filters are the most logical.

Carter

On Jun 8, 2009, at 1:49 PM, John Kennedy wrote:

Very similar..

racluster -M norep -m matrix proto saddr daddr dport -r /var/log/argus/argus.log -w - - ip and not dst net 192.168.0.0/16 and not dst net 10.0.0.0/8 and not dst net 172.16.0.0/12 | rasort -m bytes -w - | ra -LO -n -N 10 -s proto saddr daddr dport pkts bytes state - not ip proto eigrp

At this point I don't want to know about internal Lan to Lan connections. I want to know what private address is making requests to the outside world. e.g. Port 80/443 traffic not using the proxy.  DNS Requests to known bad domains etc.  P2P Traffic indications etc.

Basically I am building a report similar to Mr. Van Epp's Perl Script to provide indications that a host may be compromised.

Regards,

John


On Mon, Jun 8, 2009 at 7:31 AM, Carter Bullard <carter <at> qosient.com> wrote:
Hey John,
Glad you're figuring it out ;o)
Did you're filter look anything like this?

   ra - ip and not \(src net 10.0.0.0/8 and dst net 10.0.0.0/8\)

Carter

On Jun 5, 2009, at 11:52 PM, John Kennedy wrote:

I think I figured it out... :D Yah for me!!!! 

On Fri, Jun 5, 2009 at 4:35 PM, John Kennedy <wilson.amajohn <at> gmail.com> wrote:
Extrusion Detection:
How would I filter out internIal to internal traffic, but still be able to see traffic from an internal address going to an internet address?  The egress traffic could potentially be over any port.  I.E. I want to be able to ignore internal traffic (e.g. 10.0.0.1:11223 -> 10.2.3.5:80) and focus on any traffic bound o an internet IP address. (e.g. 10.0.0.1:11223 -> 121.10.114.137:80|21|443|6667|whatever).  Is there a way to get what I am asking for using an argus-clent and without using Perl/Bash/Ruby/Tcl etc.

In using some filters e.g. ra -L0 -n -r argus.log - not host 10.0.0.1 will filter all traffic for 10.0.0.1 even the egress traffic.

Thanks

John







Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York  10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax



Attachment (smime.p7s): application/pkcs7-signature, 3815 bytes

Gmane