9 Dec 06:12
9 Dec 17:39
Re: color graphs
Carter Bullard <carter <at> qosient.com>
2009-12-09 16:39:03 GMT
2009-12-09 16:39:03 GMT
Hey Rodney,
The output is a color graph. You don't see any colors?
Carter
On Dec 9, 2009, at 12:12 AM, Rodney McKee wrote:
How do I get something like this to give a result in color?
ragraph proto trans -nr argus.out-investigate -M 1m -w proto.png
Rgds Rodney
9 Dec 17:43
Re: Rasplit compression support
Matt Sheridan <mattmail5050 <at> gmail.com>
2009-12-09 16:43:54 GMT
2009-12-09 16:43:54 GMT
Hi Carter -
I was wondering what your thoughts might be on this moving forward? The defunct processes continue to pile up.
Not sure if its related, but if I run a basic ra on a file, I am getting some other process complaints:
ra -r ./argus.2009_12_09_1100.out.gz - host 10.10.10.10
ra[4935]: 11:42:56.648522 ArgusReadStreamSocket (0x41fc) record length is zero
gzip: stdout: Broken pipe
On Tue, Nov 17, 2009 at 10:51 AM, Matt Sheridan <mattmail5050 <at> gmail.com> wrote:
I have an update. It seems that what is becoming a zombie is the script called by the "-f" option (you may have already figured that out, I just did ;)"
I attempted to use the default contrib script included in the clients package, and it also zombied.
But as you can see:
root 1716 31417 8 10:46 pts/1 00:00:10 rastream -S localhost:561 -M time 1m -B 10s -f /opt/IDS/argus/etc/rastream-orig.sh -
root 2057 1716 0 10:47 pts/1 00:00:00 [rastream-orig.s] <defunct>
it was the rastream-orig.sh script, called by the -f option (which is used to zip up the files) that zombied. So it is possible that the problem is not within rastream, but in the way the called script is handled?On Mon, Nov 16, 2009 at 9:32 PM, Peter Van Epp <vanepp <at> sfu.ca> wrote:<snip>On Mon, Nov 16, 2009 at 04:38:38PM -0500, Matt Sheridan wrote:
> I was able to get it figured out. I admit not being fully versed in my
> compile skills :). Adding the .debug to the compile directory was the
> clarification that I missed.
>
> Here are the results from stdout. Zombie processes were created when looking
> at ps:
>
> rastream -S localhost:561 -M time 10m -B 10s -f
> /opt/IDS/argus/etc/rastream.sh -w
> /opt/IDS/argus/log/\$srcid/argus.%Y_%m_%d_%H%M.out -D1
> rastream[14067.a08e7288f12a0000]: 16:03:10.090959 main: reading files
> completed
> rastream[14067.4019a54100000000]: 16:03:10.091388 Trying 127.0.0.1 port 561
> Expecting Argus records
> rastream[14067.4019a54100000000]: 16:03:10.091652 connected
> rastream[14067.4019a54100000000]: 16:03:10.091768 ArgusGetServerSocket
> (0x87df0010) returning 3
> rastream[14067.a08e7288f12a0000]: 16:10:15.474102 ArgusRunScript(0x87d41010,
> 21198c10) filename /opt/IDS/argus/log/127.0.0.1/argus.2009_11_16_1600.out
> rastream[14067.a08e7288f12a0000]: 16:10:15.474201 ArgusRunScript(0x87d41010,
> 0x21198c10) scheduling /opt/IDS/argus/etc/rastream.sh -r
> /opt/IDS/argus/log/127.0.0.1/argus.2009_11_16_1600.out
> rastream[14067.a08e7288f12a0000]: 16:10:15.474240 ArgusRunScript(0x87d41010,
> 21198c10) returning /opt/IDS/argus/etc/rastream.sh -r /opt/IDS/argus/log/
> 127.0.0.1/argus.2009_11_16_1600.out
> rastream[16367.a08e7288f12a0000]: 16:10:16.074641 ArgusRunScript calling
> /opt/IDS/argus/etc/rastream.sh -r /opt/IDS/argus/log/
> 127.0.0.1/argus.2009_11_16_1600.out
> deleting
While I haven't used rastream (let alone run scripts from it), can
you tell what is zombieing? My first guess (although I'm not an expert on
zombies either) would be that your script isn't returning a result to
rastream which is I think what causes a zombie. It would be instructive to
know what the original pid of one of the zombies is (as the pids are showing
in the argus debug output, although I don't seem to have left one). That
should point to what task is causing the zombie.
Peter Van Epp
9 Dec 19:40
Re: ra: window difference ?
Carter Bullard <carter <at> qosient.com>
2009-12-09 18:40:42 GMT
2009-12-09 18:40:42 GMT
Hey Julien, What filter(s) are you using to generate your numbers? Carter On Nov 29, 2009, at 10:06 AM, julien wrote: > Hello Carter, > > Carter Bullard wrote on 27/11/09 16:41: >> Argus will report the last window advertisement seen for the src and dst direction >> in each status report interval and it will also indicate if the window went to zero >> during that time. This zero state is the TCP flow control indicator, and you will >> see a "S", "D" or "@" indicator in column 5 of the flags field. > [...] > > thanks a lot for all this details > >> >> So, what are you most interested in in this packet trace? Is there a need to capture >> more in the TCP windows metric? > > that's not really a problem of capture. More about interpreting different results between Wireshark and Argus from the same data. > > When I speak about IP and Window, I will assume it's the same for everyone but here count/proportions are reversed so ... > > >> On Nov 26, 2009, at 3:35 PM, julien wrote: >>> does someone know the difference between Wireshark "Window Space" (tcp.window_space) and Argus "Window Advertisement" (swin/dwin) ? >>> >>> I'm currently investigating a pcap representing a kind of DoS Synflood attack. The former returns about 25k packets with size 0 a >>> nd 230k with size<n>, the latter returns 130k& 25k (swin only) ??? >>> > > Why Wireshark would return 9% of packets with size 0 and the others with 0 (filter with tcp.windows_space == 0 or n) > and Argus returns 84% of flows with size 0 and the others with size 0 ? (with ra) > > thanks > >
9 Dec 23:02
Re: ra: window difference ?
julien <julien.t43 <at> gmail.com>
2009-12-09 22:02:55 GMT
2009-12-09 22:02:55 GMT
Carter Bullard wrote on 09/12/09 19:40: > What filter(s) are you using to generate your numbers? here for wireshark >> Why Wireshark would return 9% of packets with size 0 and the others with 0 (filter with tcp.windows_space == 0 or n) >> and Argus returns 84% of flows with size 0 and the others with size 0 ? (with ra) and for argus, I make a chart with data from the following command: ra -n -s swin -r $src_log thanks
10 Dec 17:36
Re: ra: window difference ?
Carter Bullard <carter <at> qosient.com>
2009-12-10 16:36:14 GMT
2009-12-10 16:36:14 GMT
Hey Julien, In the file you uploaded, 131803 of the flows that are reporting source windows of zero are flows that are simple single packet flows, so you and I are getting consistent numbers, so I suspect that your numbers are real. I can't explain the discrepancy, as I have looked at the code and your sample file and there shouldn't be a problem. With the numbers switched, you should be able to find a packet in your file where wireshark thinks there is a window number, and the resulting argus record reports zero. If you could create a packet file that has a packet or two that wireshark thinks has a src window value but argus reports as zero, then I can debug. Carter On Dec 9, 2009, at 5:02 PM, julien wrote: > Carter Bullard wrote on 09/12/09 19:40: >> What filter(s) are you using to generate your numbers? > > > here for wireshark > >>> Why Wireshark would return 9% of packets with size 0 and the others with 0 (filter with tcp.windows_space == 0 or n) >>> and Argus returns 84% of flows with size 0 and the others with size 0 ? (with ra) > > and for argus, I make a chart with data from the following command: > ra -n -s swin -r $src_log > > thanks >
10 Dec 17:42
Re: Rasplit compression support
Carter Bullard <carter <at> qosient.com>
2009-12-10 16:42:21 GMT
2009-12-10 16:42:21 GMT
Hey Matt,
I can't recreate your condition on any machines I have, and if you
look at the code, and I think the debug output shows this, we are doing the
waitpid() needed to harvest the zombies, so I don't know how to
proceed. If you want to provide me with access to the machine
I'll be happy to try to debug the problem.
This problem could be as simple as a poor PATH definition for the script.
Do you have a PATH definition or full path specification for commands
called in the script?
Carter
On Dec 9, 2009, at 11:43 AM, Matt Sheridan wrote:
Hi Carter -
I was wondering what your thoughts might be on this moving forward? The defunct processes continue to pile up.
Not sure if its related, but if I run a basic ra on a file, I am getting some other process complaints:
ra -r ./argus.2009_12_09_1100.out.gz - host 10.10.10.10
ra[4935]: 11:42:56.648522 ArgusReadStreamSocket (0x41fc) record length is zero
gzip: stdout: Broken pipeOn Tue, Nov 17, 2009 at 10:51 AM, Matt Sheridan <mattmail5050 <at> gmail.com> wrote:
I have an update. It seems that what is becoming a zombie is the script called by the "-f" option (you may have already figured that out, I just did ;)"
I attempted to use the default contrib script included in the clients package, and it also zombied.
But as you can see:
root 1716 31417 8 10:46 pts/1 00:00:10 rastream -S localhost:561 -M time 1m -B 10s -f /opt/IDS/argus/etc/rastream-orig.sh -
root 2057 1716 0 10:47 pts/1 00:00:00 [rastream-orig.s] <defunct>
it was the rastream-orig.sh script, called by the -f option (which is used to zip up the files) that zombied. So it is possible that the problem is not within rastream, but in the way the called script is handled?On Mon, Nov 16, 2009 at 9:32 PM, Peter Van Epp <vanepp <at> sfu.ca> wrote:<snip>On Mon, Nov 16, 2009 at 04:38:38PM -0500, Matt Sheridan wrote:
> I was able to get it figured out. I admit not being fully versed in my
> compile skills :). Adding the .debug to the compile directory was the
> clarification that I missed.
>
> Here are the results from stdout. Zombie processes were created when looking
> at ps:
>
> rastream -S localhost:561 -M time 10m -B 10s -f
> /opt/IDS/argus/etc/rastream.sh -w
> /opt/IDS/argus/log/\$srcid/argus.%Y_%m_%d_%H%M.out -D1
> rastream[14067.a08e7288f12a0000]: 16:03:10.090959 main: reading files
> completed
> rastream[14067.4019a54100000000]: 16:03:10.091388 Trying 127.0.0.1 port 561
> Expecting Argus records
> rastream[14067.4019a54100000000]: 16:03:10.091652 connected
> rastream[14067.4019a54100000000]: 16:03:10.091768 ArgusGetServerSocket
> (0x87df0010) returning 3
> rastream[14067.a08e7288f12a0000]: 16:10:15.474102 ArgusRunScript(0x87d41010,
> 21198c10) filename /opt/IDS/argus/log/127.0.0.1/argus.2009_11_16_1600.out
> rastream[14067.a08e7288f12a0000]: 16:10:15.474201 ArgusRunScript(0x87d41010,
> 0x21198c10) scheduling /opt/IDS/argus/etc/rastream.sh -r
> /opt/IDS/argus/log/127.0.0.1/argus.2009_11_16_1600.out
> rastream[14067.a08e7288f12a0000]: 16:10:15.474240 ArgusRunScript(0x87d41010,
> 21198c10) returning /opt/IDS/argus/etc/rastream.sh -r /opt/IDS/argus/log/
> 127.0.0.1/argus.2009_11_16_1600.out
> rastream[16367.a08e7288f12a0000]: 16:10:16.074641 ArgusRunScript calling
> /opt/IDS/argus/etc/rastream.sh -r /opt/IDS/argus/log/
> 127.0.0.1/argus.2009_11_16_1600.out
> deleting
While I haven't used rastream (let alone run scripts from it), can
you tell what is zombieing? My first guess (although I'm not an expert on
zombies either) would be that your script isn't returning a result to
rastream which is I think what causes a zombie. It would be instructive to
know what the original pid of one of the zombies is (as the pids are showing
in the argus debug output, although I don't seem to have left one). That
should point to what task is causing the zombie.
Peter Van Epp
10 Dec 17:44
Re: Rasplit compression support
Carter Bullard <carter <at> qosient.com>
2009-12-10 16:44:51 GMT
2009-12-10 16:44:51 GMT
Hey Matt,
If you have data files that can't be read, if you can upload the file to
ftp://qosient.com/incoming (which is a blind repository) I can debug
the file.
It is possible that grabbing the latest copy of argus-clients-3.0.2.tar.gz (which
is frozen now)can solve the problem.
Carter
On Dec 9, 2009, at 11:43 AM, Matt Sheridan wrote:
Hi Carter -
I was wondering what your thoughts might be on this moving forward? The defunct processes continue to pile up.
Not sure if its related, but if I run a basic ra on a file, I am getting some other process complaints:
ra -r ./argus.2009_12_09_1100.out.gz - host 10.10.10.10
ra[4935]: 11:42:56.648522 ArgusReadStreamSocket (0x41fc) record length is zero
gzip: stdout: Broken pipeOn Tue, Nov 17, 2009 at 10:51 AM, Matt Sheridan <mattmail5050 <at> gmail.com> wrote:
I have an update. It seems that what is becoming a zombie is the script called by the "-f" option (you may have already figured that out, I just did ;)"
I attempted to use the default contrib script included in the clients package, and it also zombied.
But as you can see:
root 1716 31417 8 10:46 pts/1 00:00:10 rastream -S localhost:561 -M time 1m -B 10s -f /opt/IDS/argus/etc/rastream-orig.sh -
root 2057 1716 0 10:47 pts/1 00:00:00 [rastream-orig.s] <defunct>
it was the rastream-orig.sh script, called by the -f option (which is used to zip up the files) that zombied. So it is possible that the problem is not within rastream, but in the way the called script is handled?On Mon, Nov 16, 2009 at 9:32 PM, Peter Van Epp <vanepp <at> sfu.ca> wrote:<snip>On Mon, Nov 16, 2009 at 04:38:38PM -0500, Matt Sheridan wrote:
> I was able to get it figured out. I admit not being fully versed in my
> compile skills :). Adding the .debug to the compile directory was the
> clarification that I missed.
>
> Here are the results from stdout. Zombie processes were created when looking
> at ps:
>
> rastream -S localhost:561 -M time 10m -B 10s -f
> /opt/IDS/argus/etc/rastream.sh -w
> /opt/IDS/argus/log/\$srcid/argus.%Y_%m_%d_%H%M.out -D1
> rastream[14067.a08e7288f12a0000]: 16:03:10.090959 main: reading files
> completed
> rastream[14067.4019a54100000000]: 16:03:10.091388 Trying 127.0.0.1 port 561
> Expecting Argus records
> rastream[14067.4019a54100000000]: 16:03:10.091652 connected
> rastream[14067.4019a54100000000]: 16:03:10.091768 ArgusGetServerSocket
> (0x87df0010) returning 3
> rastream[14067.a08e7288f12a0000]: 16:10:15.474102 ArgusRunScript(0x87d41010,
> 21198c10) filename /opt/IDS/argus/log/127.0.0.1/argus.2009_11_16_1600.out
> rastream[14067.a08e7288f12a0000]: 16:10:15.474201 ArgusRunScript(0x87d41010,
> 0x21198c10) scheduling /opt/IDS/argus/etc/rastream.sh -r
> /opt/IDS/argus/log/127.0.0.1/argus.2009_11_16_1600.out
> rastream[14067.a08e7288f12a0000]: 16:10:15.474240 ArgusRunScript(0x87d41010,
> 21198c10) returning /opt/IDS/argus/etc/rastream.sh -r /opt/IDS/argus/log/
> 127.0.0.1/argus.2009_11_16_1600.out
> rastream[16367.a08e7288f12a0000]: 16:10:16.074641 ArgusRunScript calling
> /opt/IDS/argus/etc/rastream.sh -r /opt/IDS/argus/log/
> 127.0.0.1/argus.2009_11_16_1600.out
> deleting
While I haven't used rastream (let alone run scripts from it), can
you tell what is zombieing? My first guess (although I'm not an expert on
zombies either) would be that your script isn't returning a result to
rastream which is I think what causes a zombie. It would be instructive to
know what the original pid of one of the zombies is (as the pids are showing
in the argus debug output, although I don't seem to have left one). That
should point to what task is causing the zombie.
Peter Van Epp
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
10 Dec 18:25
Re: Question about states
Carter Bullard <carter <at> qosient.com>
2009-12-10 17:25:40 GMT
2009-12-10 17:25:40 GMT
Hey Matt, Found the problem. Seems that you have a packet that is reporting a window size of 32M+ bytes, which is bigger than I expected. My bad. Here is a patch, and I'll include this patch in the official argus-clients-3.0.2 release package which is now officially frozen. Give this patch a try, or download the newest argus-clients-3.0.2.tar.gz from the website. http://qosient.com/argus. Sorry for the inconvenience, Carter On Dec 1, 2009, at 2:31 PM, Matt Brewer wrote: > Hi Carter, > > The packets that cause the overflow are available online. http://www.ddtek.biz/ctf_dc17_packets.tbz.torrent > > If you convert the ctf_dc17.pcap000 to an Argus file then attempt to read it with RA with at least adding swin and dwin fields it will crash. I followed the instructions provided by Peter and nothing changed the output. > > =========================== > | Matt Brewer > | CCNA > | www.sheridantutorials.com > =========================== > > ----- Original Message ----- > From: carter <at> qosient.com > To: "Matt Brewer" <hilather <at> gmail.com> > Sent: Tuesday, December 1, 2009 11:46:52 AM GMT -05:00 US/Canada Eastern > Subject: Re: [ARGUS] Question about states > > Hey Matt, > Need the packet file that is causing problems. > Carter > Sent from my Verizon Wireless BlackBerry > > -----Original Message----- > From: Matt Brewer <hilather <at> gmail.com> > Date: Fri, 27 Nov 2009 19:49:55 -0500 (GMT-05:00) > To: Carter Bullard<carter <at> qosient.com> > Subject: Re: [ARGUS] Question about states > > Hi Carter, > > Sorry for taking so long to reply, I've had the flu and been out of action for the last few days. > > I really like the way the -z flag works, translating the state changes to the TCP flag equivalent. The only thing I would change would be to have lower case letters represent the source flags and uppercase represent destination flags. > > Heres an example. A TCP conversation: > Syn -> > <- Synack > Ack -> > <-Fin > Rst -> > > Would be represented as sSFr > > If its not too much to ask could you work this into a command line switch? > > Also, Monday I'll be in the office again and should be able to perform the necessary debugging for the buffer overflow. > > =========================== > | Matt Brewer > | CCNA > | www.sheridantutorials.com > =========================== > > ----- Original Message ----- > From: "Carter Bullard" <carter <at> qosient.com> > To: "Matt Brewer" <hilather <at> gmail.com> > Cc: "argus-info" <argus-info <at> lists.andrew.cmu.edu> > Sent: Tuesday, November 24, 2009 11:23:31 AM GMT -05:00 US/Canada Eastern > Subject: Re: [ARGUS] Question about states > > Hey Matt, > In most cases, the only reason a field can't be printed the way you want, > is because we don't have a command line name of letter to apply the > option, usually because we didn't get around to doing it, or nobody asked > for it. > > OK, understand that the default status fields are simply a convenient way of > presenting the TCP state progression in a simple fashion. We have more > information in the Argus records, as we track the perceived TCP states per > side of the connection, but how would you like to print them? > > In raxml(), which is now obsolete (replaced with the '-M xml' option), we > printed out the entire contents of the ARGUS_TCP_DSR. That was argus-2.x. > In argus-3.0, we have more and less TCP information available, depending > on how you setup your argus(). So if you want some new TCP printing support, > we should discuss what the information is, what it means, and how to print it. > > Currently, the combined TCP status field (default TCP output with the -z option), > tracks the progression of the TCP state machine, and the direction of the TCP control > packets that establish the various TCP states is indicated by the assignment of Src > and Dst for the flow. Who sent the SYN ('s') ? The Source. Who sent the > SYN_ACK ('S')? The Destination. In fact that is how the Source and Destination > are determined, and in some cases of TCP mediated scans, the direction seems > confused, because the scanner is violating the TCP state machine to get the > targets to respond. Notice that the use of lower case and upper case tracks the > source and destination. > > The states for TCP are derived from the indications that Argus stores in the TCP > DSR. We capture "SAW_SYN", "SAW_SYN_ACK", "ESTABLISHED", "SAW_FIN", > "SAW_FIN_ACK", "SAW_RESET" on each side of the connection, as well > as the state that argus() thinks the TCP connection is in. And with each status > report, we zero most of these bits, so we can know what control packets were sent, > when. > > So, with all of that, we can print out all that you ask, just need to know what syntax > to use to print the fields. > > The flags format, because its simply the OR'ing of the TCP flags during > the status interval, you lose massive amounts of information, like direction > etc..... However, when you print the status field using the "-Z" option, you have > the flags for the source, the an '_' and then the flags for the destination. > > Maybe you haven't seen this because your default status field width isn't large > enough to print the whole indicator? > > "man" records are argus management records. They convey status of the argus > data source, which can be argus() or radium(), and act as keep alives for the > argus transport stream. I believe that they are described in the man pages. > > Carter > > On Nov 21, 2009, at 10:53 PM, Matt Brewer wrote: > >> Hello, >> >> I have a few questions and ideas I'd like to share. >> >> Some of the flows in my captures are marked EC (This is with the -z option to show mimic tcp states) and I haven't been able to determine what the C stands for, any idea what it means? >> >> Also, I've been working with the states field a lot with the -z option, and its come to my attention that the sSEfFR orientation of the flags doesn't actually give me indication of who sent what packets with what tcp flags. I'm aware I can remove the -z and use the -Zs or -Zd option and it will give me an output of with tcp flags were set for each direction, but it would be nice to actually have both options displayed. If I use the -Zb option, it looks like it attempts to print out both states deliminated by an underscore, however the field appears to be truncated to 5 characters and is often cut off. And even if it wasn't cut off, the tcp flags aren't displayed it order of occurrence. >> >> Is there any reason the state field cannot keep the occurrence of flags in order? It would be incredibly useful to display the state of the flow in order. Ideally I would like to see the tcp flags as lower case letters as the source produced flags, and upper case letters as destination produced flags. A system like this would allow us to easily determine which side of the flow tore down the TCP session, either via a FIN or RST. >> >> Also, I've recently seen the "man" protocol in my captures. What exactly is this supposed to indicate? Some of the "man" flows are marked with STP or STA in the state field. I am assuming that "man" refers to switch management protocols? Correct me if I'm wrong. >> >> Also, the "man" protocol flows with the state "STA" have zero destination or source packets. Which is very bizarre, I do not understand how a flow could exist without packets. >> >> Oh also, what is the "offset" field for? The man pages are rather vague "record byte offset in file or stream." >> >> Thanks in advance, argus is a wonderful tool. >> >> =========================== >> | Matt Brewer >> | CCNA >> | www.sheridantutorials.com >> =========================== >> > > > > > > > Carter Bullard CEO/President QoSient, LLC 150 E 57th Street Suite 12D New York, New York 10022 +1 212 588-9133 Phone +1 212 588-9134 Fax
11 Dec 01:19
Rastream call script zombie
CS Lee <geek00l <at> gmail.com>
2009-12-11 00:19:00 GMT
2009-12-11 00:19:00 GMT
hi matt,
May i know which OS platform are you on where you have zombie process for rastream call script, I remember I have it on FreeBSD but not on linux, therefore please do let me know further.
Cheers
--
Best Regards,
CS Lee<geek00L[at]gmail.com>
http://geek00l.blogspot.com
http://defcraft.net
), can
RSS Feed