headers
Nick Diel | 8 Apr 01:40

Issues with -t option

I am trying to use the -t option and am not having much luck.  I want to be able to specify a range down to the seconds, but I can't get a filter working more specific than a month.  Also is it possible to specify time in UTC format for the -t option?


Regular output (I removed IP info), note this file is from last year.
[diel <at> lander-nic ~]$ ra -r argus.out
     2007/11/07-09:01:26.022023  e         *****
     2007/11/07-09:01:26.024221  e         *****
     2007/11/07-09:01:26.028978  e         *****
     2007/11/07-09:01:26.034025  e         *****
     2007/11/07-09:01:26.044853  e         *****


Non working time specifications.
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m7d
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11/07


Working time specifications
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11
     2007/11/07-09:01:26.022023  e         *****
     2007/11/07-09:01:26.024221  e         *****
     2007/11/07-09:01:26.028978  e         *****
     2007/11/07-09:01:26.034025  e         *****
     2007/11/07-09:01:26.044853  e         *****
[diel <at> lander-nic ~]$ ra -r argus.out -t ****/11
     2007/11/07-09:01:26.022023  e         *****
     2007/11/07-09:01:26.024221  e         *****
     2007/11/07-09:01:26.028978  e         *****
     2007/11/07-09:01:26.034025  e         *****
     2007/11/07-09:01:26.044853  e         *****


Any thoughts?
Nick Diel

Permalink | Reply | 4 comments |
headers
Carter Bullard | 8 Apr 02:40

Re: Issues with -t option

Hey Nick,
Sounds like a bug.  What happens when you specify a valid range?
   "-t 12-13"
   "-t 06.12-06.13"

if you run it with a "-D 4" option, it should tell you what the unix
time range the time filter generates.

I think for month you need a 'M'?

Carter

anubis:common carter$ ra -D5 -t 12-13
ra[49162.a0052074]: 2008/04/07.20:37:21.768040 ArgusParseTime  
(0x32f000, 0x32f070, 0xa0053cc8,12,  ) retn 4: 1207584000
ra[49162.a0052074]: 2008/04/07.20:37:21.768128 ArgusParseTime  
(0x32f000, 0x32f09c, 0x32f070,13, -) retn 4: 1207587600
ra[49162.a0052074]: 2008/04/07.20:37:21.768176 ArgusCheckTimeFormat  
(0xa0053cc8, 12-13) retn 0: 1207584000-1207587600
ra[49162.a0052074]: 2008/04/07.20:37:21.768200 ArgusParseTimeArg  
(12-13, 4, 0xa0053cc8)
ra[49162.a0052074]: 2008/04/07.20:37:21.768291 ArgusAddFileList  
(0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 12
ra[49163.a0052074]: 2008/04/07.20:37:25.920039 ArgusParseTime  
(0x32f000, 0x32f070, 0x32f09c,12,  ) retn 4: 1207584000
ra[49163.a0052074]: 2008/04/07.20:37:25.920111 ArgusCheckTimeFormat  
(0xa0053cc8, 12) retn 0: 1207584000-1207587600
ra[49163.a0052074]: 2008/04/07.20:37:25.920135 ArgusParseTimeArg (12,  
4, 0xa0053cc8)
ra[49163.a0052074]: 2008/04/07.20:37:25.920184 ArgusAddFileList  
(0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 12m
ra[49180.a0052074]: 2008/04/07.20:39:21.244726 ArgusParseTime  
(0x32f000, 0x32f070, 0x32f09c,,  ) retn 5: 1207615161
ra[49180.a0052074]: 2008/04/07.20:39:21.244880 ArgusCheckTimeFormat  
(0xa0053cc8, 12m) retn 0: 1207613520-1207613580
ra[49180.a0052074]: 2008/04/07.20:39:21.244923 ArgusParseTimeArg (12m,  
4, 0xa0053cc8)
ra[49180.a0052074]: 2008/04/07.20:39:21.245022 ArgusAddFileList  
(0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 06.12-06.13
ra[49143.a0052074]: 2008/04/07.20:35:56.829484 ArgusParseTime  
(0x32f000, 0x32f070, 0xa0053cc8,06,  ) retn 4: 1207497600
ra[49143.a0052074]: 2008/04/07.20:35:56.829669 ArgusParseTime  
(0x32f000, 0x32f09c, 0x32f070,06, -) retn 4: 1207501200
ra[49143.a0052074]: 2008/04/07.20:35:56.829766 ArgusCheckTimeFormat  
(0xa0053cc8, 06.12-06.13) retn 0: 1207497600-1207501200
ra[49143.a0052074]: 2008/04/07.20:35:56.829808 ArgusParseTimeArg  
(06.12-06.13, 4, 0xa0053cc8)

anubis:cmmon carter$ ra -D5 -t 2008/02/06.12-06.13
ra[49151.a0052074]: 2008/04/07.20:36:40.358480 ArgusParseTime  
(0x32f000, 0x32f070, 0xa0053cc8,2008,  ) retn 4: 1202317200
ra[49151.a0052074]: 2008/04/07.20:36:40.358661 ArgusParseTime  
(0x32f000, 0x32f09c, 0x32f070,06, -) retn 4: 1202320800
ra[49151.a0052074]: 2008/04/07.20:36:40.358756 ArgusCheckTimeFormat  
(0xa0053cc8, 2008/02/06.12-06.13) retn 0: 1202317200-1202320800
ra[49151.a0052074]: 2008/04/07.20:36:40.358798 ArgusParseTimeArg  
(2008/02/06.12-06.13, 4, 0xa0053cc8)

On Apr 7, 2008, at 7:40 PM, Nick Diel wrote:

> I am trying to use the -t option and am not having much luck.  I  
> want to be able to specify a range down to the seconds, but I can't  
> get a filter working more specific than a month.  Also is it  
> possible to specify time in UTC format for the -t option?
>
>
> Regular output (I removed IP info), note this file is from last year.
> [diel <at> lander-nic ~]$ ra -r argus.out
>      2007/11/07-09:01:26.022023  e         *****
>      2007/11/07-09:01:26.024221  e         *****
>      2007/11/07-09:01:26.028978  e         *****
>      2007/11/07-09:01:26.034025  e         *****
>      2007/11/07-09:01:26.044853  e         *****
>
>
> Non working time specifications.
> [diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m7d
> [diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m
> [diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11/07
>
>
> Working time specifications
> [diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11
>      2007/11/07-09:01:26.022023  e         *****
>      2007/11/07-09:01:26.024221  e         *****
>      2007/11/07-09:01:26.028978  e         *****
>      2007/11/07-09:01:26.034025  e         *****
>      2007/11/07-09:01:26.044853  e         *****
> [diel <at> lander-nic ~]$ ra -r argus.out -t ****/11
>      2007/11/07-09:01:26.022023  e         *****
>      2007/11/07-09:01:26.024221  e         *****
>      2007/11/07-09:01:26.028978  e         *****
>      2007/11/07-09:01:26.034025  e         *****
>      2007/11/07-09:01:26.044853  e         *****
>
>
> Any thoughts?
> Nick Diel
Permalink | Reply | 3 comments |
headers
Nick Diel | 8 Apr 18:33

Re: Issues with -t option

Carter,

The capital M is what I needed for some of the time filters, at the end of the day yesterday I was just coping what was in the man page (lower case m).

The filter 2007/11/07 produces a filter that is a day behind:
[diel <at> lander-nic ~]$ ra -D5 -t 2007/11/07
ra[3166.c046fcb7]: 10:25:13.134115 ArgusParseTime (0xb7f82008, 0xb7f82078, 0xb7f820a4,2007,  ) retn 3: 1194332400
ra[3166.c046fcb7]: 10:25:13.134229 ArgusCheckTimeFormat (0xc8b2e0, 2007/11/07) retn 0: 1194332400-1194418800
ra[3166.c046fcb7]: 10:25:13.134271 ArgusParseTimeArg (2007/11/07, 4, 0xc8b2e0)
ra[3166.c046fcb7]: 10:25:13.134339 ArgusAddFileList (0xb7f82008, -, 1, -1, -1) returning -1


The filter 2007y11M7d is close, but an hour off (probably due to daylight savings)
[diel <at> lander-nic ~]$ ra -D5 -t 2007y11M7d
ra[3159.c056f4b7]: 10:20:54.380457 ArgusParseTime (0xb7f03008, 0xb7f03078, 0xb7f030a4,,  ) retn 3: 1207671654
ra[3159.c056f4b7]: 10:20:54.380566 ArgusCheckTimeFormat (0xc8b2e0, 2007y11M7d) retn 0: 1194415200-1194501600
ra[3159.c056f4b7]: 10:20:54.380614 ArgusParseTimeArg (2007y11M7d, 4, 0xc8b2e0)
ra[3159.c056f4b7]: 10:20:54.380663 ArgusAddFileList (0xb7f03008, -, 1, -1, -1) returning -1

It used MST(-6 GMT) which is correct for right now, but not for November should be -7 GMT.

More specific filters seems to be working great.  Would love for a way to input UTC timestamps or even just a raw time range UTC-UTC, didn't see any indication this was currently possible.

Thanks for the input!

Nick

On Mon, Apr 7, 2008 at 6:40 PM, Carter Bullard <carter <at> qosient.com> wrote:
Hey Nick,
Sounds like a bug.  What happens when you specify a valid range?
 "-t 12-13"
 "-t 06.12-06.13"

if you run it with a "-D 4" option, it should tell you what the unix
time range the time filter generates.

I think for month you need a 'M'?

Carter


anubis:common carter$ ra -D5 -t 12-13
ra[49162.a0052074]: 2008/04/07.20:37:21.768040 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,12,  ) retn 4: 1207584000
ra[49162.a0052074]: 2008/04/07.20:37:21.768128 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,13, -) retn 4: 1207587600
ra[49162.a0052074]: 2008/04/07.20:37:21.768176 ArgusCheckTimeFormat (0xa0053cc8, 12-13) retn 0: 1207584000-1207587600
ra[49162.a0052074]: 2008/04/07.20:37:21.768200 ArgusParseTimeArg (12-13, 4, 0xa0053cc8)
ra[49162.a0052074]: 2008/04/07.20:37:21.768291 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 12
ra[49163.a0052074]: 2008/04/07.20:37:25.920039 ArgusParseTime (0x32f000, 0x32f070, 0x32f09c,12,  ) retn 4: 1207584000
ra[49163.a0052074]: 2008/04/07.20:37:25.920111 ArgusCheckTimeFormat (0xa0053cc8, 12) retn 0: 1207584000-1207587600
ra[49163.a0052074]: 2008/04/07.20:37:25.920135 ArgusParseTimeArg (12, 4, 0xa0053cc8)
ra[49163.a0052074]: 2008/04/07.20:37:25.920184 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 12m
ra[49180.a0052074]: 2008/04/07.20:39:21.244726 ArgusParseTime (0x32f000, 0x32f070, 0x32f09c,,  ) retn 5: 1207615161
ra[49180.a0052074]: 2008/04/07.20:39:21.244880 ArgusCheckTimeFormat (0xa0053cc8, 12m) retn 0: 1207613520-1207613580
ra[49180.a0052074]: 2008/04/07.20:39:21.244923 ArgusParseTimeArg (12m, 4, 0xa0053cc8)
ra[49180.a0052074]: 2008/04/07.20:39:21.245022 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 06.12-06.13
ra[49143.a0052074]: 2008/04/07.20:35:56.829484 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,06,  ) retn 4: 1207497600
ra[49143.a0052074]: 2008/04/07.20:35:56.829669 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,06, -) retn 4: 1207501200
ra[49143.a0052074]: 2008/04/07.20:35:56.829766 ArgusCheckTimeFormat (0xa0053cc8, 06.12-06.13) retn 0: 1207497600-1207501200
ra[49143.a0052074]: 2008/04/07.20:35:56.829808 ArgusParseTimeArg (06.12-06.13, 4, 0xa0053cc8)

anubis:cmmon carter$ ra -D5 -t 2008/02/06.12-06.13
ra[49151.a0052074]: 2008/04/07.20:36:40.358480 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,2008,  ) retn 4: 1202317200
ra[49151.a0052074]: 2008/04/07.20:36:40.358661 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,06, -) retn 4: 1202320800
ra[49151.a0052074]: 2008/04/07.20:36:40.358756 ArgusCheckTimeFormat (0xa0053cc8, 2008/02/06.12-06.13) retn 0: 1202317200-1202320800
ra[49151.a0052074]: 2008/04/07.20:36:40.358798 ArgusParseTimeArg (2008/02/06.12-06.13, 4, 0xa0053cc8)



On Apr 7, 2008, at 7:40 PM, Nick Diel wrote:

I am trying to use the -t option and am not having much luck.  I want to be able to specify a range down to the seconds, but I can't get a filter working more specific than a month.  Also is it possible to specify time in UTC format for the -t option?


Regular output (I removed IP info), note this file is from last year.
[diel <at> lander-nic ~]$ ra -r argus.out
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****


Non working time specifications.
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m7d
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11/07


Working time specifications
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****
[diel <at> lander-nic ~]$ ra -r argus.out -t ****/11
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****


Any thoughts?
Nick Diel


Permalink | Reply | 2 comments |
headers
Carter Bullard | 8 Apr 19:21

Re: Issues with -t option

Hmmmm, yes I get the same error you do for the 2007/11/07.
I'll investigate and I'll add using UTC to the command line.

Carter


On Apr 8, 2008, at 12:33 PM, Nick Diel wrote:

Carter,

The capital M is what I needed for some of the time filters, at the end of the day yesterday I was just coping what was in the man page (lower case m).

The filter 2007/11/07 produces a filter that is a day behind:
[diel <at> lander-nic ~]$ ra -D5 -t 2007/11/07
ra[3166.c046fcb7]: 10:25:13.134115 ArgusParseTime (0xb7f82008, 0xb7f82078, 0xb7f820a4,2007,  ) retn 3: 1194332400
ra[3166.c046fcb7]: 10:25:13.134229 ArgusCheckTimeFormat (0xc8b2e0, 2007/11/07) retn 0: 1194332400-1194418800
ra[3166.c046fcb7]: 10:25:13.134271 ArgusParseTimeArg (2007/11/07, 4, 0xc8b2e0)
ra[3166.c046fcb7]: 10:25:13.134339 ArgusAddFileList (0xb7f82008, -, 1, -1, -1) returning -1


The filter 2007y11M7d is close, but an hour off (probably due to daylight savings)
[diel <at> lander-nic ~]$ ra -D5 -t 2007y11M7d
ra[3159.c056f4b7]: 10:20:54.380457 ArgusParseTime (0xb7f03008, 0xb7f03078, 0xb7f030a4,,  ) retn 3: 1207671654
ra[3159.c056f4b7]: 10:20:54.380566 ArgusCheckTimeFormat (0xc8b2e0, 2007y11M7d) retn 0: 1194415200-1194501600
ra[3159.c056f4b7]: 10:20:54.380614 ArgusParseTimeArg (2007y11M7d, 4, 0xc8b2e0)
ra[3159.c056f4b7]: 10:20:54.380663 ArgusAddFileList (0xb7f03008, -, 1, -1, -1) returning -1

It used MST(-6 GMT) which is correct for right now, but not for November should be -7 GMT.

More specific filters seems to be working great.  Would love for a way to input UTC timestamps or even just a raw time range UTC-UTC, didn't see any indication this was currently possible.

Thanks for the input!

Nick

On Mon, Apr 7, 2008 at 6:40 PM, Carter Bullard <carter <at> qosient.com> wrote:
Hey Nick,
Sounds like a bug.  What happens when you specify a valid range?
 "-t 12-13"
 "-t 06.12-06.13"

if you run it with a "-D 4" option, it should tell you what the unix
time range the time filter generates.

I think for month you need a 'M'?

Carter


anubis:common carter$ ra -D5 -t 12-13
ra[49162.a0052074]: 2008/04/07.20:37:21.768040 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,12,  ) retn 4: 1207584000
ra[49162.a0052074]: 2008/04/07.20:37:21.768128 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,13, -) retn 4: 1207587600
ra[49162.a0052074]: 2008/04/07.20:37:21.768176 ArgusCheckTimeFormat (0xa0053cc8, 12-13) retn 0: 1207584000-1207587600
ra[49162.a0052074]: 2008/04/07.20:37:21.768200 ArgusParseTimeArg (12-13, 4, 0xa0053cc8)
ra[49162.a0052074]: 2008/04/07.20:37:21.768291 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 12
ra[49163.a0052074]: 2008/04/07.20:37:25.920039 ArgusParseTime (0x32f000, 0x32f070, 0x32f09c,12,  ) retn 4: 1207584000
ra[49163.a0052074]: 2008/04/07.20:37:25.920111 ArgusCheckTimeFormat (0xa0053cc8, 12) retn 0: 1207584000-1207587600
ra[49163.a0052074]: 2008/04/07.20:37:25.920135 ArgusParseTimeArg (12, 4, 0xa0053cc8)
ra[49163.a0052074]: 2008/04/07.20:37:25.920184 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 12m
ra[49180.a0052074]: 2008/04/07.20:39:21.244726 ArgusParseTime (0x32f000, 0x32f070, 0x32f09c,,  ) retn 5: 1207615161
ra[49180.a0052074]: 2008/04/07.20:39:21.244880 ArgusCheckTimeFormat (0xa0053cc8, 12m) retn 0: 1207613520-1207613580
ra[49180.a0052074]: 2008/04/07.20:39:21.244923 ArgusParseTimeArg (12m, 4, 0xa0053cc8)
ra[49180.a0052074]: 2008/04/07.20:39:21.245022 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 06.12-06.13
ra[49143.a0052074]: 2008/04/07.20:35:56.829484 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,06,  ) retn 4: 1207497600
ra[49143.a0052074]: 2008/04/07.20:35:56.829669 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,06, -) retn 4: 1207501200
ra[49143.a0052074]: 2008/04/07.20:35:56.829766 ArgusCheckTimeFormat (0xa0053cc8, 06.12-06.13) retn 0: 1207497600-1207501200
ra[49143.a0052074]: 2008/04/07.20:35:56.829808 ArgusParseTimeArg (06.12-06.13, 4, 0xa0053cc8)

anubis:cmmon carter$ ra -D5 -t 2008/02/06.12-06.13
ra[49151.a0052074]: 2008/04/07.20:36:40.358480 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,2008,  ) retn 4: 1202317200
ra[49151.a0052074]: 2008/04/07.20:36:40.358661 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,06, -) retn 4: 1202320800
ra[49151.a0052074]: 2008/04/07.20:36:40.358756 ArgusCheckTimeFormat (0xa0053cc8, 2008/02/06.12-06.13) retn 0: 1202317200-1202320800
ra[49151.a0052074]: 2008/04/07.20:36:40.358798 ArgusParseTimeArg (2008/02/06.12-06.13, 4, 0xa0053cc8)



On Apr 7, 2008, at 7:40 PM, Nick Diel wrote:

I am trying to use the -t option and am not having much luck.  I want to be able to specify a range down to the seconds, but I can't get a filter working more specific than a month.  Also is it possible to specify time in UTC format for the -t option?


Regular output (I removed IP info), note this file is from last year.
[diel <at> lander-nic ~]$ ra -r argus.out
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****


Non working time specifications.
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m7d
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11/07


Working time specifications
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****
[diel <at> lander-nic ~]$ ra -r argus.out -t ****/11
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****


Any thoughts?
Nick Diel



Permalink | Reply | 1 comment |
headers
Nick Diel | 8 Apr 20:23

Flow Counting

I am interested in counting number of flows for things such as source address, IP pairs, time intervals, etc.  This is closely related to Stéphane Peters email: "Counting flows by time interval in argus."

After using racluster to merge status flow records, you have a file where a record represents a flow.  You can of course use racount with a filter to tell you some of this information, but it seem quite impractical to do things such as find the IP pair generating the most flows.  You could also use a set of pipes or scripts (as Stéphane Peters showed us), but this would require modification every time you wanted something slightly different.

Maybe there is a way using ra tools to do this already and I am missing it.  If not maybe we can get racluster (when specified) or another tool to "zero out"/set the trans count to 1 after we have merged status flow records.  This way things such as racluster -r megredRecords.argus -m saddr -s +trans will now list how many flows for each source address.

I am curious what the group has to say about this.  Is something like this already possible?  Is this information useful to other people than me?  Would modifying the trans column present too much ambiguity, i.e. "under these circumstances the trans column can represent number of flows else it represents aggregated record count"?


Nick

Permalink | Reply | 3 comments |
headers
Carter Bullard | 8 Apr 20:59

Re: Issues with -t option

Hey Nick,
So I figured out the problem.  When we push the time filter into a time zone
that is different than the current one (EST -> EDT), I was not correcting all
the fields that needed to be corrected.  In the example 2007/11/07, if you
are in EDT and convert to this date, the returned value is 2007/11/06.23:00:00.
I was not adjusting for a possible change of day, in this case, so we got a
day behind.

I've also added UTC as input for time.  I'll have it upload later today.

Carter
On Apr 8, 2008, at 1:21 PM, Carter Bullard wrote:

Hmmmm, yes I get the same error you do for the 2007/11/07.
I'll investigate and I'll add using UTC to the command line.

Carter


On Apr 8, 2008, at 12:33 PM, Nick Diel wrote:
Carter,

The capital M is what I needed for some of the time filters, at the end of the day yesterday I was just coping what was in the man page (lower case m).

The filter 2007/11/07 produces a filter that is a day behind:
[diel <at> lander-nic ~]$ ra -D5 -t 2007/11/07
ra[3166.c046fcb7]: 10:25:13.134115 ArgusParseTime (0xb7f82008, 0xb7f82078, 0xb7f820a4,2007,  ) retn 3: 1194332400
ra[3166.c046fcb7]: 10:25:13.134229 ArgusCheckTimeFormat (0xc8b2e0, 2007/11/07) retn 0: 1194332400-1194418800
ra[3166.c046fcb7]: 10:25:13.134271 ArgusParseTimeArg (2007/11/07, 4, 0xc8b2e0)
ra[3166.c046fcb7]: 10:25:13.134339 ArgusAddFileList (0xb7f82008, -, 1, -1, -1) returning -1


The filter 2007y11M7d is close, but an hour off (probably due to daylight savings)
[diel <at> lander-nic ~]$ ra -D5 -t 2007y11M7d
ra[3159.c056f4b7]: 10:20:54.380457 ArgusParseTime (0xb7f03008, 0xb7f03078, 0xb7f030a4,,  ) retn 3: 1207671654
ra[3159.c056f4b7]: 10:20:54.380566 ArgusCheckTimeFormat (0xc8b2e0, 2007y11M7d) retn 0: 1194415200-1194501600
ra[3159.c056f4b7]: 10:20:54.380614 ArgusParseTimeArg (2007y11M7d, 4, 0xc8b2e0)
ra[3159.c056f4b7]: 10:20:54.380663 ArgusAddFileList (0xb7f03008, -, 1, -1, -1) returning -1

It used MST(-6 GMT) which is correct for right now, but not for November should be -7 GMT.

More specific filters seems to be working great.  Would love for a way to input UTC timestamps or even just a raw time range UTC-UTC, didn't see any indication this was currently possible.

Thanks for the input!

Nick

On Mon, Apr 7, 2008 at 6:40 PM, Carter Bullard <carter <at> qosient.com> wrote:
Hey Nick,
Sounds like a bug.  What happens when you specify a valid range?
 "-t 12-13"
 "-t 06.12-06.13"

if you run it with a "-D 4" option, it should tell you what the unix
time range the time filter generates.

I think for month you need a 'M'?

Carter


anubis:common carter$ ra -D5 -t 12-13
ra[49162.a0052074]: 2008/04/07.20:37:21.768040 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,12,  ) retn 4: 1207584000
ra[49162.a0052074]: 2008/04/07.20:37:21.768128 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,13, -) retn 4: 1207587600
ra[49162.a0052074]: 2008/04/07.20:37:21.768176 ArgusCheckTimeFormat (0xa0053cc8, 12-13) retn 0: 1207584000-1207587600
ra[49162.a0052074]: 2008/04/07.20:37:21.768200 ArgusParseTimeArg (12-13, 4, 0xa0053cc8)
ra[49162.a0052074]: 2008/04/07.20:37:21.768291 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 12
ra[49163.a0052074]: 2008/04/07.20:37:25.920039 ArgusParseTime (0x32f000, 0x32f070, 0x32f09c,12,  ) retn 4: 1207584000
ra[49163.a0052074]: 2008/04/07.20:37:25.920111 ArgusCheckTimeFormat (0xa0053cc8, 12) retn 0: 1207584000-1207587600
ra[49163.a0052074]: 2008/04/07.20:37:25.920135 ArgusParseTimeArg (12, 4, 0xa0053cc8)
ra[49163.a0052074]: 2008/04/07.20:37:25.920184 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 12m
ra[49180.a0052074]: 2008/04/07.20:39:21.244726 ArgusParseTime (0x32f000, 0x32f070, 0x32f09c,,  ) retn 5: 1207615161
ra[49180.a0052074]: 2008/04/07.20:39:21.244880 ArgusCheckTimeFormat (0xa0053cc8, 12m) retn 0: 1207613520-1207613580
ra[49180.a0052074]: 2008/04/07.20:39:21.244923 ArgusParseTimeArg (12m, 4, 0xa0053cc8)
ra[49180.a0052074]: 2008/04/07.20:39:21.245022 ArgusAddFileList (0x32f000, -, 1, -1, -1) returning -1

anubis:common carter$ ra -D5 -t 06.12-06.13
ra[49143.a0052074]: 2008/04/07.20:35:56.829484 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,06,  ) retn 4: 1207497600
ra[49143.a0052074]: 2008/04/07.20:35:56.829669 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,06, -) retn 4: 1207501200
ra[49143.a0052074]: 2008/04/07.20:35:56.829766 ArgusCheckTimeFormat (0xa0053cc8, 06.12-06.13) retn 0: 1207497600-1207501200
ra[49143.a0052074]: 2008/04/07.20:35:56.829808 ArgusParseTimeArg (06.12-06.13, 4, 0xa0053cc8)

anubis:cmmon carter$ ra -D5 -t 2008/02/06.12-06.13
ra[49151.a0052074]: 2008/04/07.20:36:40.358480 ArgusParseTime (0x32f000, 0x32f070, 0xa0053cc8,2008,  ) retn 4: 1202317200
ra[49151.a0052074]: 2008/04/07.20:36:40.358661 ArgusParseTime (0x32f000, 0x32f09c, 0x32f070,06, -) retn 4: 1202320800
ra[49151.a0052074]: 2008/04/07.20:36:40.358756 ArgusCheckTimeFormat (0xa0053cc8, 2008/02/06.12-06.13) retn 0: 1202317200-1202320800
ra[49151.a0052074]: 2008/04/07.20:36:40.358798 ArgusParseTimeArg (2008/02/06.12-06.13, 4, 0xa0053cc8)



On Apr 7, 2008, at 7:40 PM, Nick Diel wrote:

I am trying to use the -t option and am not having much luck.  I want to be able to specify a range down to the seconds, but I can't get a filter working more specific than a month.  Also is it possible to specify time in UTC format for the -t option?


Regular output (I removed IP info), note this file is from last year.
[diel <at> lander-nic ~]$ ra -r argus.out
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****


Non working time specifications.
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m7d
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007y11m
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11/07


Working time specifications
[diel <at> lander-nic ~]$ ra -r argus.out -t 2007/11
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****
[diel <at> lander-nic ~]$ ra -r argus.out -t ****/11
    2007/11/07-09:01:26.022023  e         *****
    2007/11/07-09:01:26.024221  e         *****
    2007/11/07-09:01:26.028978  e         *****
    2007/11/07-09:01:26.034025  e         *****
    2007/11/07-09:01:26.044853  e         *****


Any thoughts?
Nick Diel




Permalink | Reply | 0 comments |
headers
Carter Bullard | 8 Apr 21:33

Re: Flow Counting

Hey Nick,
racluster() does this quite well.  When racluster() merges two  
records, for whatever reason,
it generates an aggregation statistic that keeps track of the number  
of records that are merged.
Use this to generate all the counts you would like.  The only trick is  
that if you use racluster()
to merge your data first, then this aggregation statistic already has  
values, and it needs to be
cleared.   You will want to remove that aggregation structure before  
you begin.

If you want to prep an entire directory of data, you can do this:
% rastrip -R dir -M replace -agr

This will remove the aggregation DSR from any record in any data file in
the entire directory struct.

If you can't, for whatever reason, delete the agr dsr, you can do it  
as a pipe.

% rastrip -r data.file -M -agr -w - - ip | racluster -m matrix -s  
saddr daddr trans

This will give you the number of transactions (or flows) for all the  
A's talking to B's in the file.

For things like time intervals, you should use rabins().

% rastrip -r data.file -M -agr -w - - ip | rabins -M soft time 10s -m  
srcid -s stime dur trans

This will give you the total flow every 10s, if there are any flows to  
report.  It will
force the startime and lasttime to coincide with the 10s time  
boundaries.

There are an infinite number of examples, so if this doesn't help,  
send more email.

Carter

On Apr 8, 2008, at 2:23 PM, Nick Diel wrote:

> I am interested in counting number of flows for things such as  
> source address, IP pairs, time intervals, etc.  This is closely  
> related to Stéphane Peters email: "Counting flows by time interval  
> in argus."
>
> After using racluster to merge status flow records, you have a file  
> where a record represents a flow.  You can of course use racount  
> with a filter to tell you some of this information, but it seem  
> quite impractical to do things such as find the IP pair generating  
> the most flows.  You could also use a set of pipes or scripts (as  
> Stéphane Peters showed us), but this would require modification  
> every time you wanted something slightly different.
>
> Maybe there is a way using ra tools to do this already and I am  
> missing it.  If not maybe we can get racluster (when specified) or  
> another tool to "zero out"/set the trans count to 1 after we have  
> merged status flow records.  This way things such as racluster -r  
> megredRecords.argus -m saddr -s +trans will now list how many flows  
> for each source address.
>
> I am curious what the group has to say about this.  Is something  
> like this already possible?  Is this information useful to other  
> people than me?  Would modifying the trans column present too much  
> ambiguity, i.e. "under these circumstances the trans column can  
> represent number of flows else it represents aggregated record count"?
>
>
> Nick
>
Permalink | Reply | 2 comments |
headers
jean-marc pouchoulon | 8 Apr 22:04
Picon

ragraph graph duration

Bonsoir,

I'm using ragraph 3.0.0 rc70 and I try to do graph duration

ragraph dur gave me back  usage message
but ragraph bytes with the same syntax  is ok

What I don't understand ?

Thanks.

jean-marc.
Permalink | Reply | 5 comments |
headers
Nick Diel | 8 Apr 23:18

Re: Counting flows by time interval in argus

I wanted to add in a solution Carter just showed me so it was part of this thread if anyone was searching.

This example assumes you have already merged status flow records, so records = flows, if not add another pipe of racluster.

rastrip -r $file -M -agr -w - | rabins -M soft time 10m -m srcid -s stime trans -c , -F raTime.conf > flowcounts.csv

raTime.conf contents (you could also add this to your rarc file):
RA_TIME_FORMAT="%H:%M"

If you have multiple collectors, you can have rabins merge on something else such as proto if you are filtering on tcp.

Nick


On Wed, Mar 26, 2008 at 1:04 PM, Stéphane Peters <stephane.peters <at> forem.be> wrote:
Hello,

Here is an example of counting flows I have just used,
to compare print flows seen by argus (filtered on port 9100)
with print requests seen by our batch server (found in a csv file).
Both lists have been feed in a spreadsheet to make a nice graphic comparison.

If someone sees a better way to do this within ra* clients without the unixes filters,
I will be happy to see how to do it.

Example saved on the wiki:
Count flows by groups of 10 minutes : show only the flow start times, cut after the 10ths of minutes, add a trailing zero and delete heading spaces to show a nice HH:MM line, count them, invert columns, insert a delimitor.  Ready to be feed in your favorite spreadsheet.
 ra -s stime -p 0 -nr $file |\
   cut -c -7 |\
   uniq -c | \
   sed -e 's/$/0/' \
       -e 's/^ *//' \
       -e 's/\(.*\) *\(.*\)/\2,\1/' > flowcounts.csv

Regards,

--
Stephane.Peters <at> forem.be, Postmaster <at> forem.be


Permalink | Reply | 502 comments |
headers
Nick Diel | 8 Apr 23:48

Re: ragraph graph duration

I think the keyword is dur, are you using that or duration?

ragraph -M dur -r file

Nick

On Tue, Apr 8, 2008 at 2:04 PM, jean-marc pouchoulon <jean-marc.pouchoulon <at> ac-montpellier.fr> wrote:
Bonsoir,

I'm using ragraph 3.0.0 rc70 and I try to do graph duration

ragraph dur gave me back  usage message
but ragraph bytes with the same syntax  is ok

What I don't understand ?

Thanks.

jean-marc.





Permalink | Reply | 4 comments |

Gmane