John Zhang | 2 May 2013 19:44
Picon

SEC on the big data security log management

Hi everyone,

I am researching the big data security log management,  such as Kibana + ElasticSearch + Logstash for my security log management, I need event correlation on this platform, i know SEC(http://simple-evcorr.sourceforge.net/)  can do event correlation. 

Do you have any idea of SEC on such big data security log platform? Any experience, any reference?

Any comment, advise will be highly appreciated!

Thanks!

John
------------------------------------------------------------------------------
Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET
Get 100% visibility into your production application - at no cost.
Code-level diagnostics for performance bottlenecks with <2% overhead
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap1
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
Risto Vaarandi | 26 Apr 2013 12:36
Picon
Favicon

asking for user opinions regarding tcp socket handling

hi all,
as you know, the support for tcp sockets was added recently to the SEC 
code which allows for integrations with applications capable of 
receiving data over TCP connections (like Graphite and various syslog 
servers).
The current implementation of 'tcpsock' action is fairly basic and only 
sends data to remote peer, ignoring any data that might be possibly sent 
from server to client. For this reason, the current implementation is 
not able to immediately detect situations where remote server closes the 
connection gracefully with the FIN packet. The fact that the connection 
has been closed can be detected by successful read from the socket which 
returns 0 bytes (similar to EOF condition when reading from pipes).
If the server is not sending any data to SEC but only expecting to 
receive data, things are easy -- there is no data available in the 
socket for reading, and when the appearance of new data is finally 
reported, reading from the socket will yield 0 bytes. However, if the 
server is sending back a lot of data, handling this might become 
somewhat complex -- in order to see EOF, all previously sent bytes must 
be read. It is not hard to periodically run a while(1) loop with calls 
to read() until no more data is available for reading, but if the server 
issues a massive byte stream, SEC could stay in the reading loop for 
more than few cycles.
Do you see this as a problem? Normally, one would only configure SEC to 
work with servers which behave in a known/predictable way and never send 
huge amounts of data back, but no implementation is free of bugs.
There are also other ways of tackling this problem -- for example, if 
any data is returned from server (be it EOF or real bytes), simply close 
the TCP socket, and let the next 'tcpsock' action to recreate it again.
What kind of implementation would you like to have?
kind regards,
risto

------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service 
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
Risto Vaarandi | 12 Apr 2013 18:17
Picon
Favicon

sec-2.7.2

hi all,
the 2.7.2 version of sec has been released which fixes a bug in parsing 
the 'rewrite' action. Due to this bug, 'rewrite' was not read in and an 
incorrect error message was reported. The new version with a bugfix is 
available at:

http://sourceforge.net/projects/simple-evcorr/files/sec/2.7.2/sec-2.7.2.tar.gz/download

kind regards,
risto

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
Vernon Nelson | 29 Mar 2013 21:12
Picon

trap suppression and threshold

ALCON,

 

      I have been trying to meet some criteria for about a week now and I cannot seem to nail it.  I am trying to meet the following requirements for matching a Juniper Netscreen trap for cpu utilization.  However, I cannot get it just right.  Any help would be greatly appreciated.

 

Problem:

      trap from Juniper comes in every minute when over cpu threshold

 

Solution I am trying to accomplish:

      1) push event to the event browser after 3 traps in 5 minutes

      2) re-alarm after 30 minutes

      3) clear alarm after 10 minutes with no traps

 

What I have works unless the CPU% changes and I cannot figure out how to get around matching that part of the trap.

 

Example traps:

1364222455 3  Mon Mar 25 14:40:55 2013  outervp01           ?  [2] private.enterprises.3224.2.3.0 (OctetString): 2013-03-25 14:41:41 [Root]system-critical-00030: SYSTEM CPU utilization is high (78 > alarm threshold:65) 1 times in 1 minute

1364222455 3  Mon Mar 25 14:40:55 2013  outervp01           ?  [2] private.enterprises.3224.2.3.0 (OctetString): 2013-03-25 14:41:41 [Root]system-critical-00030: SYSTEM CPU utilization is high (76 > alarm threshold:65) 1 times in 1 minute

     

 

Ruleset:

 

# match only when you receive 15 traps from the source VPN type=SingleWithThreshold ptype=RegExp continue=TakeNext pattern=(\w{8}vp\w+)\s+.*3224.2.3.0.*system-critical-00030.+CPU utilization is high desc= $5 high CPU alarm action=shellcmd /usr/OV/bin/event -e NDWN_EV -h $5 -d "TEST EVENT: 2 $5 system-critical-00030: SYSTEM CPU utilization is high."

window=930

thresh=15

type=SingleWith2Thresholds

ptype=RegExp

pattern=(\w{8}vp\w+)\s+.*3224.2.3.0.*system-critical-00030.+CPU utilization is high desc=$0 action=shellcmd /usr/OV/bin/event -e NDWN_EV -h $5 -d "TEST EVENT: 2 $5 system-critical-00030: SYSTEM CPU utilization is high."

window=330

thresh=3

desc2=$0

action2=shellcmd  /usr/OV/bin/event -e NUP_EV  -h $5 -d  "TEST EVENT: 2 $5 has sent 0 SYSTEM CPU utilization traps in the last 10 minutes. Validate the CPU is  below the threshold."

window2=600

thresh2=10
------------------------------------------------------------------------------
Own the Future-Intel(R) Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest. Compete 
for recognition, cash, and the chance to get your game on Steam. 
$5K grand prize plus 10 genre and skill prizes. Submit your demo 
by 6/6/13. http://altfarm.mediaplex.com/ad/ck/12124-176961-30367-2
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
Ward.P.Fontenot | 25 Mar 2013 20:12
Picon
Favicon

log / sec question

I currently have three applications logging to the same log server – though the logs are in different locations for each, and I have a working configuration for one of the environments that I’d like to implement across all of them. Can I simply start three instances of sec with the configuration files specific to each application instance or is there a more correct way of doing this?

Paul Fontenot

Enterprise Key Management & Public Key Infrastructure | EKM/PKI Engineering Team

2600 S. Price Rd. 2nd Floor | Chandler, AZ 85286

MAC S3939-022

Cell (480) 650-0301

ward.p.fontenot <at> wellsfargo.com

This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.

Attachment (smime.p7s): application/pkcs7-signature, 6359 bytes
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
Risto Vaarandi | 14 Mar 2013 16:12
Picon

(no subject)

hi all,

SEC-2.7.1 has been released and is available from:
http://sourceforge.net/projects/simple-evcorr/files/sec/2.7.1/sec-2.7.1.tar.gz/download

The main changes in this version are four new actions for writing data
to TCP, UDP and UNIX sockets. The actions are fairly simple and assume
the data moves unidirectionally only (any return data from remote peer
is ignored).

Also, the behavior of the 'write' action has changed -- in previous
releases, 'write' opened and closed the file at each access, while in
this version 'write' keeps the file open across all accesses. In order
to make file rotations possible, the SIGUSR2 signal has to be used
which closes all open files (each file will be reopened at the next
access).

kind regards,
risto

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
Jeffrey Starin | 4 Mar 2013 02:07
Picon

Beginner needs simple help about pattern matching with ftp logs

Hello.  I am a beginner with SEC and have read lots of the tutorials.  I have a few questions and sample problem.  I hope someone can help.

First off, when a rule is entered as such:

pattern=[INFO] (.+) <at> domain.com is now logged in

does sec look for any match within the string or must it match the string perfectly?

the reason I'm asking is I have logs files  that look like this in /var/log/messages:

Mar  3 18:36:43 who pure-ftpd: (? <at> 546.45.55.321) [INFO] admin-backup <at> domain.com is now logged in

Mar  3 18:33:37 who pure-ftpd: (admin-backup <at> domain.com <at> 546.45.55.321) [NOTICE] /home/domain/public_html/admin-backup//Client-contacts.xls downloaded  (10112 bytes, 116344.81KB/sec)

and i want alerts sent to me via email when either 1) someone ftp to the account, or 2) downloads a file via ftp.

and I have two separate .cfg files that I want to match on (although I'm struggling to find out how I can have one .cfg file for two separate rules -- haven't figured out how to do that yet).  Anyway, the rules.cfg file has:


type=Single
ptype=RegExp
pattern=(admin-backup) <at> domain.com is now logged in
desc=successful FTP login for account $1
action=pipe '$0' /bin/mail -s "$1 logged in! " me <at> gmail.com

and rules-download.cfg has:

type=Single
ptype=RegExp
pattern=[NOTICE](.) downloaded
desc=successful FTP download
action=pipe '$0' /bin/mail -s "FTP download!" me <at> gmail.com

Although these rules do indeed send emails, an email is sent saying someone has logged in when they are downloading, and that is not the intent, despite having [NOTICE] as the pattern in the rules-download.cfg file.

I can't seem to stop the rules.cfg pattern from triggering the rules-download.cfg intent.

Any help in understanding much appreciated.
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
Boyles, Gary P | 1 Mar 2013 18:08
Picon
Favicon

PairWithWindow Help Needed.

Hi,

I have a problem with the following “PairWithWindow” rule. It’s either my logic, a bug, or the way I understand the rule should work.

 

Here is the rule in question:

 

type=PairWithWindow

window=60

continue=GoTo END_CORRELATE_PAIR_WITH_WINDOW

ptype=RegExp

pattern=(\S+)\s+::\s+(\S+)\s+::\s+(\S+)\s+::\s+(\S+)\s+::\s+(\S+)\s+::\s+(.*)\s+::\s+(\S+.*)

context=($4 $5) -> ( sub { if ( defined $correlate1{"$_[0]::$_[1]::60"}) { return 1;} })

desc=Correlate1_PWW_60_A::$2::$4::$5

action=write /sec/log/sec.main.log %u %s; write /sec/log/sec.main.log %u $1 :: $2 :: $3 :: $4 :: $5 :: $6 :: $7

continue2=GoTo END_CORRELATE_PAIR_WITH_WINDOW

ptype2=RegExp

pattern2=(\S+)\s+::\s+(\S+)\s+::\s+(\S+)\s+::\s+(\S+)\s+::\s+(\S+)\s+::\s+(.*)\s+::\s+(\S+.*)

context2=(%4 %5 $4 $5) -> ( sub { if ( $correlate1{"$_[0]::$_[1]::60"} =~ /$_[2]::$_[3]/i) { return 1;} })

desc2=Correlate1_PWW_60_B::%2::$2::%4::$4::%5::$5

action2=write /sec/log/sec.main.log %u %s; write /sec/log/sec.main.log %u $1 :: $2 :: $3 :: $4 :: $5 :: $6 :: $7

 

The hoped-for logic is as follows:

 

I send in events in the following format:

 

# Timestamp  NodeName   Class   Monitor   Severity  Route   Message

# ---------  --------   -----  --------  --------  ------  -------

#   $1     ::  $2    ::  $3  ::   $4   ::   $5   ::  $6  ::   $7

 

 

If an event  comes in from nodeA and it matches monitorA::CRITICAL::60 in the “correlate1” table, then the PairWithWindow starts.

Since “desc=” includes node::monitor::severity I assumed that events with different nodes would start different “PairWithWindow” instances.

 

If I send in another event from nodeB, and it matches monitorA::CRITICAL::60 in the “correlate1” table, then another PairWithWindow starts (as expected).

 

If I do another event from nodeC – yet another PairWithWindow instance starts.

 

If after 60 seconds no matching event for any PairWithWindow occurs… then “action”  occurs for events nodeA, nodeB, and nodeC – as expected (see log below)

 

However, if I initiate all PairWithWindow events above, and initiate ONE “pattern2” option below… then ALL “action2” actions

get executed.  Not just the instance that pertains to a single node.

 

Now, I can fix this by adding the nodes to the “correlate1” hash-table key, but I didn’t think this is the way to handle this.

 

I had thought… that since I had node-name described in the “desc=” definition, that the events would be separated by node::monitor::severity,

and not just the monitor::severity.

 

Is this not the case.  If not… how is “desc=” used?

 

Thanks for your help.

 

Regards,

 

Gary Boyles, Intel

 

 

(Log-File Output Below)

 

(this is what happens if I wait for 60-second timer to expire)

 

1362155066 Correlate1_PWW_60_A::gpbuxA::gpbMonitor4::CRITICAL

1362155066 00 :: gpbuxA :: class :: gpbMonitor4 :: CRITICAL :: i=:n=:a=: :: Test Message

1362155071 Correlate1_PWW_60_A::gpbuxB::gpbMonitor4::CRITICAL

1362155071 00 :: gpbuxB :: class :: gpbMonitor4 :: CRITICAL :: i=:n=:a=: :: Test Message

1362155077 Correlate1_PWW_60_A::gpbuxC::gpbMonitor4::CRITICAL

 

 

(this is what happens when I send in ONE event that matches pattern2 for either A,B, or C… all 3 seem to unwind together)

 

1362155077 00 :: gpbuxC :: class :: gpbMonitor4 :: CRITICAL :: i=:n=:a=: :: Test Message

1362155281 Correlate1_PWW_60_B::gpbuxB::gpbuxB::gpbMonitor4::gpbMonitor4::CRITICAL::OK

1362155281 00 :: gpbuxB :: class :: gpbMonitor4 :: OK :: i=:n=:a=: :: Test Message

1362155281 Correlate1_PWW_60_B::gpbuxC::gpbuxB::gpbMonitor4::gpbMonitor4::CRITICAL::OK

1362155281 00 :: gpbuxB :: class :: gpbMonitor4 :: OK :: i=:n=:a=: :: Test Message

1362155281 Correlate1_PWW_60_B::gpbuxA::gpbuxB::gpbMonitor4::gpbMonitor4::CRITICAL::OK

1362155281 00 :: gpbuxB :: class :: gpbMonitor4 :: OK :: i=:n=:a=: :: Test Message

 

 

 

 

 

 

 

 

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
chris heidbrink | 28 Feb 2013 05:12
Picon
Favicon

detach mode probs

Hi, I used sec once successfully one time ago and I'm on to using it again to watch a syslog and based on a pattern to spawn a (perl) script which does some input checking and then calls an expect script to login to a device and take some action.  It is all working fine when I run it without --detach switch, but I need it to run as a daemon.  From what I can tell it seems like the first piece of the expect script is called but then nothing seems to happen, like the terminal (Stdin) can't see/send output/etc.  I saw some posts about needed full path to commands you are calling and those are there.  But I wonder if there is a better way to go about this than calling one script that calls another.  Works with running like:

/usr/bin/perl -w /usr/bin/sec.pl --conf=/etc/sec.conf --quoting --input=/var/log/local3 --pid=/var/run/sec.pid --syslog=daemon

doesn't work with --detach:

sudo /usr/bin/perl -w /usr/bin/sec.pl --conf=/etc/sec.conf --quoting --input=/var/log/local3 --pid=/var/run/sec.pid --detach --syslog=daemon

Here is my conf file:



 more /etc/sec.conf
# Executing on arbor alert

type=Single
continue=TakeNext
ptype=RegExp
pattern=\s\d\d:\d\d:\d\d\s(.*)\spfsp:\sanomaly\sTotal_traffic_rate_Misuse\s+id\s+(\d+)\s+status\s+ongoing\s+severity\s+3\s+classification\s+medium\s+impact\
s+\".*\"\s+src\s+0\.0\.0\.0\/0\s+All\s+dst\s+(\d.\d.\d.*\/32)
desc=Arbor hostname $1 alertid $2 blackholeip $3
action=spawn exec /usr/bin/addarborpfBlackholeexp.pl $1 $2 $3 $0

# that seems to execute fine when I feed it a syslog to match on:

Feb 27 22:20:27 xops01 sec.pl[31534]: Spawning shell command 'exec /usr/bin/addarborpfBlackholeexp.pl arborpfash01 43425 xxx.xxx.xx.xx/32 Feb  7 03:17:23 arborpfash01 pfsp: anomaly Total_traffic_rate_Misuse id 43425 status ongoing severity 3 classification medium impact "5.40 Mbps/6.79 Kpps" src 0.0.0.0/0 All dst xxx.xxx.xx.xx/32 test_alerting_MO start 2013-02-07 03:06:04 +0000 duration 659 percent 107.960000 rate 5e+06 rateUnit bps protocol nil flags nil url https://arbpfash01/page?id=alert_view&alert_id=43425'

but when that script calls out the expect script it starts at the first part to try to ssh / login / but doesn't go anywhere.

<snip of addarborpfBlackholeexp.pl>

 $arboroutput = qx(/usr/local/bin/expect -f /usr/bin/addarborpfBlackhole.exp '$params->{'devusername'}' '$params->{'password'}' '$params->{'device'}'
 '$params->{'name'}' '$params->{'ipversion'}' '$params->{'blackholeip'}' '$params->{'nexthopip'}' '$params->{'community'}' '$params->{'router1'}' '$params->
{'router2'}' '$params->{'prompt'}');

</snip>

# if I run without --detach it all works as expected.

Feb 27 22:18:54 xops01 sec.pl[31534]: SEC (Simple Event Correlator) 2.7.0
Feb 27 22:18:54 xops01 sec.pl[31534]: Reading configuration from /etc/sec.conf
Feb 27 22:18:54 xops01 sec.pl[31534]: Opening input file /var/log/local3
Feb 27 22:18:54 xops01 sec.pl[31534]: Stdin connected to terminal, SIGINT can't be used for changing the logging level
Feb 27 22:19:03 xops01 sec.pl[31534]: Spawning shell command 'exec /usr/bin/addarborpfBlackholeexp.pl arborpfash01 43425 xxx.xxx.xx.xx/32 Feb  7 03:17:23 arborpfash01 pfsp: anomaly Total_traffic_rate_Misuse id 43425 status ongoing severity 3 classification medium impact "5.40 Mbps/6.79 Kpps" src 0.0.0.0/0 All dst xxx.xxx.xx.xx/32 test_alerting_MO start 2013-02-07 03:06:04 +0000 duration 659 percent 107.960000 rate 5e+06 rateUnit bps protocol nil flags nil url https://arbpfash01/page?id=alert_view&alert_id=43425'


I turned up debug=6 and didn't provide me anything useful.  Appreciate any guidance / advice what could be going wrong and how to fix or if I should be going about something like this another way. 

Thanks


------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
Boyles, Gary P | 7 Feb 2013 18:30
Picon
Favicon

Window Parameter. Dynamic?

Is there any way in 2.7 to have a dynamic window parameter?

 

I have a lot of rules that are the essentially the same except for the timeframe (aka window=value).

 

Is there any way to make this dynamic (e.g. calling perl-code for the window-value)?

 

Thanks.

 

Gary Boyles

------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013 
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
Risto Vaarandi | 6 Feb 2013 18:38
Picon
Favicon

extending output types

hi all,

as you know, currently there are number of outputs which are not 
directly supported by sec. For example, with 'pipe' action one can feed 
only one event to an external program (with most programs exiting after 
the 'pipe' action closes the pipe). Also, the 'write' action opens a 
given file before each access and closes it afterwards. While this makes 
output file rotation worry-free, it is not very efficient. Finally, sec 
does not directly support various sockets as output (for example, it is 
not possible to connect directly to a syslog server on another host and 
issue syslog events without any intermediaries).

Therefore, would there be any interest to an output action which would 
leave the output open/running/connected, so that the next access of the 
same output would not have to do reopen/restart/connect again? This 
action would have to support a number of output types, e.g.,

write2 udp myserver:514 This is my event
write2 tcp myserver:514 Another event
write2 file /var/log/myevents Third event
write2 program /bin/logger -p user.notice Fourth event

It is probably easy to see that with sockets and files it is not that 
hard to distinguish them from each other, since the filename or server 
name plus port number can serve as unique identifiers. With command 
lines, however, it is somewhat more difficult, since an extra space or 
different ordering of options makes two command lines different, even if 
their effect is exactly the same. Also, since sec uses positional 
parameters, there is an issue with parsing, although this could be 
solved with the use of (). In addition, one workaround for both issues 
would be storing command line in an action list variable.
The action outlines above would also need an output rotation mechanism, 
but this could be implemented with adding this to SIGUSR2 handler 
(closing all currently open outputs).

Would the action above be of interest, and are there any other ideas for 
handling these situations?

kind regards,
risto

------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013 
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb

Gmane