Russell Fulton | 24 Oct 01:31 2014
Picon
Picon

errors running configure for incubator on Ubuntu 12.04

rful011 <at> secmontst01:~/src/syslog-ng-incubator$ ./configure 
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane… yes

[ snip ]

checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for inline... inline
./configure: line 4534: syntax error near unexpected token `shared'
./configure: line 4534: `LT_INIT(shared dlopen)’

I have looked at configure and config.log to try and figure out what it is actually checking for. 
‘inline’ is to generic to google for :(

any ideas what is wrong?

Russell

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Russell Fulton | 23 Oct 01:17 2014
Picon
Picon

Elasticsearch destination

Hi

We are already using the open source version of syslog-ng and I am about to set up some elastic search
instances and would much prefer to feed data direct from syslog-ng rather than go through logstash (I
already have a heap of patterndb parsers and performance should be way better!)

I have spent an hour or so with Google and have found various references to elastic search destination being
available but I can find no mention of it in the release notes for 3.6.1.  I have also downloaded the the
tarball and unpacked it but could not find any evidence of the module , nore is there any mention of it in the manual.

As of now what is the recommended way of getting parsed data from OS syslog-ng into ES?

Thanks, Russell

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Czanik, Péter | 22 Oct 11:34 2014

syslog-ng 3.6.1 is released

Hi,

We are proud to announce that culminating almost a year’s worth of
development, syslog-ng 3.6.1 has been released! This is the first
stable release from the 3.6 branch, the successor to 3.5, which was
originally released in November, 2013.

Read more about the highlights of this release at
https://czanik.blogs.balabit.com/2014/10/syslog-ng-3-6-1-is-released/
or all the technical details at
https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.6.1

Bye,

Peter Czanik (CzP) <peter.czanik <at> balabit.com>
BalaBit IT Security / syslog-ng upstream
http://czanik.blogs.balabit.com/
https://twitter.com/PCzanik
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Evan Rempel | 20 Oct 00:47 2014
Picon
Picon

patterndb will not accept a program containing a $

Technically speaking, any non-alphanumeric character will terminate the 
TAG field at the beginning
of the message. This is usually one of : [ or space as in the examples

program: this is the message
program[123]: this is the message
program this is the message

In practice though, syslog daemons will send TAGs that contain any 
character and the syslog-ng Agent for Windows
will forward the application name as it shows in the Windows Event Log. 
In some cases, this TAG will contain a $ character.

The patterndb-4.xsd definition disallows the $ character in the program 
pattern in pattern database files.

Can this restriction be removed to allow for the $ or is this a larger 
issue that I see?

Thanks again for all of the support.

Evan.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Davide Alberani | 9 Oct 18:08 2014
Picon

inserts into a sqlite3 database are not delayed

Hi,
I'm using a SQLite 3 database as a destination,
but it seems that the options to delay the insert
of new rows are ignored.

The destination is something like:

destination sqlite_db {
        sql(type(sqlite3)
        flags('explicit-commits')
        flush_timeout(60000)
        log_fifo_size(1000)
        flush_lines(1000)
        retry_sql_inserts(2)
        database("/tmp/logs.db")
        [...table, columns, values, indexes definitions...]
};

Every time a log arrives, it's immediately written into
the database, while I expected to have it delayed accordingly
to flush_timeout and flush_lines.

Currently I'm using syslog-ng OSE 3.4.2, but looking at
the code of the latest version doesn't seem to have
changed much.
SQLite version is 3.8.4.
libdbi 0.8.4
libdbi-drivers 0.8.3
libdbi-dbd-sqlite 0.8.3
libol 0.3.16
(Continue reading)

Richards, James L - DOA | 3 Oct 21:18 2014

Question on parsing

So I have a scenario I am having difficulties with.

 

I have an IDS sensor (suricata), and it is generating a log-file at /log_file_dir/fast.log

 

And I would like to parse this log and send it off to a remote syslog server.

 

I have put the following in my syslog-ng.conf:

 

source s_log_server { file("/log_file_dir/fast.log " program_override("snort")); };

 

added a destination for the remote server:

 

destination d_log_server { udp ("fqdn.of.primary.elsa.box" port(514)); };

 

Then in the log{ section I have put this:

 

Destination (d_log_server);

 

Logs are making it to the remote box, but in an unparsed format…

 

How do I get this to trigger a parser in syslog-ng?

 

Thanks much,

 

 

 

 

 

Jim

 

 

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Jim Hendrick | 3 Oct 02:33 2014

syslog-ng as "shipper" into ELK stack

Hi,

   I am working on configuring Elasticsearch, Logstash & Kibana (ELK) to 
test it as a backend search tool for large volumes of logs.

I decided to put Redis in front of Logstash as a "broker" for the 
incoming logs, and syslog-ng as the "shipper" so it looks like this:

syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana

It works very well using the redis destination in syslog-ng, although I 
am having performance problems with logstash & elasticsearch default 
configurations keeping up.

(I topped out today sending ~7000 events per second, and saw an insane 
amount of swapping going on)

Not so much a specific question (I'll be working on heap & thread 
settings and am pretty confident I can get it to handle at least this 
moderate load) but I was wondering if anyone else is working in this area.

Also, in this configuration logstash is simply "parsing" the data it 
pulls from redis and sending it into elasticsearch.

Seems like something syslog-ng might be able to do directly.

Is anyone aware of any plans to implement an elasticsearch destination?

Feel free to contact me on or off list if you want to discuss this.

Thanks!!
Jim

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Doug McClure | 2 Oct 23:43 2014
Picon

File Source limits in OSE and PE?

Are there limits to the number of unique files that can be monitored with an installation of OSE or PE?

For example, could I monitor 500 unique log files on a given server?

Tks!

Doug
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

wiskbroom | 30 Sep 20:34 2014
Picon

Syslog-NG.conf to Fork to Two Log Aggregators

Hello;

I have syslog clients that I would like to configure to send log-data to a middle-man/intermediary syslog-NG server.  Once received on the intermediary, I want to immediately fork that data onto a different log-server, not syslog-NG; satisfying a requirement to feed two systems.

The reason for the fork is because the non-syslog-NG-server is running a proprietary logging system, and it must, at least for now, be capable of seeing *most* of my logs.  It, the non-syslog-NG-server, is incapable of retransmitting to my syslog-NG server, nor would I trust it to do so.

My questions to the list are,
1.   Has anyone successfully done something similar?
2.   Any recommendations/gotchas I should be aware of?
3.   Can I also configure syslog-NG to also resend Splunk data?  Or do I have to run a Splunk Univ Forwarder configured similarly to my intermediary syslog-NG server to achieve that?   (Yes, I know, OT question, sorry...)

Thank you in advance,

.vp





______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Justin Kala | 30 Sep 16:29 2014
Picon

Create Pattern-DB rules

Hello
 
 
I am trying to create Pattern-DB for the following Authorization messages coming from O/S.
Can you help on creating a matching rule .
 
Note: I do not have PATTERN-DB parser utility.I am going to create the db-parser.xml manually and put this rules inside the file.
 
2014-09-28T14:12:44-04:00 abcdef01-app/abcdef01-app sshd[11019]: [ID 800047 auth.notice] Failed password for root from 100.200.255.01 port 54438 ssh2
2014-09-28T14:03:46-04:00 abcdef01-app/abcdef01-app sshd[27420]: [ID 800047 auth.notice] Failed publickey for root from 100.200.255.02 port 59219 ssh2
2014-09-28T14:08:28-04:00 abcdef01-app/abcdef01-app sshd[3954]: [ID 800047 auth.notice] Failed keyboard-interactive for root from 100.200.255.03 port 65410 ssh2
2014-09-28T14:10:11-04:00 abcdef01-app/abcdef01-app sshd[5222]: [ID 293258 auth.error] libsldap: Status: 49  Mesg: openConnection: simple bind failed - Invalid credentials


Thanks & Regards
Kaladhar
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

jetjnkr | 29 Sep 21:48 2014
Picon

Continued struggle with getting shell env vars into log statements

Greetings,

I'm trying to get an environment variable (As defined in the shell - take for example "HOSTNAME").  

echo $HOSTAME provides the expected string....  I then would like to append this to a set of messages based upon a specific destination.  I'm using syslog-ng 3.2.5. In this case the 'hostname' is just an example,  I know that the sending hostname is automatically prepended  - used here because it's commonly defined on systems.

I've included the following config lines from my syslog-ng.conf file.

destination d_web_call      { file("/var/log/web_call" template("<$PRI> $DATE $HOST $MSGHDR$MSG host=`HOSTNAME` \n") template_escape(no)); };
filter f_web_call                 { facility(local1) and ( match("^.*apache-call.*$" value("MSGHDR")) );};

log {
        source(s_sys);
        filter(f_web_call);
        destination (d_web_call);
        flags(final,flow-control);
};


Using the following test string:

/usr/bin/logger -t apache-call-tst -p local1.notice "This is a test message 25"

I get the following:

<141> Sep 26 17:21:18 lab.xxxx.com apache-call-tst: This is a test message25 host= 



I've received a few pointers from folks here (including going to 3.5 which can't be completed in the short term....).  If tried things with 'define' and using the '$' but not back-tick for the shell variable.  It is not clear when this value is set (if at all) within the config file.

Thanks,
Jim.


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


Gmane