Al Itchon | 25 Nov 17:26 2015

rabbitmq errors

I’m seeing an error in my log indicating that messages aren’t being sent to a queue in RabbitMQ.  This happens a few times a week.  Here’s the error:


Nov 24 13:29:28 syslog09 syslog-ng[7883]: Network error while inserting into AMQP server; time_reopen='10'


I have about 1000 servers sending tcp messages through a call to /dev/tcp to port on the syslog-ng server.  It seems messages aren’t sent to the queue for about 1 or 2 minutes, and then it resumes sending again without error.  Could this be a network issue or something within syslog-ng that’s limiting the messages?

Member info:

Scheidler, Balázs | 22 Nov 19:00 2015

unit test valgrind coverage


I've wanted to make our unit tests valgrind clean for a long time, here's where I am at right now, the branch fixes a number of production related bugs as well.

Feedback, help appreciated.

I am at 57 successful and 18 failing unit tests so far.

Member info:

Al Itchon | 20 Nov 04:48 2015


I have about 1000 servers sending tcp messages through a call to /dev/tcp to a port on my syslog-ng server.  I’m assuming each call to /dev/tcp is a single connection and a single message, so am I correct that the log_fetch_limit should be set to 1 since it applies to each connection?


Also, should I bother adjusting the so-rcvbuf?  I know the manual mentions that if using UDP, the receive buffer should be increased.  I’m using TCP and my message rate is about 250 messages/sec.

Member info:

Czanik, Péter | 18 Nov 10:26 2015

rabbitmq source


Someone asked me about the RabitMQ source. As I don't use RabitMQ I don't have much personal experience, so I ask you about it:

What are your experiences with the RabitMQ source?
What is your message rate?
What is your use case?
Any problems, missing features?

If for any reason, you don't want to share your experiences publicly, please e-mail me directly!
Any feedback is very welcome!

Member info:

Scheidler, Balázs | 13 Nov 16:29 2015

RFC: groupingby parser


I have just created a pull request for the groupingby() parser that I am seeking feedback for:

The commit message is pretty detailed on what it does, but let me reproduce it here for simplicity.

Any feedback is appreciated. Thanks.

dbparser: add groupingby() parser

This patch adds a new parser that can perform simple correllation on log
messages, e.g.  when multiple input log messages describe the same event.

In a way it is similar to the SQL GROUP BY operation, where an aggregate of
a set of input records can be calculated.

The major difference between SQL GROUP BY and groupingby() is that the first
_always_ operates on a enumerable list of records, whereas groupingby()
works on a stream of data.

groupingby() produces related groups by using a sliding window on time, e.g.
it can be specified how much time we need to look back to group related
messages together.

As a specific use-case, let's see Linux audit logs. Linux audit logs tend to
be broken to several lines generated as a list of lines.  These tend to be
pretty close in time, however there might be multiple events logged at
around the same time, which get mixed up in the output.

The example below is the audit log for an ntpdate execution:

    type=SYSCALL msg=audit(1440927434.124:40347): arch=c000003e syscall=59 success=yes exit=0 a0=7f121cef0b88 a1=7f121cef0c00 a2=7f121e690d98 a3=2 items=2 ppid=4312 pid=4347 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ntpdate" exe="/usr/sbin/ntpdate" key=(null)
    type=EXECVE msg=audit(1440927434.124:40347): argc=3 a0="/usr/sbin/ntpdate" a1="-s" a2=""
    type=CWD msg=audit(1440927434.124:40347):  cwd="/"
    type=PATH msg=audit(1440927434.124:40347): item=0 name="/usr/sbin/ntpdate" inode=2006003 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
    type=PATH msg=audit(1440927434.124:40347): item=1 name="/lib64/" inode=5243184 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
    type=PROCTITLE msg=audit(1440927434.124:40347): proctitle=2F62696E2F7368002F7573722F7362696E2F6E7470646174652D64656269616E002D73

These lines are connected by their 2nd field, msg equals to audit(1440927434.124:40347).

This can be processed by the groupingby() parser in a similar way that
db-parser() could do correllation.

These are the options for groupingby():

  * key(): specifies the key for the grouping, e.g. the value that must be the
    same for all messages in the group

  * scope(): specifies one of three values: "global", "host", "process", meaning
    the same as in db-parser; whether to apply grouping for all messages
    received by syslog-ng (global), only messages coming from the same host
    (host), or the same process/pid combination.

  * where(): specifies a filter condition, messages not matching the filter
    will NOT be added to the group.  where() only has access to a single
    message, the current one being processed.

  * having(): specifies a filter condition that must match in order for the
    group to generate an aggregate message. having() has access to the
    entire group through the "context".

  * timeout(): specifies the maximum time to wait for all messages in the
    group to arrive. After this time, the group is assumed to be complete
    and is aggregation is triggered.

  * aggregate(): this specifies the aggregate message that's going to be
    generated when the group is complete.
     - tags():
     - value():
     - inherit-mode():

  * inject-mode(): how the aggregate message is injected into the syslog-ng
    message routing, can be one of: "pass-through", "internal".

  * trigger(): trigger the closure of the group by matching an incoming
    message. If the filter condition specified here matches the incoming
    message, it will cause the aggregate message to be calculated, emitted
    and the group be discarded from the state table.

A few use-cases where this can be useful:
  * Linux audit logs
  * postfix logs

Signed-off-by: Balazs Scheidler <balazs.scheidler <at>>

Member info:

Fekete, Róbert | 13 Nov 13:43 2015

OSE 3.7.2 questions


I've seen that you had a few patches merged to 3.7.2, and I'm not sure what to include in the docs about them: 

 - There were some csv-parser changes, is any of them user-visible?
 - What does linux-audit-parser do? Does it require any configuration, or it just works?


Member info:

Gareth Allen | 12 Nov 09:34 2015

Remote server not keeping message intact

Hi all

I'm sending Apache logs to a remote syslog-ng server, but the remote
server isn't keeping the message intact.

My Apache log format:
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\"
\"%{User-Agent}i\"" combined

What the log looks like: - - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200
3594 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"

My syslog-ng configuration:
source s_apache {
 file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));

destination d_apache_tcp {
 tcp("x.x.x.x" port(514));

log { source(s_apache); destination(d_apache_tcp); };

Log server:
source s_net {

template apache {

destination apache {
    file("/var/log/apachetest" template(apache));

What I see in /var/log/apachetest:
- - [12/Nov/2015:08:30:59 +0000] "GET / HTTP/1.1" 200 3594 "-"
"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/46.0.2490.71 Safari/537.36"

As you can see the IP at the beginning of the log entry is being
removed.  I've tried using $MSG and $MSGONLY.

Any ideas would be greatly appreciated.
Member info:

Rory Toma | 11 Nov 01:44 2015

Having trouble with java-modules

I've downloaded syslog-ng-3.6.4 (I'm not using master because I can't 
get the configure script to generate correctly)

I'm following the steps here:

and have even added "--enable-java"

I notice that even with JAVA_HOME set on the command line, config.log 
picks up the wrong java, and passing --with-java-home has no effect.

In any case, I cannot get java modules installed. Is there an additional 
command to run to create/populate the java modules directory?

This is on CentOS-6.7 x86_64
Member info:

Alexander Urcioli | 2 Nov 01:44 2015

Syslog-ng 3.5 , "destination plugin python not found ..."

Trying to use a custom python destination I wrote. I wrote the python class, did the necessary PYTHONPATH stuff and wrote a separate .conf file for syslog-ng. When testing this file with the command "syslog-ng -f seclog.conf -Fevd" I get the following error:

Error parsing destination, destination plugin python not found in seclog.conf at line 9, column 31:

destination d_SECLOG_SERVER { python(

Is the python destination not supported in 3.5?

Member info:

Scheidler, Balázs | 31 Oct 13:28 2015

format-json reverse order?


I've encountered a case where format-json orders keys not alphabetically, but rather in the other direction. Can you remember any reason for that?

This is the only patch needed to fix the order, but I've figured there may have been some reason behind the ordering.

 static gint
 vp_walk_cmp(const gchar *s1, const gchar *s2)
-  return strcmp(s2, s1);
+  return strcmp(s1, s2);

Thanks in advance,

Member info:

vijay amruth | 29 Oct 06:08 2015

Regex Solaris from Linux hosts in Syslog-ng config file

Hello All,

We are drawing logs from several hosts which include solaris(10,11) , linux (centos, ubuntu, rhel) into syslog servers, I want to be able to separate solaris logs, is there any pattern we can match for solaris logs that you may know ?

Vijay Amrut.
Member info: