Czanik, Péter | 18 Dec 11:59 2014

insider 2014-12: incubator 0.4.1 released; anonymization; monitoring;

Dear syslog-ng users,

This is the 39th issue of the syslog-ng Insider, a monthly newsletter
that brings you syslog-ng-related news.

FEATURED NEWS

syslog-ng incubator 0.4.1 released

------------------------

We are proud to announce that version 0.4.1 of syslog-ng incubator is
released. It is compatible with syslog-ng version 3.6. Some features
previously included in the incubator such as Riemann and Graphite
support have been ported to syslog-ng. It also adds many interesting
new features: kafka and java destination support, ZMQ source and
destination and a grok parser.

Ports and packages are available for Fedora, FreeBSD and openSUSE, but
with some of the features missing, as dependencies are not yet
available packaged.

For details, read
https://czanik.blogs.balabit.com/2014/12/syslog-ng-incubator-0-4-1-released/

Data Privacy, Anonymization and Log Data

----------------------------------------

Data privacy is becoming a hot topic, especially in the European
(Continue reading)

Roberto Carna | 17 Dec 14:41 2014
Picon

Email alerts en syslog-ng 3.3.5

Dear, I have this syslog platform:

- Debian 7
- Syslog-ng 3.3.5.-4 (installed via apt-get)
- Php-syslog-ng (graphical interface)
- Postfix

I need to configure syslog-ng in order to send email
alerts/notifications about some specific events.

Is it possible to implement with this syslog-ng version and what is the way???

Thanks a lot!!!

Roberto
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Jim Hendrick | 16 Dec 19:24 2014

syslog-ng memory usage grows

Hi,

   I am continuing to test syslog-ng as the parser and shipper into an 
elasticsearch cluster.

Right now I have syslog-ng 3.6.1 receiving logs at about ~7000 EPS and 
running them through a patterndb parser that splits out the (bluecoat 
proxy) fields into key-value pairs, writing them to a redis destination 
using format-json.

syslog-ng 3.6.1
Installer-Version: 3.6.1
Revision:
Compile-Date: Dec  9 2014 19:42:20
Available-Modules: dbparser,json-plugin,afuser,affile,afmongodb,tfgeoip,afprog,redis,afstomp,afsql,afsocket,system-source,afamqp,pseudofile,confgen,afsocket-tls,csvparser,basicfuncs,graphite,syslogformat,afsocket-notls,linux-kmsg-format,cryptofuncs
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: off
Enable-Linux-Caps: off

I then use logstash to pull from redis and feed elasticsearch (the 
thought being that this would provide a buffer for the messages)

Over the weekend I had syslog-ng crash (unfortunately no core) but now 
that I am watching it more closely, I see what appears to be continuous 
growth in memory usage (which leads me to suppose this was the cause of 
the crash).
(Continue reading)

Scheidler, Balázs | 15 Dec 21:09 2014

RFC: debugger


Hi,

I was involved in a number of projects where syslog-ng was used in a relatively complex setup: we wanted to structure incoming log messages and transform them into nice & clean JSON messages.

In a number of cases, this required several parsers to get right (csv-parser, db-parser and sometimes regexps) and the resulting configuration was pretty complicated.

I've figured, that if we have a domain specific language in the form of the syslog-ng configuration file, we should also have a debugger that makes it easy to find out what went wrong. And this is what I have did, some proof-of-concept code is available on the f/debugger branch on github:

https://github.com/balabit/syslog-ng/tree/f/debugger

Here's how it works:

# start up syslog-ng with the -i option to indicate we
# want the debugger enabled

$ sbin/syslog-ng -iFe

Once syslog-ng starts up, you'll see something like this:
Waiting for breakpoint...

For now, all processing elements within syslog-ng (called LogPipe) breaks into the debugger. Later, I want to implement real breakpoints, where you can selectively attach breakpoints to elements in the configuration file.

# on another console send a log message to syslog-ng:
$ logger almafa

This is what happens on the syslog-ng side:

Breakpoint hit etc/syslog-ng.conf:11:2
11           unix-stream("log");
Dec 15 16:14:05  bazsi: almafa
(syslog-ng)


At a breakpoint, syslog-ng displays where the break has occurred, quotes the source line and displays the message that is being processed.

Now it becomes interactive, you can inspect the message with various commands:

(syslog-ng) help
syslog-ng interactive console, the following commands are available

  help, h, or ?            Display this help
  continue or c            Continue until the next breakpoint
  print, p                 Print the current log message
  drop, d                  Drop the current message
  quit, q                  Tell syslog-ng to exit
(syslog-ng) p $MSG
almafa
(syslog-ng) p
MESSAGE=almafa
PROGRAM=bazsi
LEGACY_MSGHDR=bazsi:
.unix.pid=18703
.unix.uid=1000
.unix.gid=1000
TAGS=


(syslog-ng) p "$(format-json --key *)"
{"_unix":{"uid":"1000","pid":"18703","gid":"1000"},  ... }



With the "continue" command, you can request syslog-ng to proceed to the next LogPipe, where it stops again. "drop" drops the current message, "quit" tells syslog-ng to exit.

I would really like to extend this to further ways, like:
  • real breakpoints, instead of stopping everywhere
  • inject messages
  • change messages to make configuration testable
  • single-step (to the next primitive LogPipe) and step-over (to the next 'real' LogPipe that is found in the configuration file)
  • interactive REPL to template functions to make it easier to interact with them

What do you think?

--
Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

guamar | 15 Dec 15:09 2014

Syslog-ng Drops Logs

Hello,

i'm IT technical at CNRSt Morocco.

I work on a project to centralize logs with syslog-ng , but  so some logs are dropped .

I have 10 clients and on each client over 12 website is hosted.

thank you to find attached the config file.

 

thanks.
Cordialement.

----------------------------------------------------------------------------------------------------------------------------------------------

Thami Guamar

Service informatique

Institut Marocain de l'Information Scientifique et Technique (IMIST)-

Centre National pour la Recherche Scientifique et Technique (CNRST)

Angle avenue Allal El Fassi, avenue des FAR,Quartier Hay Ryad B.P. 8027 Rabat-Nations Unies

10102 Rabat, Maroc

E-mail :guamar <at> imist.ma

Tél :  (212) 5 37 56 99 00/36                                                          

Fax : 212 5 37 56 99 01                                                                             

 

Attachment (syslog-ng_server.conf): application/octet-stream, 12 KiB
Attachment (qyqlog-ng_client.conf): application/octet-stream, 1897 bytes
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Balazs Scheidler | 6 Dec 13:43 2014
Picon

Re: [syslog-ng 3.6.1] booting with openrc, but syslog-ng thinks i boot with sytemd

Hi Lone_Wolf,

Could you pls give the patch a try? Should fix your issue.

Thanks.


On Thu, Dec 4, 2014 at 12:43 PM, Juhász, Viktor <viktor.juhasz <at> balabit.com> wrote:
Hi,

There is a pull request on to 3.6/master:

I created it on morning, but it is not listed in the pull requests list

Viktor


On Thu, Dec 4, 2014 at 12:27 PM, Balazs Scheidler <bazsi77 <at> gmail.com> wrote:

Hi,

Viktor could you pls share the fix for this? I know I promised to port it, but got distracted.

Thanks.

On Dec 4, 2014 11:36 AM, "LoneVVolf" <lonewolf <at> xs4all.nl> wrote:
On 04-12-14 09:58, Tibor Benke wrote:
I think you've found a bug in service-management.c:


If you have syslog-ng compiled with --enable-systemd, it will always be the active service-management unit. I thought we addressed this in a patch by jviktor.
Tibor,

Is that code run during compiling/building or at run-time ?
Archlinux developers build packages on machines with an archlinux base installation with systemd installed & active .
If service-management.c is executed at build-time, that would explain why syslog-ng tries to use systemd on my system.

Given the multi-platform support of syslog-ng i would expect such checks to be executed at run-time .




2014-12-04 0:04 GMT+01:00 Balazs Scheidler <bazsi77 <at> gmail.com>:

Also, you might want to use the system() source, which tries to determine the optimal setup to fetch local messages. So no need to use I unix-dgram or internal directly.

Balazs, thanks for the tip. After the problem is fixed, i'll look into using that.

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq






--
Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Jason Long | 6 Dec 07:17 2014
Picon

Create a Log file for Each server.

Hello all.
How are you?
I have a windows sever with Syslog agent installed on it and it forward all logs to my Linux box. My Syslog-NG
collected it very well but I want to Forward another Windows Events into my Syslog server but How can I
modify my current config file for create another file for this new server? I use Logstash and Kibana on my
Linux Box, Can it detect my new Log file automatically? 

# syslog-ng configuration file.
#
# This should behave pretty much like the original syslog on RedHat. But
# it could be configured a lot smarter.
#
# See syslog-ng(8) and syslog-ng.conf(5) for more information.
#

options {
flush_lines (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
};

source s_netsyslog {
udp(ip(0.0.0.0) port(514) flags(no-hostname));
tcp(ip(0.0.0.0) port(514) flags(no-hostname));
};

destination d_netsyslog { file("/var/log/network.log" owner("root") group("root") perm(0644)); };

log { source(s_netsyslog); destination(d_netsyslog); };

as you see it collect Syslog from any IP address.

Cheers.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Juhász, Viktor | 5 Dec 13:44 2014

Sorry, it is just a test

Test
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

LoneVVolf | 3 Dec 14:40 2014
Picon
Picon

[syslog-ng 3.6.1] booting with openrc, but syslog-ng thinks i boot with sytemd

Hi,

First i'd like to say i have been using syslog-ng for years and am very pleased with it.
Thank you for a reliable and stable product.

I'm running archlinux x86_64 and boot with openrc 0.13.5 . I do have systemd installed so i can use udev functionality.
Today I updated from syslog-ng 3.5.6 to 3.6.1 and found this in rc.log :

[2014-12-03T13:27:21.835298] Using /dev/log Unix socket with systemd is not possible. Changing to systemd-syslog source, which supports socket activation.;
 * Starting syslog-ng ...
[2014-12-03T13:27:21.890729] Using /dev/log Unix socket with systemd is not possible. Changing to systemd-syslog source, which supports socket activation.;
[2014-12-03T13:27:21.902289] Failed to acquire /run/systemd/journal/syslog socket, disabling systemd-syslog source;
 [ ok ]

This is the source part of my /etc/syslog-ng/syslog-ng.conf
source src {
  unix-dgram("/dev/log");
  internal();
  file("/proc/kmsg");
};


I use the official archlinux syslog-ng package, https://www.archlinux.org/packages/extra/x86_64/syslog-ng/ .

It uses these build flags :
./configure --prefix=/usr --sysconfdir=/etc/syslog-ng --libexecdir=/usr/lib \ --sbindir=/usr/bin --localstatedir=/var/lib/syslog-ng --datadir=/usr/share/syslog-ng \ --with-pidfile-dir=/run --disable-spoof-source --enable-ipv6 --enable-sql \ --enable-systemd --with-systemdsystemunitdir=/usr/lib/systemd/system
syslog-ng does appear to work normally, but i'd like to get rid of the incorrect error message.

Lone_Wolf

(full rc.log & syslog-ng.conf attached)
Attachment (rc.log): text/x-log, 27 KiB
 <at> version: 3.6
 <at> include scl.conf
#
# /etc/syslog-ng/syslog-ng.conf
#

options {
  stats_freq (0);
  flush_lines (0);
  time_reopen (10);
  log_fifo_size (10000);
  chain_hostnames (off);
  use_dns (no);
  use_fqdn (no);
  create_dirs (no);
  keep_hostname (yes);
  perm(0640);
  group("log");
};

source src {
  unix-dgram("/dev/log");
  internal();
  file("/proc/kmsg");
};

destination d_authlog { file("/var/log/auth.log"); };
destination d_syslog { file("/var/log/syslog.log"); };
destination d_cron { file("/var/log/crond.log"); };
destination d_daemon { file("/var/log/daemon.log"); };
destination d_kernel { file("/var/log/kernel.log"); };
destination d_lpr { file("/var/log/lpr.log"); };
destination d_user { file("/var/log/user.log"); };
destination d_uucp { file("/var/log/uucp.log"); };
destination d_mail { file("/var/log/mail.log"); };
destination d_news { file("/var/log/news.log"); };
destination d_ppp { file("/var/log/ppp.log"); };
destination d_debug { file("/var/log/debug.log"); };
destination d_messages { file("/var/log/messages.log"); };
destination d_errors { file("/var/log/errors.log"); };
destination d_everything { file("/var/log/everything.log"); };
destination d_iptables { file("/var/log/iptables.log"); };
destination d_acpid { file("/var/log/acpid.log"); };
destination d_console { usertty("root"); };

# Log everything to tty12
destination console_all { file("/dev/tty12"); };

filter f_auth { facility(auth); };
filter f_authpriv { facility(auth, authpriv); };
filter f_syslog { program(syslog-ng); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kernel { facility(kern) and not filter(f_iptables); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };
filter f_ppp { facility(local2); };
filter f_debug { not facility(auth, authpriv, news, mail); };
filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news, cron) and not
program(syslog-ng) and not filter(f_iptables); };
filter f_everything { level(debug..emerg) and not facility(auth, authpriv); };
filter f_emergency { level(emerg); };
filter f_info { level(info); };
filter f_notice { level(notice); };
filter f_warn { level(warn); };
filter f_crit { level(crit); };
filter f_err { level(err); };
filter f_iptables { match("IN=" value("MESSAGE")) and match("OUT=" value("MESSAGE")); };
filter f_acpid { program("acpid"); };

log { source(src); filter(f_acpid); destination(d_acpid); };
log { source(src); filter(f_authpriv); destination(d_authlog); };
log { source(src); filter(f_syslog); destination(d_syslog); };
log { source(src); filter(f_cron); destination(d_cron); };
log { source(src); filter(f_daemon); destination(d_daemon); };
log { source(src); filter(f_kernel); destination(d_kernel); };
log { source(src); filter(f_lpr); destination(d_lpr); };
log { source(src); filter(f_mail); destination(d_mail); };
log { source(src); filter(f_news); destination(d_news); };
log { source(src); filter(f_ppp); destination(d_ppp); };
log { source(src); filter(f_user); destination(d_user); };
log { source(src); filter(f_uucp); destination(d_uucp); };
#log { source(src); filter(f_debug); destination(d_debug); };
log { source(src); filter(f_messages); destination(d_messages); };
log { source(src); filter(f_err); destination(d_errors); };
log { source(src); filter(f_emergency); destination(d_console); };
log { source(src); filter(f_everything); destination(d_everything); };
log { source(src); filter(f_iptables); destination(d_iptables); };

# Log everything to tty12
#log { source(src); destination(console_all); };
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Jason Long | 26 Nov 12:25 2014
Picon

How can I have Persian and Arabic names in Syslog?

Hello Folks.
How are you?
I installed a Syslog agent on Windows server and some of directories on Windows server are Persian. In Linux box I installed Logstash,Syslog-ng and Kibana but when my Linux receive Logs from Windows server the name shown as "????". How can I solve it?

Cheers.

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Lucas, Sascha | 25 Nov 08:34 2014
Picon

Re: Remote tags

Hi Nikolay,

> Could anyone here advice me if it is possible to set a tags() on a log
> entry on one machine, send this log message to a remote syslog-ng and
> use this tags() in a filter on the remote machine?

As Fabien pointed out, it is possible. I'm doing something similar using rfc5424 protocol:

The first thing I do is rewriting the log to append local scoped macro data into the sdata structure (here I'm
using $SOURCEIP, where you want $tags). When I read rfc5424 I remember, that there are custom
data-structures where you can store your tags (I decided to abuse .SDATA.origin.ip for my purpose):

rewrite r_sdata {
        set("$SOURCEIP" value(".SDATA.origin.ip"));
};

The second thing is to use the syslog-driver (capable of sending and receiving rfc5424):

destination d_logserver { syslog("X.X.X.X" transport("udp")); };

And finely the log line

log { source(s_network); source(src); rewrite(r_sdata); destination(d_logserver); };

On the server I have a source capable of rfc5424:

source s_network { syslog( transport("udp") flags(validate-utf8) so-rcvbuf(2097152)); };

The transferred Information is directly available on the server in the macro ${.SDATA.origin.ip}. Your
tags may be a bit special, because multiple tags would be transferred as a comma separated string.
Matching on a single tag would probably mean to rewrite the log again. This time with something like
set("${.SDATA.your.structure}. " value("tags"));.

HTH, Sascha.

Aufsichtsratsvorsitzender: Herbert Vogel
Geschäftsführung: Michael Krüger
Sitz der Gesellschaft: Halle/Saale
Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
UST-ID-Nr. DE 158253683

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der
richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort
den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht
gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und
die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der
bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die
GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


Gmane