Syslog-ng user and developer list

Xuri Nagarin | 18 May 2013 02:35

Filtering binary data fields and catch all

Hi,

My log source sends data over syslog in CEF (Arcsight format).

I have a simple parser written in Syslog-NG that takes advantage of the CEF format (that is all messages are in eight fields separated by a pipe char).

-----------xxxxxxxxxxxxxxxxx-------------------

parser p_cef {
    csv-parser(columns("cef.ff","cef.vendor","cef.product","cef.c4","cef.c5","cef.c6","cef.c7","cef.c8")
    delimiters("|")
    flags(drop-invalid)
    );
};

destination d_file { file("/var/log/net/${cef.vendor}/${cef.product}/logfile"); };

log { source(s_tcp); parser(p_cef); destination(d_file); };
-----------xxxxxxxxxxxxxxxxx-------------------

Some events coming in contain binary data that get translated into control characters and create thousands of directories with garbage in the names.

How do I handle binary data in the message? Should I use the sanitize function in the destination/file definition or is there a better way to do it?

Also, how do I create a catch-all destination for all the messages that do not match my parser?

Thanks!

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Peter Czanik | 16 May 2013 15:08

syslog-ng with mongodb user survey

Hello,
Mongodb support is available in syslog-ng for more than a year now. We 
are aware, that there are many people using it, but now we would like to 
ask for some feedback about the use cases and environments. Here are 
just a few sample questions, but any feedback regarding mongodb support 
is very welcome:
- average and peak message rates sent to the mongodb destination
- what is your related syslog-ng.conf
- how did you tune your mongodb server (if you did)
- how and how often do you query your logs in mongodb
- etc.
Thanks for your help!
Bye,

--

-- 
Peter Czanik (CzP) <czanik <at> balabit.hu>
BalaBit IT Security / syslog-ng upstream
http://czanik.blogs.balabit.com/

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Gergely Nagy | 16 May 2013 14:35

[PATCH (3.5)] system-source: Make /proc/kmsg optional

In certain environments (vserver and OpenVZ come to mind), /proc/kmsg is
not readable, not even for root. On these systems, emit a warning, and
skip it, instead of aborting the startup.

Reported-by: Tamas Pal <folti <at> balabit.hu>
Signed-off-by: Gergely Nagy <algernon <at> balabit.hu>
---
 modules/system-source/system-source.c |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c
index 3c72859..955ade7 100644
--- a/modules/system-source/system-source.c
+++ b/modules/system-source/system-source.c
 <at>  <at>  -167,8 +167,17  <at>  <at>  system_sysblock_add_linux_kmsg(GString *sysblock)
       close (fd);
     }

-  system_sysblock_add_file(sysblock, kmsg, -1,
-                           "kernel", "kernel", format);
+  if (access(kmsg, R_OK) == -1)
+    {
+      msg_warning("system(): The kernel message buffer is not readable, "
+                  "please check permissions if this is unintentional.",
+                  evt_tag_str("device", kmsg),
+                  evt_tag_errno("error", errno),
+                  NULL);
+    }
+  else
+    system_sysblock_add_file(sysblock, kmsg, -1,
+                             "kernel", "kernel", format);
 }

 gboolean
--

-- 
1.7.10.4

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Peter Czanik | 16 May 2013 13:12

insider 2013-05: syslog-ng configurator on Android; Using syslog-ng with Splunk; EU data protection and logging

Dear syslog-ng users,

This is the 24th issue of the syslog-ng Insider, a monthly newsletter 
that brings you syslog-ng related news.

FEATURED NEWS

syslog-ng configurator app for Android

--------------------------------------

There is now a new configurator app available for Android, developed as 
a hobby project by one of the syslog-ng team members. The focus of the 
application is to create a syslog-ng.conf wich provides optimum 
performance based on a number of questions.

For more details and download locations read the authors blog at 
http://pzolee.blogs.balabit.com/2013/04/little-syslog-ng-configurator-application-for-android/

syslog-ng and Splunk

--------------------

We often receive questions, how to use syslog-ng and Splunk together in 
a logging infrastructure. We collected the most popular usage scenarios 
into a white paper, together with example configurations to make testing 
and integration even more easy.

The WP is available at 
http://www.balabit.com/support/documentation/pdf/syslog-ng_splunk_deployment_guide_en.pdf

syslog-ng is participating GSoC again

-------------------------------------

Just as last year, syslog-ng is participating Google Summer of Code 
under the umbrella of the openSUSE project. We have candidates for 
developing a native mysql destination, a redis destination, a XMPP 
(jabber) destination and log signing, which is a big improvement from 
last year, where we only had a single candidate for a similar number of 
development projects. The application process is closed now and there 
are still a couple of weeks to go, before the final list of approved 
students is announced.

http://news.opensuse.org/2013/04/25/opensuse-hedgewars-and-owncloud-are-moving-gsoc-along-participate-and-submit-your-proposals-fast/

Big changes ahead for EU data protection regulation

---------------------------------------------------
This summer will most likely bring big changes in the regulation of Data 
Protection in the European Union. We collected these proposed changes 
and also how syslog-ng and proper central log management can help to 
comply with these regulations:

http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu-

<http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu-data-protection-regulation/>data-protection-regulation/ 
<http://jluby.blogs.balabit.com/2013/04/22/big-changes-ahead-for-eu-data-protection-regulation/>

Compiling syslog-ng with MS SQL support on RHEL / CentOS & Co.

--------------------------------------------------------------

During the past few months many people asked, how to log from syslog-ng 
to MS SQL on RHEL or CentOS. If you cannot buy syslog-ng PE 
(http://www.balabit.com/network-security/syslog-ng/central-syslog-server), 
follow these steps to compile all the necessary components yourselfand 
configure the MS SQL part:

https://czanik.blogs.balabit.com/2013/04/compiling-syslog-ng-with-ms-sql-support-for-rhel-centos-co/

SHORT NEWS:

  *

    RFC5424 syslog between syslog-ng and rsyslog:
    https://czanik.blogs.balabit.com/2013/04/using-the-rfc5424-syslog-protocol-with-plain-tcp-between-rsyslog-and-syslog-ng/

  *

    Linuxwochen: syslog-ng, AMQP, and 3D printers:
    http://czanik.blogs.balabit.com/2013/05/czp-linuxwochen-syslog-ng-amqp-and-3d-printers/

Your feedback and news tips about the next issue is welcome at 
documentation <at> balabit.com <mailto:documentation <at> balabit.com> To read 
this newsletter on-line, visit: http://insider.blogs.balabit.com/

--

-- 
Peter Czanik (CzP) <czanik <at> balabit.hu>
BalaBit IT Security / syslog-ng upstream
http://czanik.blogs.balabit.com/

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Russell Fulton | 11 May 2013 04:29

rewrite part of the message...

Apologies to those of you who have already seen this on the ELSA list.

I want to get rid of the explanatory essay that accompanies some eventlog messages from windows.

I tried this:

rewrite r_snarex { subst("\s+This event is generated when[^|]+\|", "|", value("MSGONLY") type("pcre"));
};

and added it to log section, but it did not work.  I have tried various variations on the theme too.

Russell
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Russell Fulton | 11 May 2013 04:15

weird filter problem

Currently on 3.2.4.

I am using patterndb to classify my incoming logs and want to send those that match pattern rules do a
particular destination.

filter f_unknown { "${.classifier.class}" ne 'unknown' ; };

but this appears to always fail (if I change ne to eq then I get everything).

I have also tried using match with the same result.

Russell

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Rakesh Rajasekharan | 10 May 2013 18:52

Re: syslog-ng and logrotate

Yup ... got it.. so for now , i need to rely on the logrotate outside of syslog-ng....

Thanks,
Rakesh.

On Fri, May 10, 2013 at 6:50 PM, Jakub Jankowski <shasta <at> toxcorp.com> wrote:

On 2013-05-10, Gergely Nagy wrote:

> Rakesh Rajasekharan <rakesh.rajasekharan <at> gmail.com> writes:
>
>> yes something similar .. but instead of the day, month can we do it based
>> on the size. say once it crosses 1 GB , archive it and have a new log file
>> created. Can we do this from syslog-ng ?
>
> No, that is not possible yet. There are plans to add support for it,
> there was even a patch a year or so ago, but it needs to be cleaned up
> and updated before proposing it for merging into 3.5. And as such, if
> and when this feature appears, it won't be in the 3.3 or 3.4 branches.

And until this is implemented, my suggestion is to use logrotate as you'd
use with classic BSD syslogd. Don't forget to send SIGHUP to syslog-ng in
postrotate hooks.

HTH,

--
Jakub Jankowski|shasta <at> toxcorp.com|http://toxcorp.com/
GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Rakesh Rajasekharan | 10 May 2013 11:27

syslog-ng and logrotate

I am using the syslog-ng 3.3.7 version. My question is , can we manage log rotation within the syslog-ng conf file ?

Please advice

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Dylan Kulesza | 9 May 2013 00:38

Question on custom log writer message output

I'm working on a custom module to integrate with a third parties native log format. My intent is to have messages come into syslog-ng and processed as usual and then sent out a custom destination driver.

Right now I've hacked together different code to make it work (tcp socket connection per log source) and now I'm at the point of actually sending a custom message. I've tried to stay as "true" to the syslog-ng as possible and have leveraged the log_forward_msg method to send my LogMessage. I was hoping I could just prepend data to the LogMessage but realized after doing all the other leg work that it wasn't a simple string :)

So, my question is - what would the easiest way to leverage the existing queue->log_forward_msg (doesn't require the socket to be open vs examples such as spoof_source in afsocket) to write a custom message? I see that LogTemplate may have what I need, but after submerging myself in syslog-ng for the past week I'm not seeing clearly... Can anyone lend a hint/helping hand?

What I'm trying to do:

Open Socket

Send Magic/StartPacket

Prepend all log messages with a byte message - for example:

040404040400010MESSAGE (Of course Message would be in bytes/hex).

It seems I would create an NVENTRY for my prepend message and then override log_writer_format_log? to do this? Not 100% clear how I would accomplish this. I also don't want to change any of the core/lib syslog-ng to accomplish this. Should be implemented purely as a module.

Thanks!

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

bugzilla | 8 May 2013 17:08

[Bug 235] New: String arguments to block parameters get mistreated

https://bugzilla.balabit.com/show_bug.cgi?id=235

           Summary: String arguments to block parameters get mistreated
           Product: syslog-ng
           Version: 3.4.x
          Platform: All
        OS/Version: All
            Status: ASSIGNED
          Severity: blocker
          Priority: unspecified
         Component: syslog-ng
        AssignedTo: algernon <at> balabit.hu
        ReportedBy: algernon <at> balabit.hu
Type of the Report: regression
   Estimated Hours: 0.0

I'm not quite sure who and where reported this bug first, but I'm recording it within Bugzilla, so we can keep
track of it.

With the config below, syslog-ng fails to start:

==========================================================
 <at> version: 3.5

block source s_logfile (filename(messages)) {
  file("/var/log/`filename`" );
};

source s_example {
  s_logfile(filename("logfile.log"));
  s_logfile(filename(logfile.log));
  s_logfile(filename(foo bar baz));
};
==========================================================

Error parsing affile, syntax error, unexpected LL_IDENTIFIER in source block s_logfile at line 2, column 19:
                                                  included from etc/block-syntax.conf line 8, column 36

  file("/var/log/"logfile.log
                  ^^^^^^^^^^^

The first line of s_example makes it fail badly. This is because the lexer recognises "logfile.log" as a
block_string, and stores the quotation marks too,
therefore the backtick expansion creates an invalid string. We should not store the "s in this case.

I managed to bisect this down to cbd8b195e80f54f9b398456d434e39d9d23c2bdd, but have not managed to
figure out how to fix the problem yet, without breaking
anything else.

syslog-ng 3.3 is not affected, but the last s_logfile() statement doesn't work there.

--

-- 
Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

headers

Gergely Nagy | 8 May 2013 12:52

[PATCH (3.5)] afsocket: Fix compilation with systemd support enabled

self->super.socket_options->sock_type is replaced with
self->super.transport_mapper->sock_type, follow up on this in the
systemd-enabled codepath too.

Signed-off-by: Gergely Nagy <algernon <at> balabit.hu>
---
 modules/afsocket/afunix-source.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/modules/afsocket/afunix-source.c b/modules/afsocket/afunix-source.c
index dd8b8bb..c3dcbe7 100644
--- a/modules/afsocket/afunix-source.c
+++ b/modules/afsocket/afunix-source.c
 <at>  <at>  -74,7 +74,7  <at>  <at>  afunix_sd_acquire_named_socket(AFSocketSourceDriver *s, gint *result_fd,
 	  if (sd_is_socket_unix(fd, 0, -1, filename, 0))
 	    {
 	      /* check if it matches our idea of the socket type */
-	      if (sd_is_socket_unix(fd, self->super.socket_options->sock_type, -1, filename, 0))
+	      if (sd_is_socket_unix(fd, self->super.transport_mapper->sock_type, -1, filename, 0))
                 {
                   *result_fd = fd;
                   break;
 <at>  <at>  -84,7 +84,7  <at>  <at>  afunix_sd_acquire_named_socket(AFSocketSourceDriver *s, gint *result_fd,
                   msg_error("The systemd supplied UNIX domain socket is of a different type, check the configured driver
and the matching systemd unit file",
 		            evt_tag_str("filename", filename),
 		            evt_tag_int("systemd-sock-fd", fd),
-			    evt_tag_str("expecting", self->super.socket_options->sock_type == SOCK_STREAM ?
"unix-stream()" : "unix-dgram()"),
+			    evt_tag_str("expecting", self->super.transport_mapper->sock_type == SOCK_STREAM ?
"unix-stream()" : "unix-dgram()"),
                             NULL);
                   return FALSE;
                 }
--

-- 
1.7.10.4

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

61	May 2013
107	April 2013
52	March 2013
166	February 2013
135	January 2013
113	December 2012
114	November 2012
161	October 2012
133	September 2012
168	August 2012
103	July 2012
130	June 2012
163	May 2012
129	April 2012
190	March 2012
176	February 2012
198	January 2012
166	December 2011
290	November 2011
184	October 2011
191	September 2011
175	August 2011
171	July 2011
181	June 2011
159	May 2011
203	April 2011
225	March 2011
267	February 2011
254	January 2011
206	December 2010
188	November 2010
206	October 2010
227	September 2010
116	August 2010
185	July 2010
113	June 2010
57	May 2010
108	April 2010
109	March 2010
84	February 2010
97	January 2010
147	December 2009
122	November 2009
98	October 2009
96	September 2009
92	August 2009
161	July 2009
108	June 2009
86	May 2009
111	April 2009
121	March 2009
154	February 2009
145	January 2009
54	December 2008
108	November 2008
112	October 2008
90	September 2008
74	August 2008
89	July 2008
85	June 2008
83	May 2008
86	April 2008
89	March 2008
121	February 2008
146	January 2008
157	December 2007
145	November 2007
88	October 2007
101	September 2007
157	August 2007
142	July 2007
84	June 2007
105	May 2007
106	April 2007
128	March 2007
112	February 2007
155	January 2007
96	December 2006
110	November 2006
142	October 2006
120	September 2006
97	August 2006
87	July 2006
111	June 2006
129	May 2006
128	April 2006
184	March 2006
73	February 2006
93	January 2006
76	December 2005
85	November 2005
197	October 2005
83	September 2005
98	August 2005
87	July 2005
142	June 2005
144	May 2005
125	April 2005
127	March 2005
158	February 2005
119	January 2005
112	December 2004
101	November 2004
109	October 2004
96	September 2004
86	August 2004
85	July 2004
83	June 2004
121	May 2004
90	April 2004
99	March 2004
98	February 2004
92	January 2004
57	December 2003
85	November 2003
50	October 2003
75	September 2003
76	August 2003
60	July 2003
122	June 2003
29	May 2003

Mon	Tue	Wed	Thu	Fri	Sat	Sun

		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31