Balazs Scheidler | 1 Mar 10:50 2012
Picon

Bazsi's blog: Project Lumberjack to improve Linux logging


Project Lumberjack to improve Linux logging

In a lively discussion at the RedHat offices two weeks ago in Brno, a number of well respected individuals were discussing how logging in general, and Linux logging in particular could be improved. As you may have guessed I was invited because of syslog-ng, but representatives of other logging related projects were also in nice numbers: Steve Gibbs (auditd), Lennart Poettering (systemd, journald), Rainer Gerhards (rsyslog), William Heinbockel (CEE, Mitre) and a number of nice people from the RedHat team.

We discussed a couple of pain points for logging, logging is usually an afterthought during development, computer based processing, correllation of application logs is nearly impossible. We roughly agreed that the key to improve the situation is to involve the community at large, initiate a momentum and try to get application developers on board and have them create structured logs. We also agreed that this will not happen overnight and we need to take a gradual approach.

To move into that direction, the benefits of good logging needs to be communicated and delivered to both application developers and their users.

We also talked about what kind of building blocks are needed to deliver a solution fast, and concluded that we basically have everything available, and even better they are open source. The key is to tie these components together, document best practices and perhaps provide better integration.

Thus project Lumberjack was born, hosted as a Fedora project at https://fedorahosted.org/lumberjack/.

The building blocks that need some care are:

  • some applications already produce logs in structured format, those should be integrated (auditd for instance)
  • we need to define a mechanism to submit structured logs to local logging services  for further processing (ELAPI and some enhanced syslog)
  • we need to make sure that local logging services cope with structured data (already available for a long time now)
  • we need to define a mechanism to store messages in a structured form and a way query them
  • last, but not least we need to define a naming scheme for event data which CEE can bring to the table

Most of these is already possible by using a combination of tools and proper configuration, however learning how to do this is not a trivial undertaking for those who only want to develop or use applications.

Changing that is the primary aim of Project Lumberjack. If you are interested in logging, make sure to check that out.

 

 



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Jonathan Kaufman | 1 Mar 16:21 2012

Problem with Syslog-NG 3.2.5 on Aix 7.1 ... It coredumps..


Hello All,
	I am hoping someone out there has a working installation of 3.2.5 on
Aix 7.1 and can help me get mine working.

I am currently trying to use a self-compiled version of 3.2.5 on an Power7
CPU Aix 7.1 TL1 SP3 box. I was able to get Syslog-NG to compile on Aix 7.1
after I mangled a new libtool release into the source so it would compile
shared objects.

Unfortunately, when I start syslog-ng it core dumps repeatedly.

Core dumps start filling up /sftw/syslog-ng/var and a ps -ef  shows only
the "supervising syslog-ng process".

a dbx syslog-ng gives this output...

[root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > dbx syslog-ng
Type 'help' for help.
[using memory image in core.10092546.01150742]
reading symbolic information ...

Segmentation fault in alloca at 0xd2970930 ($t1)
0xd2970930 (alloca+0x8) 800c0000         lwz   r0,0x0(r12)
(dbx) where
alloca() at 0xd2970930
evt_str_append_escape_bs() at 0xd2970520
evtrec_format_plain() at 0xd296feb4
evt_format() at 0xd2970104
msg_event_send(e = 0x00000005), line 166 in "messages.c"
main_loop_run(cfg = (nil)), line 148 in "main.c"
main(argc = 1, argv = 0x2ff227f4), line 470 in "main.c"
(dbx)

I have also tried using the Syslog-ng rpm's from perzl & Bull. Both of them
core dump as well, so I went back to trying to get a self-compiled release
to work as I have some control over that.

I used IBM VisualAge C/C++ v11.1.0.9 as the C compiler.

[root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > lslpp -L | grep vac
  vac.Bnd                   11.1.0.1    C     F    XL C for AIX Media
Defined
  vac.C                     11.1.0.9    C     F    IBM XL C Compiler
  vac.aix53.lib             11.1.0.9    C     F    XL C for AIX Libraries
for AIX
  vac.include               11.1.0.9    C     F    IBM XL C Compiler
Include
  vac.lib                   11.1.0.9    C     F    XL C for AIX Libraries
  vacpp.Bnd                 11.1.0.1    C     F    IBM XL C/C++ Media
Defined
  vacpp.cmp.aix53.lib       11.1.0.9    C     F    IBM XL C/C++ Libraries
for AIX
  vacpp.cmp.core            11.1.0.9    C     F    IBM XL C/C++ Compiler
  vacpp.cmp.include         11.1.0.9    C     F    IBM XL C/C++ Compiler
Include
  vacpp.cmp.lib             11.1.0.9    C     F    IBM XL C/C++ Libraries
  vacpp.cmp.rte             11.1.0.9    C     F    IBM XL C/C++ Compiler
  vacpp.cmp.tools           11.1.0.9    C     F    IBM XL C/C++ Tools
  vacpp.tnb                 11.1.0.1    C     F    IBM XL C/C++ Evaluation

I mangled libtool 2.4.2 into the source directories so it would
detect/compile shared libraries (it wouldn't otherwise).

I compiled and installed eventlog 0.2.12 in /sftw/syslog-ng, I also
compiled OpenSSL 1.0.0g into it staticly.

I am initially trying to start it using the default configuration files.

There were no errors during the compile, and a syslog-ng -s did NOT
coredump.

I believe syslog-ng is using the following libraries and their locations
(dynamically linked).

[root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > ldd syslog-ng
syslog-ng needs:
         /sftw/syslog-ng/lib/libsyslog-ng.a(libsyslog-ng.so.0)
         /usr/lib/libnsl.a(shr.o)
         /opt/freeware/lib/libgmodule-2.0.so
         /opt/freeware/lib/libglib-2.0.so
         /usr/lib/libpthread.a(shr_xpg5.o)
         /usr/lib/libc.a(shr.o)
         /sftw/syslog-ng/lib/libevtlog.a(libevtlog.so.0)
         /usr/lib/librtl.a(shr.o)
         /opt/freeware/lib/libpcre.a(libpcre.so.0)
         /usr/lib/libthread.a(shr.o)
         /usr/lib/libpthreads_compat.a(shr.o)
         /usr/lib/libpthreads.a(shr_xpg5.o)
         /usr/lib/libtli.a(shr.o)
         /opt/freeware/lib/libglib-2.0.a(libglib-2.0.so.0)
         /opt/freeware/lib/libintl.a(libintl.so.1)
         /usr/lib/libiconv.a(shr4.o)
         /usr/lib/libpthreads.a(shr_comm.o)
         /unix
         /usr/lib/libcrypt.a(shr.o)
         /usr/lib/libpthreads.a(shr.o)
         /usr/lib/libc.a(pse.o)

and the configure command when compiling from source was:

./configure --prefix=/sftw/syslog-ng \
        --disable-spoof-source \
        --enable-dynamic-linking \
        --enable-debug \
        --enable-ssl

and I added -g to the CFLAGS

I am using glib2-2.28.6-1 from perzl as well..
I plan on removing the --enable-debug if I can get it working

Any ideas?

Jonathan Kaufman

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Sandor Geller | 1 Mar 18:02 2012

Re: Problem with Syslog-NG 3.2.5 on Aix 7.1 ... It coredumps..

I had the same issue using Sun Studio for building 64-bit binaries on
Solaris. Could you apply this patch to eventlog, rebuild it and retry?

--- src/evtstr.c-orig   2010-12-03 14:44:25.000000000 +0100
+++ src/evtstr.c        2010-12-03 14:45:21.000000000 +0100
 <at>  <at>  -48,6 +48,7  <at>  <at> 
 #ifdef _MSC_VER
 #include <malloc.h>
 #endif
+#include <alloca.h>

 /* event string handling */

On Thu, Mar 1, 2012 at 3:21 PM, Jonathan Kaufman
<jkaufman <at> footlocker.com> wrote:
>
> Hello All,
>        I am hoping someone out there has a working installation of 3.2.5 on
> Aix 7.1 and can help me get mine working.
>
>
> I am currently trying to use a self-compiled version of 3.2.5 on an Power7
> CPU Aix 7.1 TL1 SP3 box. I was able to get Syslog-NG to compile on Aix 7.1
> after I mangled a new libtool release into the source so it would compile
> shared objects.
>
> Unfortunately, when I start syslog-ng it core dumps repeatedly.
>
>
> Core dumps start filling up /sftw/syslog-ng/var and a ps -ef  shows only
> the "supervising syslog-ng process".
>
> a dbx syslog-ng gives this output...
>
> [root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > dbx syslog-ng
> Type 'help' for help.
> [using memory image in core.10092546.01150742]
> reading symbolic information ...
>
> Segmentation fault in alloca at 0xd2970930 ($t1)
> 0xd2970930 (alloca+0x8) 800c0000         lwz   r0,0x0(r12)
> (dbx) where
> alloca() at 0xd2970930
> evt_str_append_escape_bs() at 0xd2970520
> evtrec_format_plain() at 0xd296feb4
> evt_format() at 0xd2970104
> msg_event_send(e = 0x00000005), line 166 in "messages.c"
> main_loop_run(cfg = (nil)), line 148 in "main.c"
> main(argc = 1, argv = 0x2ff227f4), line 470 in "main.c"
> (dbx)
>
>
> I have also tried using the Syslog-ng rpm's from perzl & Bull. Both of them
> core dump as well, so I went back to trying to get a self-compiled release
> to work as I have some control over that.
>
> I used IBM VisualAge C/C++ v11.1.0.9 as the C compiler.
>
> [root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > lslpp -L | grep vac
>  vac.Bnd                   11.1.0.1    C     F    XL C for AIX Media
> Defined
>  vac.C                     11.1.0.9    C     F    IBM XL C Compiler
>  vac.aix53.lib             11.1.0.9    C     F    XL C for AIX Libraries
> for AIX
>  vac.include               11.1.0.9    C     F    IBM XL C Compiler
> Include
>  vac.lib                   11.1.0.9    C     F    XL C for AIX Libraries
>  vacpp.Bnd                 11.1.0.1    C     F    IBM XL C/C++ Media
> Defined
>  vacpp.cmp.aix53.lib       11.1.0.9    C     F    IBM XL C/C++ Libraries
> for AIX
>  vacpp.cmp.core            11.1.0.9    C     F    IBM XL C/C++ Compiler
>  vacpp.cmp.include         11.1.0.9    C     F    IBM XL C/C++ Compiler
> Include
>  vacpp.cmp.lib             11.1.0.9    C     F    IBM XL C/C++ Libraries
>  vacpp.cmp.rte             11.1.0.9    C     F    IBM XL C/C++ Compiler
>  vacpp.cmp.tools           11.1.0.9    C     F    IBM XL C/C++ Tools
>  vacpp.tnb                 11.1.0.1    C     F    IBM XL C/C++ Evaluation
>
> I mangled libtool 2.4.2 into the source directories so it would
> detect/compile shared libraries (it wouldn't otherwise).
>
> I compiled and installed eventlog 0.2.12 in /sftw/syslog-ng, I also
> compiled OpenSSL 1.0.0g into it staticly.
>
> I am initially trying to start it using the default configuration files.
>
> There were no errors during the compile, and a syslog-ng -s did NOT
> coredump.
>
> I believe syslog-ng is using the following libraries and their locations
> (dynamically linked).
>
> [root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > ldd syslog-ng
> syslog-ng needs:
>         /sftw/syslog-ng/lib/libsyslog-ng.a(libsyslog-ng.so.0)
>         /usr/lib/libnsl.a(shr.o)
>         /opt/freeware/lib/libgmodule-2.0.so
>         /opt/freeware/lib/libglib-2.0.so
>         /usr/lib/libpthread.a(shr_xpg5.o)
>         /usr/lib/libc.a(shr.o)
>         /sftw/syslog-ng/lib/libevtlog.a(libevtlog.so.0)
>         /usr/lib/librtl.a(shr.o)
>         /opt/freeware/lib/libpcre.a(libpcre.so.0)
>         /usr/lib/libthread.a(shr.o)
>         /usr/lib/libpthreads_compat.a(shr.o)
>         /usr/lib/libpthreads.a(shr_xpg5.o)
>         /usr/lib/libtli.a(shr.o)
>         /opt/freeware/lib/libglib-2.0.a(libglib-2.0.so.0)
>         /opt/freeware/lib/libintl.a(libintl.so.1)
>         /usr/lib/libiconv.a(shr4.o)
>         /usr/lib/libpthreads.a(shr_comm.o)
>         /unix
>         /usr/lib/libcrypt.a(shr.o)
>         /usr/lib/libpthreads.a(shr.o)
>         /usr/lib/libc.a(pse.o)
>
> and the configure command when compiling from source was:
>
> ./configure --prefix=/sftw/syslog-ng \
>        --disable-spoof-source \
>        --enable-dynamic-linking \
>        --enable-debug \
>        --enable-ssl
>
> and I added -g to the CFLAGS
>
> I am using glib2-2.28.6-1 from perzl as well..
> I plan on removing the --enable-debug if I can get it working
>
>
> Any ideas?
>
>
> Jonathan Kaufman
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Mary A Waddick | 1 Mar 19:20 2012

using syslog-ng and parsing data from both Windows XP and Windows 7 machines

Hi,

I am using nxlog to send data from both Windows XP and Windows 7 machines to a Unix machine using syslog-ng .

My nxlog.conf files are configured to send im_mseventlog data for the Windows XP boxes and im_msvistalog data for the Windows 7 boxes.

(See attached file: new nxlog.conf)

Therefore I get slightly different data for each machine. The examples on your website don't show me how to parse out all of the im_mseventlog or im_msvistalog data from the different columns in msg. Can you help me with getting the data?

My syslog-ng.conf looks like this. I was told that the eventlog data would have the columns that I included in my table, but I am unable to figure out how to pull that data out of the msg column using the provided macros. I have searched and searched, but have not found any examples.


(See attached file: syslog-ng.conf)

Thank you for your help.


    Mary Anne Waddick
    Raytheon Technical Services
    Senior Software Engineer II
    (317) 306-2691 (desk)
Attachment (new nxlog.conf): application/octet-stream, 1314 bytes
Attachment (syslog-ng.conf): application/octet-stream, 1528 bytes
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Jonathan Kaufman | 1 Mar 20:03 2012

Re: Problem with Syslog-NG 3.2.5 on Aix 7.1 ... It coredumps..

Awesome!

Thank you very much.

Just in case anyone else is working on Aix 7.1 builds of Syslog-NG here is
what worked (or didn't) for me.

	1.  Update to the LATEST version of IBM VisualAge C/C++, I couldn't
get it to compile until I updated to the 2012 patch set (I was at the early
2011 patch set) for V11
	2.  There were also a few Aix APARs that would have affected
compiling code dealing with the assembler or whatnot. I sidestepped them
and upgraded to the latest TL and SP (TL1 SP3)
	3.  I couldn't get  the bundled libtool that is included with the
Syslog-NG source to recognize that you can create shared libraries, so I
couldn't create shared libraries for the modules (that was an issue)
		I "upgraded" the bundled libtool to 2.4.2 which fixed it so I
could created shared libraries for the modules. The make install didn't
copy them, so I had to do it manually but at least it works.
		I would expect this isn't the preferred method for fixing this,
but so far it doesn't seem to have a downside.
	4.  I had the libiconv library from perzl.org loaded for a different
application, and if Syslog-NG used that either during compile or runtime it
would coredump. Moral of the story seems to be to use the IBM libiconv
library.
	5.  And lastly  Sandor's suggestion.  Add the alloca.h include to
evtstr.c
		if you are using the IBM VisualAge compiler you may be able to
use the -ma compiler option. I don't know if it does the same thing, but if
I used the option (and didn't add it to the source) syslog-ng didn't core.
		I will likely just add it to the source, but thought to mention
the -ma option *seems* to work just the same.

Hopefully this will be the last of my instability woes on Aix 7.1 with
Syslog-NG.

Jonathan Kaufman

|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Sandor Geller <Sandor.Geller <at> morganstanley.com>                                                                                                   |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"Syslog-ng users' and developers' mailing list" <syslog-ng <at> lists.balabit.hu>                                                                      |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |03/01/2012 11:03 AM                                                                                                                               |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: [syslog-ng] Problem with Syslog-NG 3.2.5 on Aix 7.1 ... It	coredumps..                                                                     |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Sent by:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |syslog-ng-bounces <at> lists.balabit.hu                                                                                                                |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|

I had the same issue using Sun Studio for building 64-bit binaries on
Solaris. Could you apply this patch to eventlog, rebuild it and retry?

--- src/evtstr.c-orig   2010-12-03 14:44:25.000000000 +0100
+++ src/evtstr.c        2010-12-03 14:45:21.000000000 +0100
 <at>  <at>  -48,6 +48,7  <at>  <at> 
 #ifdef _MSC_VER
 #include <malloc.h>
 #endif
+#include <alloca.h>

 /* event string handling */

On Thu, Mar 1, 2012 at 3:21 PM, Jonathan Kaufman
<jkaufman <at> footlocker.com> wrote:
>
> Hello All,
>        I am hoping someone out there has a working installation of 3.2.5
on
> Aix 7.1 and can help me get mine working.
>
>
> I am currently trying to use a self-compiled version of 3.2.5 on an
Power7
> CPU Aix 7.1 TL1 SP3 box. I was able to get Syslog-NG to compile on Aix
7.1
> after I mangled a new libtool release into the source so it would compile
> shared objects.
>
> Unfortunately, when I start syslog-ng it core dumps repeatedly.
>
>
> Core dumps start filling up /sftw/syslog-ng/var and a ps -ef  shows only
> the "supervising syslog-ng process".
>
> a dbx syslog-ng gives this output...
>
> [root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > dbx syslog-ng
> Type 'help' for help.
> [using memory image in core.10092546.01150742]
> reading symbolic information ...
>
> Segmentation fault in alloca at 0xd2970930 ($t1)
> 0xd2970930 (alloca+0x8) 800c0000         lwz   r0,0x0(r12)
> (dbx) where
> alloca() at 0xd2970930
> evt_str_append_escape_bs() at 0xd2970520
> evtrec_format_plain() at 0xd296feb4
> evt_format() at 0xd2970104
> msg_event_send(e = 0x00000005), line 166 in "messages.c"
> main_loop_run(cfg = (nil)), line 148 in "main.c"
> main(argc = 1, argv = 0x2ff227f4), line 470 in "main.c"
> (dbx)
>
>
> I have also tried using the Syslog-ng rpm's from perzl & Bull. Both of
them
> core dump as well, so I went back to trying to get a self-compiled
release
> to work as I have some control over that.
>
> I used IBM VisualAge C/C++ v11.1.0.9 as the C compiler.
>
> [root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > lslpp -L | grep vac
>  vac.Bnd                   11.1.0.1    C     F    XL C for AIX Media
> Defined
>  vac.C                     11.1.0.9    C     F    IBM XL C Compiler
>  vac.aix53.lib             11.1.0.9    C     F    XL C for AIX Libraries
> for AIX
>  vac.include               11.1.0.9    C     F    IBM XL C Compiler
> Include
>  vac.lib                   11.1.0.9    C     F    XL C for AIX Libraries
>  vacpp.Bnd                 11.1.0.1    C     F    IBM XL C/C++ Media
> Defined
>  vacpp.cmp.aix53.lib       11.1.0.9    C     F    IBM XL C/C++ Libraries
> for AIX
>  vacpp.cmp.core            11.1.0.9    C     F    IBM XL C/C++ Compiler
>  vacpp.cmp.include         11.1.0.9    C     F    IBM XL C/C++ Compiler
> Include
>  vacpp.cmp.lib             11.1.0.9    C     F    IBM XL C/C++ Libraries
>  vacpp.cmp.rte             11.1.0.9    C     F    IBM XL C/C++ Compiler
>  vacpp.cmp.tools           11.1.0.9    C     F    IBM XL C/C++ Tools
>  vacpp.tnb                 11.1.0.1    C     F    IBM XL C/C++ Evaluation
>
> I mangled libtool 2.4.2 into the source directories so it would
> detect/compile shared libraries (it wouldn't otherwise).
>
> I compiled and installed eventlog 0.2.12 in /sftw/syslog-ng, I also
> compiled OpenSSL 1.0.0g into it staticly.
>
> I am initially trying to start it using the default configuration files.
>
> There were no errors during the compile, and a syslog-ng -s did NOT
> coredump.
>
> I believe syslog-ng is using the following libraries and their locations
> (dynamically linked).
>
> [root <at> mlwitt71]:[/sftw/syslog-ng/sbin] > ldd syslog-ng
> syslog-ng needs:
>         /sftw/syslog-ng/lib/libsyslog-ng.a(libsyslog-ng.so.0)
>         /usr/lib/libnsl.a(shr.o)
>         /opt/freeware/lib/libgmodule-2.0.so
>         /opt/freeware/lib/libglib-2.0.so
>         /usr/lib/libpthread.a(shr_xpg5.o)
>         /usr/lib/libc.a(shr.o)
>         /sftw/syslog-ng/lib/libevtlog.a(libevtlog.so.0)
>         /usr/lib/librtl.a(shr.o)
>         /opt/freeware/lib/libpcre.a(libpcre.so.0)
>         /usr/lib/libthread.a(shr.o)
>         /usr/lib/libpthreads_compat.a(shr.o)
>         /usr/lib/libpthreads.a(shr_xpg5.o)
>         /usr/lib/libtli.a(shr.o)
>         /opt/freeware/lib/libglib-2.0.a(libglib-2.0.so.0)
>         /opt/freeware/lib/libintl.a(libintl.so.1)
>         /usr/lib/libiconv.a(shr4.o)
>         /usr/lib/libpthreads.a(shr_comm.o)
>         /unix
>         /usr/lib/libcrypt.a(shr.o)
>         /usr/lib/libpthreads.a(shr.o)
>         /usr/lib/libc.a(pse.o)
>
> and the configure command when compiling from source was:
>
> ./configure --prefix=/sftw/syslog-ng \
>        --disable-spoof-source \
>        --enable-dynamic-linking \
>        --enable-debug \
>        --enable-ssl
>
> and I added -g to the CFLAGS
>
> I am using glib2-2.28.6-1 from perzl as well..
> I plan on removing the --enable-debug if I can get it working
>
>
> Any ideas?
>
>
> Jonathan Kaufman
>
>
>
______________________________________________________________________________

> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
______________________________________________________________________________

Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

                    Visit us on-line at footlocker.com.

The information in this e-mail, and any attachment therein, is confidential
and for use by the addressee only. If you are not the intended recipient,
please return the e-mail to the sender and delete it from your computer.
Although the Company attempts to sweep e-mail and attachments for viruses,
it does not guarantee that either are virus-free and accepts no liability
for any damage sustained as a result of viruses.

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Daniel Neubacher | 2 Mar 10:40 2012

Losing to much remote sent logs

Hello there,

I’ve started playing around with syslog-ng 3.3.4 ose a few days ago but I’m still experiencing some trouble. First of all we want to use syslog-ng to send all of our logs via udp to a central syslog server. This includes of course syslogs, apache logs and custom generated applogs. These logs are generated from 400 clients and produces a minimum of 300 mio. log lines a day.

The problem is really simple: I’m losing log lines :P Most of the time everything goes well but when the logs are peaking high 1-5% logs are getting lost.

Last night the stats of the server and a client said 0 drops but when I counted the lines I found lost lines. The server has 24g ram & 8 cores and I can rule out a network problem for sure.

 

So now to my questions, has anyone else an idea where I can tweak my cfg or where I have to look to find more clues? Is tcp the only way to get around it?

I’ve attached my syslog server cfg. The so_rcvbuf buffer is the same size as the os net.core.rmem settings. And as described in the various balabit blog posts I played around with log_fetch_limit and flush_lines already.

 

syslog-ng.conf:

<at> version: 3.3

 

options {

    threaded(yes);

    owner("root");

    group("root");

    perm(0660);

 

    dir_owner("root");

    dir_group("root");

    dir_perm(0770);

    create_dirs(yes);

 

    stats_freq(600);

    stats_level(2);

    chain_hostnames(yes);

    normalize_hostnames(yes);

    check_hostname(yes);

 

    dns_cache(yes);

    dns_cache_size(16384);

    dns_cache_expire(3600);

    dns_cache_expire_failed(60);

 

    log_msg_size(16384);

    log_fifo_size(100000);

 

 

    use_fqdn(yes);

#disabled 4 debugging

#    flush_lines(200);

};

 

source s_src {

        unix-dgram("/dev/log");

        internal();

        file("/proc/kmsg" program_override("kernel"));

};

 

source s_net {

udp(

        log_fetch_limit(400)

        so_rcvbuf(51200000)

        keep_hostname(yes)

        keep_timestamp(no)     

        ip("10.8.4.10")                                

        port(514)  

);

tcp(

        so_rcvbuf(51200000)

        so_keepalive(yes)

        keep_hostname(no)

        keep_timestamp(no)

        ip("10.8.4.10")

        port(514)

 

);

syslog();

};

 

filter f_syslog {

     not program(access.log) and

     not program(error.log) and

     not program(beetle.log) and

     not program(edge.log);

 

};

 

filter f_apache {

    program(access.log) or

    program(error.log);

};

 

filter f_applogs {

    program(beetle.log)

    or program(edge.log);

};

 

template t_plain {

    template("$MSG\n"); template_escape(no);

};

 

destination d_messages { file("/var/log/messages"); };

destination d_remote { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };

destination d_apache { file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); };

destination d_applogs { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); };

 

log {

    source(s_src);

    destination(d_messages);

};

 

log {

    source(s_net);

    filter(f_syslog);

    destination(d_remote);

};

 

log {

    source(s_net);

    filter(f_apache);

    destination(d_apache);

};

 

log {

    source(s_net);

    filter(f_applogs);

    destination(d_applogs);

};

 

 

Thanks

Daniel Neubacher

 

Attachment (syslog-ng.conf): application/octet-stream, 2959 bytes
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Jenei Gábor | 2 Mar 13:12 2012
Picon
Picon

start problems

Hello,

I've already adopted the recommended init script for debian, but still 
my syslog-ng 3.4 doesn't seem to work. At the moment actually I can run 
my compiled program just by entering ./syslog-ng, however it does not 
seem to have access to files, because I don't see any trace in my log 
files of starting. If I try to start the program by its init script it 
waits for about a minute, and then says: failed. However after init 
script there are still running instances. Can you tell me what my 
problem is?

Gabor
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

José Moreno | 2 Mar 14:40 2012
Picon

Packet fragmentation issue

Hi all,

I'm running syslog-ng 2.4.1, log sources send to a log server which beside keeping the original data as is in
files, forwards them in real time to a SIEM, spoofing source IP.

My problem comes after some logs are too long to fit in a single frame, log server fragments those packets
when sending them to SIEM and spoofing is not performed for them.

Enviado desde mi iPhone
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

José Moreno | 2 Mar 14:48 2012
Picon

Re: Packet fragmentation issue

Sorry, my previous message went out unfinished and I see I've placed it as an answer to someone else's question.

I just wanted to add that I was posting because I had not seen this issue in the list; Sorry if I'm wrong.

Thanks very much in advance.
Kind regards.

Enviado desde mi iPhone

El 02/03/2012, a las 14:40, José Moreno <jmorenoa <at> gmail.com> escribió:

> Hi all,
> 
> I'm running syslog-ng 2.4.1, log sources send to a log server which beside keeping the original data as is
in files, forwards them in real time to a SIEM, spoofing source IP.
> 
> My problem comes after some logs are too long to fit in a single frame, log server fragments those packets
when sending them to SIEM and spoofing is not performed for them.
> 
> Enviado desde mi iPhone
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Martin Holste | 2 Mar 15:59 2012
Picon

Re: Losing to much remote sent logs

If possible, I would try swapping the $HOST macro for $SOURCEIP to
avoid doing any DNS lookups, cached or not.  It's unlikely to help,
but it sounds like you've already tried the basic tuning things.  I
will say that I'm very surprised you're losing log lines.  What is
your peak logs per second, and how long are the peaks?

On Fri, Mar 2, 2012 at 3:40 AM, Daniel Neubacher
<daniel.neubacher <at> xing.com> wrote:
> Hello there,
>
> I’ve started playing around with syslog-ng 3.3.4 ose a few days ago but I’m
> still experiencing some trouble. First of all we want to use syslog-ng to
> send all of our logs via udp to a central syslog server. This includes of
> course syslogs, apache logs and custom generated applogs. These logs are
> generated from 400 clients and produces a minimum of 300 mio. log lines a
> day.
>
> The problem is really simple: I’m losing log lines :P Most of the time
> everything goes well but when the logs are peaking high 1-5% logs are
> getting lost.
>
> Last night the stats of the server and a client said 0 drops but when I
> counted the lines I found lost lines. The server has 24g ram & 8 cores and I
> can rule out a network problem for sure.
>
>
>
> So now to my questions, has anyone else an idea where I can tweak my cfg or
> where I have to look to find more clues? Is tcp the only way to get around
> it?
>
> I’ve attached my syslog server cfg. The so_rcvbuf buffer is the same size as
> the os net.core.rmem settings. And as described in the various balabit blog
> posts I played around with log_fetch_limit and flush_lines already.
>
>
>
> syslog-ng.conf:
>
>  <at> version: 3.3
>
>
>
> options {
>
>     threaded(yes);
>
>     owner("root");
>
>     group("root");
>
>     perm(0660);
>
>
>
>     dir_owner("root");
>
>     dir_group("root");
>
>     dir_perm(0770);
>
>     create_dirs(yes);
>
>
>
>     stats_freq(600);
>
>     stats_level(2);
>
>     chain_hostnames(yes);
>
>     normalize_hostnames(yes);
>
>     check_hostname(yes);
>
>
>
>     dns_cache(yes);
>
>     dns_cache_size(16384);
>
>     dns_cache_expire(3600);
>
>     dns_cache_expire_failed(60);
>
>
>
>     log_msg_size(16384);
>
>     log_fifo_size(100000);
>
>
>
>
>
>     use_fqdn(yes);
>
> #disabled 4 debugging
>
> #    flush_lines(200);
>
> };
>
>
>
> source s_src {
>
>         unix-dgram("/dev/log");
>
>         internal();
>
>         file("/proc/kmsg" program_override("kernel"));
>
> };
>
>
>
> source s_net {
>
> udp(
>
>         log_fetch_limit(400)
>
>         so_rcvbuf(51200000)
>
>         keep_hostname(yes)
>
>         keep_timestamp(no)
>
>         ip("10.8.4.10")
>
>         port(514)
>
> );
>
> tcp(
>
>         so_rcvbuf(51200000)
>
>         so_keepalive(yes)
>
>         keep_hostname(no)
>
>         keep_timestamp(no)
>
>         ip("10.8.4.10")
>
>         port(514)
>
>
>
> );
>
> syslog();
>
> };
>
>
>
> filter f_syslog {
>
>      not program(access.log) and
>
>      not program(error.log) and
>
>      not program(beetle.log) and
>
>      not program(edge.log);
>
>
>
> };
>
>
>
> filter f_apache {
>
>     program(access.log) or
>
>     program(error.log);
>
> };
>
>
>
> filter f_applogs {
>
>     program(beetle.log)
>
>     or program(edge.log);
>
> };
>
>
>
> template t_plain {
>
>     template("$MSG\n"); template_escape(no);
>
> };
>
>
>
> destination d_messages { file("/var/log/messages"); };
>
> destination d_remote {
> file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };
>
> destination d_apache {
> file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
> destination d_applogs {
> file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
>
>
> log {
>
>     source(s_src);
>
>     destination(d_messages);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_syslog);
>
>     destination(d_remote);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_apache);
>
>     destination(d_apache);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_applogs);
>
>     destination(d_applogs);
>
> };
>
>
>
>
>
> Thanks
>
> Daniel Neubacher
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


Gmane