K K | 1 Jul 02:51 2008
Picon

Re: migrate over to PCRE?

I prefer PCRE, have no problem converting to PCRE only.

I've been told PCRE runs faster, a definite consideration for
high volume sites. We've gone out of our way to simplify regexes
to avoid performance hits, with a faster expression parser,
 I could back down on optimizations.

What other modules do you foresee?

Kevin
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Balazs Scheidler | 1 Jul 14:07 2008
Picon

Re: Latest RHEL5 build doesn't contains fix for PID

On Tue, 2008-07-01 at 14:47 +0300, Kostyantyn Gushtin wrote:
> Hi,
> I just download and try latest build 2.1beta2
> (syslog-ng-2.1beta1-1.i386.rpm , libdbi8-0.8.2bb2-2.i386.rpm). 
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Are you sure you downloaded beta2? Can you show me the file size,
modification date, etc?

> But I have the same problem with PID macro as before, it contains non
> valid integer values(some text). I try syslog-ng rpm's for RHEL5.

--

-- 
Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Kostyantyn Gushtin | 1 Jul 15:31 2008
Picon

Re: Latest RHEL5 build doesn't contains fix for PID

Balazs Scheidler wrote:
On Tue, 2008-07-01 at 14:47 +0300, Kostyantyn Gushtin wrote:
Hi, I just download and try latest build 2.1beta2 (syslog-ng-2.1beta1-1.i386.rpm , libdbi8-0.8.2bb2-2.i386.rpm).
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Are you sure you downloaded beta2? Can you show me the file size, modification date, etc?
But I have the same problem with PID macro as before, it contains non valid integer values(some text). I try syslog-ng rpm's for RHEL5.
I download it again and install on clean machine but result was the same.
Some info from package :

Name        : syslog-ng                    Relocations: (not relocatable)
Version     : 2.1beta1                          Vendor: Balabit IT Ltd.
Release     : 1                             Build Date: Fri 18 Apr 2008 07:31:08 PM EEST
Install Date: (not installed)               Build Host: sun64.balabit
Group       : System Environment/Daemons    Source RPM: syslog-ng-2.1beta1-1.src.rpm
Size        : 643256                           License: GPL
Signature   : (none)
Packager    : Tamas Pal <folti <at> balabit.com>
URL         : http://www.balabit.com
Summary     : Next generation system logging daemon
Description :
 Syslog-ng is a next generation system logger daemon which provides more
 capabilities and is has a more flexible configuration then the traditional
 syslog daemon.

But I think now the problem is that there is no beta2 package for RHEL-5.
For example for RHEL-4 there is syslog-ng-2.1beta2-1.i386.rpm, but for RHEL-5 there is syslog-ng-2.1beta1-1.i386.rpm
So I downloaded old version beta1 that has this bug. So, can you build new version for RHEL-5 ? (I mean beta2 or syslog-ng-2.1beta2-1.i386.rpm for Red Hat Enterprise Linux 5)
--
Kostyantyn Gushtin
Software engineer

Solutions for Your success
http://www.n-ix.com
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Balazs Scheidler | 1 Jul 19:27 2008
Picon

Re: Latest RHEL5 build doesn't contains fix for PID

On Tue, 2008-07-01 at 16:31 +0300, Kostyantyn Gushtin wrote:
> Balazs Scheidler wrote: 
> > On Tue, 2008-07-01 at 14:47 +0300, Kostyantyn Gushtin wrote:
> >   
> > > Hi,
> > > I just download and try latest build 2.1beta2
> > > (syslog-ng-2.1beta1-1.i386.rpm , libdbi8-0.8.2bb2-2.i386.rpm). 
> > >     
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > 
> > Are you sure you downloaded beta2? Can you show me the file size,
> > modification date, etc?
> > 
> >   
> > > But I have the same problem with PID macro as before, it contains non
> > > valid integer values(some text). I try syslog-ng rpm's for RHEL5.
> > >     
> > 
> > 
> > 
> > 
> >   
> I download it again and install on clean machine but result was the
> same. 
> Some info from package :

> But I think now the problem is that there is no beta2 package for
> RHEL-5.
> For example for RHEL-4 there is syslog-ng-2.1beta2-1.i386.rpm, but for
> RHEL-5 there is syslog-ng-2.1beta1-1.i386.rpm
> So I downloaded old version beta1 that has this bug. So, can you build
> new version for RHEL-5 ? (I mean beta2 or
> syslog-ng-2.1beta2-1.i386.rpm for Red Hat Enterprise Linux 5)

Hmm.. let me check. You are right, the i386 compilation failed, and I
didn't notice. The amd64 binary was compiled without problems.

I queued another build job, and now it compiled fine. It is being
uploaded to our website, it usually takes a couple of minutes.

--

-- 
Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Joe Shaw | 1 Jul 19:52 2008

Odd behavior with very small log_fifo_size()

Hi,

I was doing a little bit of testing with syslog-ng today, and I think
that with a very small log_fifo_size() it was dropping messages but
not telling me.

If I set my log_fifo_size to 1, here are the results:

[jshaw <at> ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ ./loggen -r 1000000
-s 1024000 -I 60 localhost 2000
average rate = 96.48 msg/sec, count=5789
[jshaw <at> ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ ls -lh /ita/jshaw/messages
-rw-------  1 jshaw ita 101M Jul  1 13:30 /ita/jshaw/messages

However, 1024000 * 5789 = 5.9 GB

Grepping for drop messages in the output file:

[jshaw <at> ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ grep -i drop
/ita/jshaw/messages

returns nothing.  I have stats_freq(5) in my config file, and that
messages file is the only output.  I do have other statistical lines,
which imply to me that no messages are dropped:

Jul  1 13:49:52 s_local <at> ars1dev7 syslog-ng[6344]: Log statistics;
processed='center(queued)=6045', processed='center(received)=6045',
processed='destination(d_file)=6045', processed='source(s_local)=6045'

But it seems clear to me that they are -- or loggen is lying.

With more normal log_fifo_sizes (100, 2000) the throughput rates from
loggen are lower and the file sizes match.

Any ideas?

Thanks,
Joe
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Ed Ravin | 2 Jul 00:19 2008
Picon

syslog-ng-2.1beta1 on NetBSD 3.0, duplicate suppression docs?

I encountered two problems building syslog-ng-2.1beta1 on NetBSD 3.0:

1) I needed to patch loggen.c (stole this patch from a FreeBSD site,
apparently it happens on that OS too)

--- tests/loggen/loggen.c.orig  2008-01-13 00:04:22.000000000 -0800
+++ tests/loggen/loggen.c       2008-03-26 08:27:43.000000000 -0700
 <at>  <at>  -226,7 +226,9  <at>  <at> 
           memset(&hints, 0, sizeof(hints));
           hints.ai_family = AF_UNSPEC;
           hints.ai_socktype = sock_type;
+#ifdef AI_ADDRCONFIG
           hints.ai_flags = AI_ADDRCONFIG;
+#endif
           hints.ai_protocol = 0;
           if (getaddrinfo(argv[optind], argv[optind + 1], &hints, &res) != 0)
             {
-------------------------------------

2) I built syslog-ng with --enable-tcp-wrappers, which needlessly put
"-lwrap" on the LIBS line in the Makefiles for programs that did not
actually use TCP wrappers.  This causes link-time errors, since
libwrap requires you to declare two symbols in your program.  The fix
was to run this in the source directory:

    find tests/ -name Makefile -exec perl -p -i -e 's/-lwrap//' {} ";"

-------------------------------------

I also have a couple of observations/questions:

3) It turns out I really wanted syslog-ng2.1beta2, but although that
was announced as released, only the beta1 tarball is actually on the
source download site.  I had to go to the snapshots.  The above
fixes were also needed for the beta2 sources.

-------------------------------------

4) I'm looking for info on how to use the duplicate suppression
option - it doesn't seem to be in the docs included in the source.
Can someone give me a quick summary?  This was (IMHO) the only
extant flaw in syslog-ng and I'm looking forward to trying it out!

Thanks,

	-- Ed
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

chris packham | 2 Jul 01:04 2008
Picon

Re: syslog-ng-2.1beta1 on NetBSD 3.0, duplicate suppression docs?

Hi Ed,

On Tue, 2008-07-01 at 18:19 -0400, Ed Ravin wrote:
> 4) I'm looking for info on how to use the duplicate suppression
> option - it doesn't seem to be in the docs included in the source.
> Can someone give me a quick summary?  This was (IMHO) the only
> extant flaw in syslog-ng and I'm looking forward to trying it out!

You just need to add suppress(<seconds>) to your destination config.

e.g. to suppress duplicate messages that occur within 60 seconds

destination d_file { file("/var/log/messages" template(t_logfile)
suppress(60)); };

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Balazs Scheidler | 2 Jul 11:13 2008
Picon

Re: Odd behavior with very small log_fifo_size()

On Tue, 2008-07-01 at 13:52 -0400, Joe Shaw wrote:
> Hi,
> 
> I was doing a little bit of testing with syslog-ng today, and I think
> that with a very small log_fifo_size() it was dropping messages but
> not telling me.
> 
> If I set my log_fifo_size to 1, here are the results:
> 
> [jshaw <at> ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ ./loggen -r 1000000
> -s 1024000 -I 60 localhost 2000
> average rate = 96.48 msg/sec, count=5789
> [jshaw <at> ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ ls -lh /ita/jshaw/messages
> -rw-------  1 jshaw ita 101M Jul  1 13:30 /ita/jshaw/messages
> 
> However, 1024000 * 5789 = 5.9 GB
> 
> Grepping for drop messages in the output file:
> 
> [jshaw <at> ars1dev7 ~/syslog-ng-2.0.9/tests/loggen]$ grep -i drop
> /ita/jshaw/messages
> 
> returns nothing.  I have stats_freq(5) in my config file, and that
> messages file is the only output.  I do have other statistical lines,
> which imply to me that no messages are dropped:
> 
> Jul  1 13:49:52 s_local <at> ars1dev7 syslog-ng[6344]: Log statistics;
> processed='center(queued)=6045', processed='center(received)=6045',
> processed='destination(d_file)=6045', processed='source(s_local)=6045'
> 
> But it seems clear to me that they are -- or loggen is lying.
> 
> With more normal log_fifo_sizes (100, 2000) the throughput rates from
> loggen are lower and the file sizes match.

syslog-ng does not log message loss on file destinations, because that'd
clutter the log statistics line, on the asssumption that no messages are
can be lost there.

However this is not always the case, if you log_fifo_size() is smaller
than the window size of incoming sources, messages can be lost.

Here is a description of how flow-control and window sizes work in
syslog-ng:

http://www.balabit.hu/dl/html/syslog-ng-admin-guide_en.html/ch08s03.html

Please let me (and the doc team) know if the description could be made
clearer somehow.

--

-- 
Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Balazs Scheidler | 2 Jul 13:04 2008
Picon

Re: syslog-ng-2.1beta1 on NetBSD 3.0, duplicate suppression docs?

On Tue, 2008-07-01 at 18:19 -0400, Ed Ravin wrote:
> I encountered two problems building syslog-ng-2.1beta1 on NetBSD 3.0:
> 
> 1) I needed to patch loggen.c (stole this patch from a FreeBSD site,
> apparently it happens on that OS too)
> 
> --- tests/loggen/loggen.c.orig  2008-01-13 00:04:22.000000000 -0800
> +++ tests/loggen/loggen.c       2008-03-26 08:27:43.000000000 -0700
>  <at>  <at>  -226,7 +226,9  <at>  <at> 
>            memset(&hints, 0, sizeof(hints));
>            hints.ai_family = AF_UNSPEC;
>            hints.ai_socktype = sock_type;
> +#ifdef AI_ADDRCONFIG
>            hints.ai_flags = AI_ADDRCONFIG;
> +#endif
>            hints.ai_protocol = 0;
>            if (getaddrinfo(argv[optind], argv[optind + 1], &hints, &res) != 0)
>              {

Thanks. I've committed a fix for this issue.

> -------------------------------------
> 
> 2) I built syslog-ng with --enable-tcp-wrappers, which needlessly put
> "-lwrap" on the LIBS line in the Makefiles for programs that did not
> actually use TCP wrappers.  This causes link-time errors, since
> libwrap requires you to declare two symbols in your program.  The fix
> was to run this in the source directory:
> 
>     find tests/ -name Makefile -exec perl -p -i -e 's/-lwrap//' {} ";"

Hmm, all test programs link against ../../src/libsyslog-ng.a which
define these two symbols. I guess those symbols are not resolved from
the .a file, presumably because it is at an earlier position in the link
command line.

> 
> -------------------------------------
> 
> I also have a couple of observations/questions:
> 
> 3) It turns out I really wanted syslog-ng2.1beta2, but although that
> was announced as released, only the beta1 tarball is actually on the
> source download site.  I had to go to the snapshots.  The above
> fixes were also needed for the beta2 sources.

You are right. I've prepared the binary release (for those who have
binary subscription) and then forgot to actually do the source release.

I did that now. Sorry for the confusion.

> 
> -------------------------------------
> 
> 4) I'm looking for info on how to use the duplicate suppression
> option - it doesn't seem to be in the docs included in the source.
> Can someone give me a quick summary?  This was (IMHO) the only
> extant flaw in syslog-ng and I'm looking forward to trying it out!

You already got that from Chris, the original author of the feature.

--

-- 
Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Joe Shaw | 2 Jul 19:21 2008

Re: Odd behavior with very small log_fifo_size()

Hi,

On Wed, Jul 2, 2008 at 5:13 AM, Balazs Scheidler <bazsi <at> balabit.hu> wrote:
> syslog-ng does not log message loss on file destinations, because that'd
> clutter the log statistics line, on the asssumption that no messages are
> can be lost there.
>
> However this is not always the case, if you log_fifo_size() is smaller
> than the window size of incoming sources, messages can be lost.

Could something be added if -v were passed to syslog-ng then?  It's
(very) useful diagnostic information when you're tuning an
installation and I probably wouldn't have noticed if not for the data
that loggen outputs and the fact that the log file seemed small.

> Here is a description of how flow-control and window sizes work in
> syslog-ng:
>
> http://www.balabit.hu/dl/html/syslog-ng-admin-guide_en.html/ch08s03.html
>
> Please let me (and the doc team) know if the description could be made
> clearer somehow.

I hadn't seen this before -- I was only using the docs shipped with
the tarball -- and it's good information.

My setting of the log_fifo_size() in testing was for two reasons: (1)
to see how setting it affected memory usage when being hit with many
large messages and (2) to demonstrably see what its behavior was if it
was overloaded and had to drop messages.  I would have liked a little
more info in the latter.

As for the docs, it does mention that without the flow control
messages may be lost, but doesn't indicate how this might be detected
or what happens if they are.

Thanks,
Joe
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html


Gmane