Hi Folks,
I'm trying to run a test to check insert rates.
If I run this command:

./loggen -r 5000 -D -I 10 127.0.0.1 514

The output shows:
average rate = 5441.60 msg/sec, count=54420, time=10.007, msg size=256, bandwidth=1360.40 kB/sec

But, my stats don't show that many messages received:

syslog-ng[6660]: Log statistics; dropped=\'pipe(/dev/xconsole)=0\', processed=\'center(queued)=24232\', processed=\'center(received)=8077, processed=\'destination(df_logzilla)=8077\'

As you can see, it sent 55k messages, but I only received 8k.
Am I doing something wrong?

Here are my options in the syslog-ng config:
options {
      long_hostnames(off);
      log_msg_size(8192);
      flush_lines(1);
      log_fifo_size(16384);
      time_reopen(10);
      use_dns(yes);
      dns_cache(yes);
      use_fqdn(yes);
      keep_hostname(yes);
      chain_hostnames(no);
      perm(0644);
     stats_freq(60);

};


--
______________________________________________________________

Clayton Dukes
______________________________________________________________

Hi Folks,
I'm trying to run a test to check insert rates.
If I run this command:

./loggen -r 5000 -D -I 10 127.0.0.1 514

The output shows:
average rate = 5441.60 msg/sec, count=54420, time=10.007, msg size=256, bandwidth=1360.40 kB/sec

But, my stats don't show that many messages received:

syslog-ng[6660]: Log statistics; dropped=\'pipe(/dev/xconsole)=0\', processed=\'center(queued)=24232\', processed=\'center(received)=8077, processed=\'destination(df_logzilla)=8077\'

As you can see, it sent 55k messages, but I only received 8k.
Am I doing something wrong?

Here are my options in the syslog-ng config:
options {
      long_hostnames(off);
      log_msg_size(8192);
      flush_lines(1);
      log_fifo_size(16384);
      time_reopen(10);
      use_dns(yes);
      dns_cache(yes);
      use_fqdn(yes);
      keep_hostname(yes);
      chain_hostnames(no);
      perm(0644);
     stats_freq(60);

};


--
______________________________________________________________

Clayton Dukes
______________________________________________________________
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

I should have mentioned that this is logging directly to a file.

destination df_logzilla {
   file("/var/log/logzilla/syslog.log"
   template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
   );
};

What do you get if you send the loggen data to a simple netcat session with its output redirected to a flat file?  Do you see all 55k messages using wc -l?

Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax.  Namely, using tags(".source.log2") in your filter.

Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax.  Namely, using tags(".source.log2") in your filter.

On Wed, Mar 31, 2010 at 12:57 PM, Zoltán Pallagi <pzolee <at> balabit.hu> wrote:

Hi,

I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng.

However, I can suggest you an other solution:
use the program_override option. This will override the $PROGRAM macro with the specified value.
For example:
source s_app {
file("/var/log/log1.log" program_override("/var/log/log1.log"));
file("/opt//log/log2.log" tags("log2") program_override("/opt/log/log2.log"));
file("/opt/log/log3.log" tags("log3") program_override("/opt/log/log3.log"));
};

After that, you can use a specified program filter on the central logging server side to separate them.

2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:

Hi all,

I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the

source system and want to separate them again on the central logging server.

The client configuration looks like this:

----------------------------------------------------------------------------------------------------

source s_app {
file("/var/log/log1.log");
file("/opt//log/log2.log" tags("log2"));
file("/opt/log/log3.log" tags("log3"));
};

options {
};

destination d_app { tcp("logrelay01.domain.com" port(13074)); };

log {
source(s_app);
destination(d_app);
};
----------------------------------------------------------------------------------------------------

 

The log relay does nothing than forward the messages to the central logging server with the following config:

----------------------------------------------------------------------------------------------------

options {
time_sleep(20);
log_fifo_size(1000);
dns_cache(2000);
dns_cache_expire(87600);
keep_hostname(yes);
};

source s_remote { tcp(ip("0.0.0.0") port(13074)); };

 

destination remote_tcp { tcp("centrallog01.domain.com" port(13074)); };

 

log {
source(s_remote);
destination(remote_tcp);
};
----------------------------------------------------------------------------------------------------

 

On the central logging server I use filters to separate the logfiles again:

----------------------------------------------------------------------------------------------------

<at> version: 3.0

include "/opt/config/syslogng-inc.conf";

 

options {
time_sleep(20);
dns_cache(2000);
dns_cache_expire(87600);
keep_hostname(yes);
create_dirs(yes);
};

source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); };

============================================

 

This is the confoguration in /opt/config/syslogng-inc.conf

# Filter
filter f_log1 { host("web00(09|10)"); };
filter f_log2 { host("web00(09|10)") and tags("log2"); };
filter f_log3 { host("web00(09|10)") and tags("log3"); };

 

#Configuration for Destinations

destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); };
destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); };
destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); };

# Logfile log1
log {
source(s_remote);
filter(f_log1);
destination(d_log1);
};

 

# Logfile log2
log {
source(s_remote);
filter(f_log2);
destination(d_log2);
};

 

# Logfile log3

log {
source(s_remote);
filter(f_log3);
destination(d_log3);
};

----------------------------------------------------------------------------------------------------

 

Does anybody have an idea, why it does not work as expected.

 

Thanks for any suggestion and/or idea.

 

Best regards Stefan

 

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

--
pzolee

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

Ok, so '.sources' has nothing to do with user-provided tags, but in his example Marci uses:

source s_tcp2 {
tcp(ip(192.168.1.2) port(1514) tags("tcp", "windows));
};

#Match on tags "tcp" or "udp"
filter f_net {
tags("tcp", "udp");
};

Which seems to imply that arbitrary user tags can be set and then matched on in a filter later.  So, it seems what is missing from Stefan's config was:

source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes) tags("log2"); };

Which would allow his later filter statement

filter f_log2 { host("web00(09|10)") and tags("log2"); };

to succeed.  Right?

What do you get if you send the loggen data to a simple netcat session with its output redirected to a flat file?  Do you see all 55k messages using wc -l?

On Thu, Apr 1, 2010 at 6:51 AM, Clayton Dukes <cdukes <at> gmail.com> wrote:

I should have mentioned that this is logging directly to a file.

destination df_logzilla {
   file("/var/log/logzilla/syslog.log"
   template("$HOST\t$FACILITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
   );
};

On Wed, Mar 31, 2010 at 11:47 PM, Clayton Dukes <cdukes <at> gmail.com> wrote:

Hi Folks,
I'm trying to run a test to check insert rates.
If I run this command:

./loggen -r 5000 -D -I 10 127.0.0.1 514

The output shows:
average rate = 5441.60 msg/sec, count=54420, time=10.007, msg size=256, bandwidth=1360.40 kB/sec

But, my stats don't show that many messages received:

syslog-ng[6660]: Log statistics; dropped=\'pipe(/dev/xconsole)=0\', processed=\'center(queued)=24232\', processed=\'center(received)=8077, processed=\'destination(df_logzilla)=8077\'

As you can see, it sent 55k messages, but I only received 8k.
Am I doing something wrong?

Here are my options in the syslog-ng config:
options {
      long_hostnames(off);
      log_msg_size(8192);
      flush_lines(1);
      log_fifo_size(16384);
      time_reopen(10);
      use_dns(yes);
      dns_cache(yes);
      use_fqdn(yes);
      keep_hostname(yes);
      chain_hostnames(no);
      perm(0644);
     stats_freq(60);

};


--
______________________________________________________________

Clayton Dukes
______________________________________________________________

--
______________________________________________________________

Clayton Dukes
______________________________________________________________

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

Ok, so '.sources' has nothing to do with user-provided tags, but in his example Marci uses:

source s_tcp2 {
tcp(ip(192.168.1.2) port(1514) tags("tcp", "windows));
};

#Match on tags "tcp" or "udp"
filter f_net {
tags("tcp", "udp");
};

Which seems to imply that arbitrary user tags can be set and then matched on in a filter later.  So, it seems what is missing from Stefan's config was:

source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes) tags("log2"); };

Which would allow his later filter statement

filter f_log2 { host("web00(09|10)") and tags("log2"); };

to succeed.  Right?

On Thu, Apr 1, 2010 at 8:37 AM, Zoltán Pallagi <pzolee <at> balabit.hu> wrote:

Martin Holste wrote:

Please step in and correct me if I'm wrong here, but according to Marci's blog post at http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html it would appear that this is possible using different syntax.  Namely, using tags(".source.log2") in your filter.

No, you are using a wrong tag name.
In this case, you can use the following tag:
tags(".source.s_app")

this is an on-the-fly generated tag,every incoming message has it one given with the following formula: ".source.<sourcename>"

On Wed, Mar 31, 2010 at 12:57 PM, Zoltán Pallagi <pzolee <at> balabit.hu> wrote:

Hi,

I'm afraid that you may misunderstand the working of this feature. The tag field exists only within a running syslog-ng and just a virtual part of the message. The sent message doesn't contain tag fields that's why you cannot filter these tags with another syslog-ng.

However, I can suggest you an other solution:
use the program_override option. This will override the $PROGRAM macro with the specified value.
For example:
source s_app {
file("/var/log/log1.log" program_override("/var/log/log1.log"));
file("/opt//log/log2.log" tags("log2") program_override("/opt/log/log2.log"));
file("/opt/log/log3.log" tags("log3") program_override("/opt/log/log3.log"));
};

After that, you can use a specified program filter on the central logging server side to separate them.

2010.03.31. 16:39 keltezéssel, Hoenig, Stefan, VF-Group írta:

Hi all,

I got a problem to get the "tags" feature working on our syslog-ng 3.1. I want to collect messages from 3 different files on the

source system and want to separate them again on the central logging server.

The client configuration looks like this:

----------------------------------------------------------------------------------------------------

source s_app {
file("/var/log/log1.log");
file("/opt//log/log2.log" tags("log2"));
file("/opt/log/log3.log" tags("log3"));
};

options {
};

destination d_app { tcp("logrelay01.domain.com" port(13074)); };

log {
source(s_app);
destination(d_app);
};
----------------------------------------------------------------------------------------------------

 

The log relay does nothing than forward the messages to the central logging server with the following config:

----------------------------------------------------------------------------------------------------

options {
time_sleep(20);
log_fifo_size(1000);
dns_cache(2000);
dns_cache_expire(87600);
keep_hostname(yes);
};

source s_remote { tcp(ip("0.0.0.0") port(13074)); };

 

destination remote_tcp { tcp("centrallog01.domain.com" port(13074)); };

 

log {
source(s_remote);
destination(remote_tcp);
};
----------------------------------------------------------------------------------------------------

 

On the central logging server I use filters to separate the logfiles again:

----------------------------------------------------------------------------------------------------

<at> version: 3.0

include "/opt/config/syslogng-inc.conf";

 

options {
time_sleep(20);
dns_cache(2000);
dns_cache_expire(87600);
keep_hostname(yes);
create_dirs(yes);
};

source s_remote { tcp (ip("0.0.0.0") port(13074) keep-alive(yes)); };

============================================

 

This is the confoguration in /opt/config/syslogng-inc.conf

# Filter
filter f_log1 { host("web00(09|10)"); };
filter f_log2 { host("web00(09|10)") and tags("log2"); };
filter f_log3 { host("web00(09|10)") and tags("log3"); };

 

#Configuration for Destinations

destination d_log1 { file("/var/logs/log1/combined.log" perm(0755) dir_perm(0755)); };
destination d_log2 { file("/var/logs/log2/combined.log" perm(0755) dir_perm(0755)); };
destination d_log3 { file("/var/logs/log3/combined.log" perm(0755) dir_perm(0755)); };

# Logfile log1
log {
source(s_remote);
filter(f_log1);
destination(d_log1);
};

 

# Logfile log2
log {
source(s_remote);
filter(f_log2);
destination(d_log2);
};

 

# Logfile log3

log {
source(s_remote);
filter(f_log3);
destination(d_log3);
};

----------------------------------------------------------------------------------------------------

 

Does anybody have an idea, why it does not work as expected.

 

Thanks for any suggestion and/or idea.

 

Best regards Stefan

 

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

--
pzolee

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

-- pzolee

______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

Hello,

I'm using the syslog-ng application version 3.1.0. I would like to change the severity and/or the facility of an event.

Here is the configuration for this test :

template t_format {

        template ("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC;$HOST;$FACILITY;$PRIORITY;$PROGRAM;$MSG\n");

};

source s_local_test {

        unix-stream("/dev/log");

};

destination d_test {

        file ( "/tmp/test"

                template (t_format)

        );

};

rewrite r_rewrite_set {

        set (

                "my_program"

                value("PROGRAM")

        );

};

log {

        source (s_local_test);

        rewrite (r_rewrite_set);

        destination (d_test);

};

I generated a message by using "logger" :

# logger -i -p local0.info  "Test Message : ABCDEFGHIJKLMNOPQRST ## $(date) ##"

=> /tmp/test:

2010-04-01 15:12:14;pc-dev.dom;local0;info;MY_PROGRAM;Test Message : ABCDEFGHIJKLMNOPQRST ## jeu avr  1 15:12:14 CEST 2010 ##

Ok, I can change the PROGRAM name...

...So I changed the rule "r_rewrite_set" to modify the value to "facility" :

rewrite r_rewrite_set {

        set (

                "local4"

                value("FACILITY")

        );

};

And when I restarted the syslog-ng, I had the following errors :

# /etc/init.d/syslog-ng restart

Macros are read-only, they cannot be changed in rewrite rules, falling back to MESSAGE instead; macro='FACILITY'

Restarting syslog-ng: Stopping syslog-ng:                  [  OK  ]

Starting syslog-ng: Macros are read-only, they cannot be changed in rewrite rules, falling back to MESSAGE instead; macro='FACILITY'

                                                           [  OK  ]

I have the same error when I changed the "PRIORITY" of the rule "r_rewrite_set" :

> Macros are read-only, they cannot be changed in rewrite rules, falling back to MESSAGE instead; macro='PRIORITY'

I tried to send a message by using "logger" but as it written in the above error, the MESSAGE was changed...

My questions are :

   1. Which macros are read-only ? PRIORITY, FACILITY, another ?

   2. Is there a way to change the facility or the priority of an event ?

   3. If not, why can't I change the facility or the priority ?

Regards,

Yann