Matthew Hall | 1 Feb 03:10 2011
Picon

(no subject)

From: Matthew Hall <mhall <at> mhcomputing.net>
Date: Mon, 31 Jan 2011 18:00:01 -0800
Subject: [PATCH] adding JSON support to 3.2 (if allowable)

Hello Balint, Gergely, Bazsi, and list,

Here I present my backport of tfjson support to 3.2 for consideration of 
inclusion if this is allowd for that branch.

Regards,
Matthew Hall.

Matthew Hall (1):
  tfjson: $(format_json) template function implementation.

 configure.in               |   27 ++++-
 modules/Makefile.am        |    3 +-
 modules/tfjson/Makefile.am |   11 ++
 modules/tfjson/tfjson.c    |  305 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 344 insertions(+), 2 deletions(-)
 create mode 100644 modules/tfjson/Makefile.am
 create mode 100644 modules/tfjson/tfjson.c

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Matthew Hall | 1 Feb 03:10 2011
Picon

(no subject)

From 36ccc97227e1d4b8dcf9cea0ead0662851cb7732 Mon Sep 17 00:00:00 2001
From: Matthew Hall <mhall <at> mhcomputing.net>
Date: Mon, 31 Jan 2011 17:43:28 -0800
Subject: [PATCH] tfjson: $(format_json) template function implementation.

Implements a $(format_json) template function, largely based on the
work of Balint Kovacs <blint <at> balabit.hu>, built on top of the json-c
library.

The syntax of the template function is as follows:

  $(format_json [--select <GLOB>] [--exclude <GLOB>]
                [--skip-builtins]
                [MACRO_NAMES...] KEY=VALUE)

There can be multiple selects, excludes, macro names and key=value
pairs. The VALUE part of the key=value pair can contain template
macros.

Macro names must be listed without the "$" sign, however.

If --skip-builtins is specified, all built-in variables will be
excluded.

As an example, this is a valid argument list (without the line
breaks):

  $(format_json --select .classification.* --select useracct.*
                --exclude *.*id HOST my_own_key='$SOURCEIP ($HOST)')

(Continue reading)

Matthew Hall | 1 Feb 03:24 2011
Picon

[PATCH] adding JSON support to 3.2 (if allowable)

Hello Balint, Gergely, Bazsi, and list,

Here I present my backport of tfjson support to 3.2 for consideration of 
inclusion if this is allowd for that branch.

The last send went bad because I am new to git format-patch. Sorry!

Regards,
Matthew Hall.

Matthew Hall (1):
  tfjson: $(format_json) template function implementation.

 configure.in               |   27 ++++-
 modules/Makefile.am        |    3 +-
 modules/tfjson/Makefile.am |   11 ++
 modules/tfjson/tfjson.c    |  305 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 344 insertions(+), 2 deletions(-)
 create mode 100644 modules/tfjson/Makefile.am
 create mode 100644 modules/tfjson/tfjson.c

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Matthew Hall | 1 Feb 03:29 2011
Picon

[PATCH] tfjson: $(format_json) template function implementation.

Implements a $(format_json) template function, largely based on the
work of Balint Kovacs <blint <at> balabit.hu>, built on top of the json-c
library.

The syntax of the template function is as follows:

  $(format_json [--select <GLOB>] [--exclude <GLOB>]
                [--skip-builtins]
                [MACRO_NAMES...] KEY=VALUE)

There can be multiple selects, excludes, macro names and key=value
pairs. The VALUE part of the key=value pair can contain template
macros.

Macro names must be listed without the "$" sign, however.

If --skip-builtins is specified, all built-in variables will be
excluded.

As an example, this is a valid argument list (without the line
breaks):

  $(format_json --select .classification.* --select useracct.*
                --exclude *.*id HOST my_own_key='$SOURCEIP ($HOST)')

This will include every key that begins with ".classification." or
"useracct.", but will exclude any, that has a dot, and ends with
"id". The latter keys will be excluded even if they match any of the
select patterns.

(Continue reading)

Guillaume Rousse | 1 Feb 15:46 2011
Picon

Impossibility to use multiple occurences of the same parser

Hello list.

My syslog-ng configuration is modular, so as to be easily deployed on
all our hosts. There is a main configuration files, and one additional
included file by service. Our LDAP and Kerberos servers are hosted on
the same host (LDAP is our Kerberos backend).

Main file has:
parser p_db {
    db-parser();
};
filter f_drop {
    tags("dropthis");
};
destination d_drop {
};

LDAP file has:
log {
    source(s_sys);
    filter(f_ldap);
    parser(p_db);

    log {
        filter(f_drop);
        destination(d_drop);
        flags(final);
    };

    log {
(Continue reading)

Gergely Nagy | 1 Feb 16:11 2011
Picon

Re: [RFC]: value_pairs() demo

On Sat, 2011-01-29 at 07:40 -0800, Evan Rempel wrote:
> > destination d_mongo {
> >   mongodb(
> >     value_pairs(builtins(no) select("*") exclude(".classifier.rule_id")
> >                 "$HOST" "$MESSAGE"
> >                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
> >   );
> > };
> > 
> > And this will do exactly what it says: skip builtins, select everything
> > that is left, and exclude ".classifier.rule_id" from that, and then add
> > a few extra stuff on our own.
> 
> I think that the "builtin(no)" option should be abandon in favour of something else.

In my opinion, it'd be better to clarify what builtin() is for. At the
moment, there's a short list of builtin macros:

HOST, HOST_FROM, MESSAGE, PROGRAM, PID, MSGID, SOURCE, LEGACY_MSGHDR
(defined in lib/logmsg.c), and there's a few standard macros, like
$UNIXTIME.

By default, the standard macros that are not part of the builtins, will
not be included unless explicitly requested, which is a shame, and
that's what makes builtins() confusing, imo.

If builtins() dealt with the standard macros, it'd be much easier - and
I plan to figure out how to do just that. That will also affect select()
and exclude() too.

(Continue reading)

Lay, James | 1 Feb 16:14 2011

Re: Upgrade gotchas?

This upgrade went fine with some minor issues.  Thanks for all the great assistance from the folks on this list.

 

James

 

From: syslog-ng-bounces <at> lists.balabit.hu [mailto:syslog-ng-bounces <at> lists.balabit.hu] On Behalf Of Lay, James
Sent: Friday, January 28, 2011 3:26 PM
To: syslog-ng <at> lists.balabit.hu
Subject: [syslog-ng] Upgrade gotchas?

 

Hey All!

 

I’m about to do an upgrade from 3.0.1, to 3.2.2.  Any surprises or gotchas I should be aware of?  I’ve done some searching online and didn’t see too much in the way of what I needed to do, if anything, besides ./configure make and make install of eventlog and syslog-ng.  My config below is what I am workin with:

 

<at> version:3.0

options { use_dns(no); };

 

source s_local {

        unix-stream("/dev/log");

        udp(ip(0.0.0.0) port(514));

        tcp(ip(0.0.0.0) port(50000));

        file("/proc/kmsg");

};

 

destination d_file {

        file("/syslogs/messages");

};

 

log {

        source(s_local);

        destination(d_file);

};

 

Anything besides changing the version number?  Thanks all.

 

James

 

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Evan Rempel | 1 Feb 17:29 2011
Picon
Picon

Re: [RFC]: value_pairs() demo

Gergely Nagy wrote:
> On Sat, 2011-01-29 at 07:40 -0800, Evan Rempel wrote:
>>> destination d_mongo {
>>>   mongodb(
>>>     value_pairs(builtins(no) select("*") exclude(".classifier.rule_id")
>>>                 "$HOST" "$MESSAGE"
>>>                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
>>>   );
>>> };
>>>
>>> And this will do exactly what it says: skip builtins, select everything
>>> that is left, and exclude ".classifier.rule_id" from that, and then add
>>> a few extra stuff on our own.
>> I think that the "builtin(no)" option should be abandon in favour of something else.
> 
> In my opinion, it'd be better to clarify what builtin() is for. At the
> moment, there's a short list of builtin macros:
> 
> HOST, HOST_FROM, MESSAGE, PROGRAM, PID, MSGID, SOURCE, LEGACY_MSGHDR
> (defined in lib/logmsg.c), and there's a few standard macros, like
> $UNIXTIME.
> 
> By default, the standard macros that are not part of the builtins, will
> not be included unless explicitly requested, which is a shame, and
> that's what makes builtins() confusing, imo.
> 
> If builtins() dealt with the standard macros, it'd be much easier - and
> I plan to figure out how to do just that. That will also affect select()
> and exclude() too.
> 
> Perhaps it can be renamed to builtin-macros() then?
> 
>> It is really nothing more than a power-select or power-exclude but it does not
>> honour the order requirement of the select/exclude options.
> 
> Yep, and that's by design. There's a priority among the selectors:
> explicit selects ("$HOST", "$MESSAGE" and key-value pairs) are the
> highest, followed by builtins() and select()/exclude() on the lowest
> priority.
> 
> Thus, if one turns builtins() off, one can still explicitly add
> key-value pairs that use builtin stuff. Likewise, if any builtins are
> excluded, they can still be explicitly added, however, since builtins()
> has higher priority than select()/exclude(), if they're turned off,
> select()/exclude() will not see them at all.

This is a good example of why things are so confusing.
The paragraph above is contradictory within itself.

On one hand you state that if "one builtins() off, once can still explictitly add
key-valued pairs" and on the other hand you state "if they're turned off,
select()/exclude() will not see them at all".

I realize that you need to add the builtins using something other than select, but
that seems confusing too.

>> In the above example
>> you have excluded the built in macros but then used a select("*") which implies adding
>> everything back in. If you had done these in the oposite order, what semantic would 
>> be intended.
> 
> That's due to the explicit > builtins() > select/exclude priority order.
> 
>> It is unclear to me what is defined as a builtin macro and which ones are not.
> 
> Indeed, it is unclear - even to me. I plan to fix that, though (see
> above).
> 
>> It is also unclear where the $UNIXTIME came from since it was not shown at all
>> in the example that apparently incleded everything.
> 
> Yep, unfortunately the way macros and builtins are handled in syslog-ng
> is a bit... unclear, and chaotic. I'm trying to figure out an easy way
> to fix this, and make builtins() include all of the built-in macros,
> including $UNIXTIME and the rest.
> 
>> Perhaps just relying on the select/exclude (which should probably be renamed to include/exclude)
>> would be sufficient since in most cases at least some of the builtin macros will be desired and
>> like in your example where you included the $HOST and $MESSAGE it would have been almost
>> as easy to merely exclude the others by name and not use the builtin option at all.
> 
> The problem with that, is that there's no other easy way to exclude all
> of the builtin macros, which might be preferable in some cases.

I am not so much concerned with making the process of excluding all of the
built in macros easy, but am much more concerned with making the syntax consistent,
deterministic and obvious so that when writing, on probably more importantly for trouble
shooting, when reading the configuration file.

Creating a system where there is a priority of options, and then having
the order define the priority for some of the options makes a system that is more easily
misunderstood.

destination d_mongo {
   mongodb(
     value_pairs(builtins(no) select("*") exclude(".classifier.rule_id")
                 "$HOST" "$MESSAGE"
                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
   );
};

would seem to be very different from

destination d_mongo {
   mongodb(
     value_pairs(select("*") exclude(".classifier.rule_id")
                 "$HOST" "$MESSAGE"
                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
                 builtins(no)
   );
};

but given the proposal that "builtins(no)" has highest priority,
these two examples actually have the same semantics.

And this just won't do what you expect either.

destination d_mongo {
   mongodb(
     value_pairs(select("*") exclude(".classifier.rule_id")
                 select("HOST") select("MESSAGE")
                 ("PROGRAM" "$PROGRAM[$PID]") ("TIMESTAMP" "$UNIXTIME"))
                 builtins(no)
   );
};

> Thanks a lot for the detailed feedback by the way, it's most
> appreciated!

That's nice. Sometimes I am told I am argumentative :-(

--

-- 
Evan Rempel
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Sergey Senozhatsky | 1 Feb 17:34 2011
Picon

2.6.38: CAP_SYSLOG

Hello,

During 2.6.38 development CAP_SYSLOG has been introduced to perform syslog 
operations, older CAP_SYS_ADMIN is not sufficient anymore.

commit 38ef4c2e437d11b5922723504b62824e96761459
commit ce6ada35bdf710d16582cc4869c26722547e6f11

do_syslog now is as follows:

int do_syslog(int type, char __user *buf, int len, bool from_file)
{
	[..]
        if (type == SYSLOG_ACTION_OPEN || !from_file) {
                if (dmesg_restrict && !capable(CAP_SYSLOG))
                        goto warn; /* switch to return -EPERM after 2.6.39 */
                if ((type != SYSLOG_ACTION_READ_ALL &&
                     type != SYSLOG_ACTION_SIZE_BUFFER) &&
                    !capable(CAP_SYSLOG))
                        goto warn; /* switch to return -EPERM after 2.6.39 */
        }
	[..]

CAP_SYSLOG introduced to libcap in version 2.20.

#define CAP_SYSLOG           34

	Sergey
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

Gergely Nagy | 1 Feb 18:20 2011
Picon

Re: 2.6.38: CAP_SYSLOG

On Tue, 2011-02-01 at 18:34 +0200, Sergey Senozhatsky wrote: 
> Hello,
> 
> During 2.6.38 development CAP_SYSLOG has been introduced to perform syslog 
> operations, older CAP_SYS_ADMIN is not sufficient anymore.

It's a known issue, but no suitable solution (that doesn't break in
interesting ways under pressure) has been found yet.

Since CAP_SYSLOG breaks userspace, I'm hoping that this will be reverted
before the 2.6.38 release. Though, looking at recent lkml traffic, I'll
have to Cc a few more people regarding the issue.

--

-- 
|8]

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html


Gmane