Attila Szalai | 30 Jun 11:00 2016

getting some information about the sender process

Hi,

Maybe I'm totally wrong, but I have/had the impression that syslog-ng collects some information about the client process if that connects through a unix domain socket. But actually I could not find anything related to this neither in the documentation nor the code. Is this something that only exists in my mind or I just missed something?
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

David Campeau | 28 Jun 18:56 2016

Re: Syslog-ng Multiple Instances

Hello,

I've been using syslog-ng to filter syslog before forwarding on to a log collector. However, I need to spin up a second instance for testing purposes. I've found a little bit of information on-line, but it hasn't completed the entire picture.

This is the command used to start up the 2nd instance. I'm pointing to separate .conf .persist .pid and .ctl files -- However, it's still not working. I suspect the issue is due to OS log sources. How do a change log sources?

syslog-ng --cfgfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.conf --persist-file=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.persist --pidfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.pid --control=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.ctl &

This is the upper part of the syslog-ng.conf file for the 2nd instance I wish to run.

<at> version: 3.3.4
<at> include "scl.conf"
    options {
        time-reap(30);
        mark-freq(10);
        keep-hostname(yes);
        chain-hostnames(no);
        use-dns(no);
##       log-fifo-size(500000);                ## Tuning Options
##      flush_lines(10000);                   ## Tuning Options
##       flush_timeout(10000);                ## Tuning Options
    };

        source s_second_instance {
        syslog(transport("udp") port("518"));       #### Will receive test syslog on port 518
        };

    destination d_syslog_udp {
        syslog("10.X.X.X"
            transport("udp")
            port("514")
            throttle(4000)
        );
    };



I'm hoping someone has experience or has seen information on how to run a 2nd instance on the same box.

Best Regards,

David

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Hirose, Shinsaku | 24 Jun 16:10 2016

a log message is output in the two line

Hello, all,

I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6.
I got it from eple epel repository.

I am troubled in how to use the source file driver.
The touble is that a log message is output in the two lines on remote syslog server.

How to reproduce is followings.

1. Prepare two hosts running syslog-ng.

  Host_A configuration is followings.
  ----------------------------------------
  source test {
    file("/tmp/a.log");
  };
  destination d_remote { udp("192.168.0.2"); };
  log { source(test); destination(d_remote); };
  ----------------------------------------

  Host_B(192.168.0.2) configuration is defaults.

2.Execute following command on Host_A.

  $ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log

3.Check the log on Host_B.

  As the result, a log message is output in the two lines on Host_B.

  One line is following. The num of "a" is 8192.
  aaaaaaaaaaaaa....... 

  The other line is following. The num of "a" is 1.
  a

I hope a log message is output in the one line on Host_B.
Is my hope readily achievable?

Please advise me.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Czanik, Péter | 21 Jun 09:11 2016

CzP in Nuremberg

Hi,

For the next couple of days I'll be in Nuremberg for the openSUSE conference: https://events.opensuse.org/conference/oSC16 I'll give a presentation on syslog-ng Friday in the afternoon at 13:30 in the Roter Salon. I'll wear my syslog-ng superhero t-shirt, so I'll be easy to spot and ready to answer syslog-ng questions any time.

If you are in Nuremberg, but not at the conference, I'm happy to talk to you about syslog-ng also outside of the conference floor. You can reach me by e-mail or on twitter as <at> pczanik

See you soon in Nuremberg,

Peter

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Marco Mignone | 16 Jun 17:11 2016

Syslog-ng 3.7.3 + Elasticsearch V2.3 - Error init pipeline

Hi All,
I am trying to setup syslog-ng to use elasticsearch as its destination on Ubuntu 14.04.
This the version of syslog (the unofficial versions installed from laszlo_budai rep:

syslog-ng 3.7.3
Installer-Version: 3.7.3
Revision: 3.7.3-8
Compile-Date: Jun  1 2016 16:33:00
Available-Modules: basicfuncs,linux-kmsg-format,riemann,afuser,afstomp,afprog,json-plugin,afsmtp,affile,csvparser,mod-java,pseudofile,confgen,afsocket,afamqp,redis,sdjournal,kvformat,syslogformat,afsql,system-source,mod-python,graphite,dbparser,geoip-plugin,afmongodb,cryptofuncs

Elastic search is:

 "name" : "Theresa Cassidy",
 "cluster_name" : "elasticsearch",
 "version" : {
   "number" : "2.3.3",
   "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
   "build_timestamp" : "2016-05-17T15:40:04Z",
   "build_snapshot" : false,
   "lucene_version" : “5.5.0"


And my custom configuration in /etc/syslog-ng/conf.d/test.conf which is:

<at> module mod-java

source s_net {
 udp();
 tcp();
};

destination d_elastic {
 elasticsearch(
   index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
   type("test")
   client_lib_dir("/usr/share/elasticsearch/lib")
  );
};

log {
 source(s_net);
 destination(d_elastic);
 flags(flow-control);
};


When I try to launch syslog in debug mode this is what I get:

[2016-06-16T15:54:29.378356] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar;
[2016-06-16T15:54:29.382446] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar;
[2016-06-16T15:54:29.382660] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/kafka.jar;
[2016-06-16T15:54:29.382862] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/http.jar;
[2016-06-16T15:54:29.383052] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/hdfs.jar;
[2016-06-16T15:54:29.383258] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/elastic.jar;
[2016-06-16T15:54:29.383479] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-common.jar;
[2016-06-16T15:54:29.383670] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/log4j-1.2.16.jar;
[2016-06-16T15:54:29.383917] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar;
[2016-06-16T15:54:29.384098] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar;
[2016-06-16T15:54:29.384293] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar;
[2016-06-16T15:54:29.384494] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar;
[2016-06-16T15:54:29.386104] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar;
[2016-06-16T15:54:29.386342] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar;
[2016-06-16T15:54:29.386507] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar;
[2016-06-16T15:54:29.386677] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar;
[2016-06-16T15:54:29.386865] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar;
[2016-06-16T15:54:29.387044] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.6.6.jar;
[2016-06-16T15:54:29.387216] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.8.2.jar;
[2016-06-16T15:54:29.387394] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar;
[2016-06-16T15:54:29.387673] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar;
[2016-06-16T15:54:29.388476] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar;
[2016-06-16T15:54:29.388647] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar;
[2016-06-16T15:54:29.388818] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar;
[2016-06-16T15:54:29.388972] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar;
[2016-06-16T15:54:29.389518] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar;
[2016-06-16T15:54:29.389711] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar;
[2016-06-16T15:54:29.390094] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar;
[2016-06-16T15:54:29.390283] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar;
[2016-06-16T15:54:29.390488] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar;
[2016-06-16T15:54:29.390659] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar;
[2016-06-16T15:54:29.390935] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar;
[2016-06-16T15:54:29.391176] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar;
[2016-06-16T15:54:29.394616] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar;
[2016-06-16T15:54:29.395279] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar;
[2016-06-16T15:54:29.395458] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar;
[2016-06-16T15:54:29.395970] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar;
[2016-06-16T15:54:29.396734] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar;
[2016-06-16T15:54:29.397919] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar;
[2016-06-16T15:54:29.398106] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar;
[2016-06-16T15:54:29.398281] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar;
[2016-06-16T15:54:29.398440] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar;
[2016-06-16T15:54:29.398610] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar;
[2016-06-16T15:54:29.398784] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar;
[2016-06-16T15:54:29.398925] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar;
[2016-06-16T15:54:29.501879] Add path to classpath: /usr/lib/syslog-ng/3.7/java-modules/syslog-ng-core.jar;
[2016-06-16T15:54:29.519443] Error initializing message pipeline;


Was anyone able to make this work or suggest a way to fix this?
I hope I am on the right place and if not apologies in advance.

Thanks,
Marco

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Czanik, Péter | 16 Jun 15:15 2016

Insider 2016-06: Red Hat; 3.8 RPMs; BMW; MongoDB; FLARE; Java;

Dear syslog-ng users,

This is the 50th issue of syslog-ng Insider, a monthly newsletter that
brings you syslog-ng-related news.

NEWS

Meet syslog-ng at Red Hat Summit

--------------------------------

This year Balabit is sponsoring the Red Hat Summit again. The event
will be held in Moscone Center North in San Francisco, June 27-29.
Visit Balabit’s booth (booth #918) and meet Balázs Scheidler, the
author of syslog-ng. You can request an appointment at
http://pages2.balabit.com/red-hat-summit-appointment-request/

If you want to meet him outside of the conference floor, there will
also be a meetup in Café Prague on the 30th of June:
https://www.eventbrite.com/e/techie-happy-hour-optimize-your-log-data-in-the-world-of-big-data-tickets-25952864717

Preview syslog-ng 3.8 RPM packages and send us your feedback

------------------------------------------------------------

While syslog-ng 3.8 does not yet have an alpha release, it already has
many interesting features. As it is still under heavy development, we
can't recommend it for production use. On the other hand, any feedback
is very welcome. New features include disk buffer, Elasticsearch 2
support, grouping-by, Rust-based parsers, and so on:
https://czanik.blogs.balabit.com/2016/05/state-of-syslog-ng-3-8-rpm-packaging/

syslog-ng in the BMW i3

-----------------------

While most people know syslog-ng as a central syslog server, there is
another use, which is less known but most likely has a lot larger
installed base. It is syslog-ng embedded. Read how BMW is utilizing
syslog-ng at https://czanik.blogs.balabit.com/2016/06/embedded-syslog-ng-bmw-i3-all-electric-car/
and let us know how you use syslog-ng!

MongoDB changes in 3.8

----------------------

Under the hood, the driver to access MongoDB databases was replaced in
syslog-ng 3.8. Read about the advantages of the new driver and how it
affects configuration at:
https://syslog-ng.org/mongodb-destination-receives-face-lift/

FLARE: Filtered Log Alert and Reporting Engine

----------------------------------------------

FLARE staplies together syslog-ng, ELK, Alerting and Incident Response
with metadata. The aim was to develop a solution that is able to
provide a unified view of log events and incidents, helping the work
of operations admins with actionable insight. It is an internal
software developed at the University of Victoria by long-time members
of the syslog-ng community. You can read the complete presentation
about this syslog-ng & PatternDB-based solution at
https://www.bc.net/flare-stapling-together-syslog-ng-elk-alerting-and-incident-response-metadata

Writing syslog-ng Java destination drivers

------------------------------------------

If a destination is not supported by syslog-ng out of the box, you can
develop a new driver in C or one of the language bindings. Here are
the first steps for creating a Java-based destination driver:
https://vithulanmv.wordpress.com/2016/06/07/the-syslog-ng-java-destinations/

Your feedback and news, or tips about the next issue are welcome at
documentation <at> balabit.com . To read this newsletter online, visit:
http://insider.blogs.balabit.com/

Peter Czanik (CzP) <peter.czanik <at> balabit.com>
Balabit / syslog-ng upstream
http://czanik.blogs.balabit.com/
https://twitter.com/PCzanik
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Dmitry Shmulevich | 15 Jun 08:47 2016
Picon

Custom plugin/module

Hello,

I'm looking for an advice.

I'm sending the logs from my machine to a third-party server. The protocol between the client (syslog-ng service) and the third-party server is quite simple: for each received log entry the server replies with a confirmation message on the same tcp socket.
Currently I'm using a simple syslog-ng config to send logs via network driver to a specified ip/port. However, since syslog-ng service doesn't receive/read the messages from the server, the connection eventually dies.

Is there any way to modify syslog-ng config to allow reads after writes? I didn't find anything similar in the documentation.

An alternative way seems to be implementing a syslog-ng module or a plugin. Would that be the right approach?
It seems that I can customize _LogProtoClient structure, and create a plugin. But I have no clue on how the overall API looks like, and what are the steps to build the module.

Could you guys point me to the right resources?
Is there any sample plugin, or a guidance for developing custom plugins?

Thank you in advance,
Dmitry

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Czanik, Péter | 13 Jun 14:37 2016

Embedded syslog-ng: BMW i3 all-electric car

Hi,

A few years ago I wrote about syslog-ng running on all Kindle e-readers. Recently I found, that BMW also uses syslog-ng in the i3 all-electric car: https://czanik.blogs.balabit.com/2016/06/embedded-syslog-ng-bmw-i3-all-electric-car/

If you have another interesting use case, either embedded or the more traditional central server, I'd be glad to add it to our "Powered by syslog-ng" page at https://syslog-ng.org/powered-by-syslog-ng/

Bye,
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Evan Rempel | 11 Jun 19:32 2016
Picon
Picon

Is there a standard for naming tag/value pairs when parsing

There was a project by Mitre (https://www.mitre.org/) called the Common 
Event Expression (https://cee.mitre.org/) that was going to be the 
official standard for metadata names for events, but that project has 
been stopped.

Other than the two references that the CEE project has for logging 
standardization efforts, does anyone know of any major efforts by any 
group to define a standard for metadata naming?

Evan.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Robin Blanchard | 10 Jun 23:14 2016

UTF-8 handling

Hi,

I've got syslog-ng-3.7.3-3.el7.centos.x86_64 feeding logstash-2.2.4-1.noarch.

I've had flags(sanitize-utf8) enabled on the relevant syslog-ng listener, yet LS continues to complain
with ':expected_charset=>"UTF-8"', to the point of LS ultimately choking.  The underlying messages are
indeed garbage, eg:

$ pcregrep -n "[\x80-\xFF]" garbage.txt |tail -1
299:\\xc2\u0003\u0003\\xe0\u0001|\\x9d\\x9bX\\xa3wt}\\xe4\u001e\\xd6g\u0013j\\x9c!j\\xd9#*\\x95#\\xf8������

Should I instead be employing flags(validate-utf8) ?

Ultimately, I'd like for syslog-ng to either re-encode messages into proper UTF8, or -- if not possible --
just squelch/drop them….


Thanks in advance




--------------------------------------------------------------------------------------------------------------------------
This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides
consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and
carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it
advises and do not have the authority to bind Capital or such funds to any transaction or agreement.

The information in this e-mail, and any attachment therein, is confidential and for use by the addressee
only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail,
or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the
intended recipient, please return the e-mail to the sender and delete it from your computer. This email is
for information purposes only, nothing contained herein constitutes an offer to sell or buy securities,
as such an offer may only be made from a properly authorized offering document. Although Nephila attempts
to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts
no liability for any damage sustained as a result of viruses.
--------------------------------------------------------------------------------------------------------------------------
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Nutan Shinde | 9 Jun 11:59 2016
Picon

Invalid parsing of syslog messages having timezone

Hi,

Following is the syslog message received from Cisco router :

*Mar  1 09:30:25.249 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down

As, you can see UTC is included in the above timestamp. That is why value of $PROGRAM is UTC and $MSGONLY is %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel2, changed state to down.

What should I include in the syslog-ng.conf so that time zone is ignored?
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


Gmane