Czanik, Péter | 21 Jul 15:46 2016

initial rpm packaging of syslog-ng rust parser modules

Hi,

As you might be aware, syslog-ng 3.8 -- which is still under development -- introduced the possibility of writing parser modules in Rust (https://www.rust-lang.org/). There are some ready to use modules available at https://github.com/ihrwein/syslog-ng-rust-modules Three of these received now initial RPM packaging: regex-parser, actiondb-parser and correlation-parser.

My packages are mostly untested, version numbers are messed up, documentation is not yet bundled, etc., but still in good enough shape for initial testing. You can find some documentation in the github repo under https://github.com/ihrwein/syslog-ng-rust-modules

Packages are built from the latest git head syslog-ng and rust-modules sources. You can download the rpm packaged rust modules and syslog-ng from:

Fedora / EPEL7 (x86_64): https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/

openSUSE Leap / Tumbleweed: https://build.opensuse.org/project/show/home:czanik:syslog-ng38

Note: syslog-ng 3.8 is not yet released, so use of these packages in a production environment is not recommended.

Bye,
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Schoonover, Mark E HHHH | 18 Jul 18:41 2016
Picon

NG Rewrite Capabilities on a Relay

Hello,

 

I’ve run into a situation where I need to rewrite messages before sending them to our central NG server. When NG is used as a relay, is the rewrite functionality available? I’ve been through the manual but there isn’t a mention of what functionality is available – other than writing logs to disk.

 

Regards,

 

Mark Schoonover

Infrastructure Engineering Manager

ENE   : Tools, Instrumentation and Common Services Team

Office: 32.8697° N, 116.9711° W

Phone : 770-261-7934

Email : mark.schoonover <at> cigna.com

Confidential, unpublished property of Cigna Health & Life Insurance Co.. Do not duplicate or distribute. Use and distribution limited solely to authorized personnel. ©2016 CIGNA.

 

------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE: If you have received this email in error,
please immediately notify the sender by e-mail at the address shown. 
This email transmission may contain confidential information.  This
information is intended only for the use of the individual(s) or entity to
whom it is intended even if addressed incorrectly.  Please delete it from
your files if you are not the intended recipient.  Thank you for your
compliance.  Copyright (c) 2016 Cigna
==============================================================================

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

rmkml | 13 Jul 00:16 2016

New ETPLC project with Syslog-NG for checking >9000 Threats on your logs!

Hello,

I am pround to announce the new http://etplc.org open source project
update with Syslog-NG for checking more than 9000 Threats on your
webserver/proxy logs!

It's a open source project, all feedbacks / informations are welcome.

Easy to use since 3 years ago ;)

1) add ETPLC on your Syslog-NG configuration like that:
(of course check before perl+etplc PATH and source/filter/destination configurations...)

destination d_prog { program("/usr/bin/perl /var/tmp/etplc_12jul2016a.pl -f
/var/tmp/emergingall_sigs11jul2016a_snort290b.rules -s"); };
log { source(s_src); destination(d_prog); };

2) ETPLC send alert to localhost:514/udp with "-s" option

3) See All options with "-h"

4) Already supported format is Squid, Apache, Nginx, ForeFront, BlueCoat, McAfee Web Gateway, IIS logs...

5) ETPLC exist on Perl and Python versions

ETPLC available on:
  -main http://etplc.org
  -http://sourceforge.net/projects/etplc/
  -https://github.com/rmkml/etplc
  -https://hub.docker.com/r/rmkml/etplc/
  -http://twitter.com/rmkml

Special THX to InfoSec community and  <at> EmergingThreats team!

Best Regards
 <at> Rmkml
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Kiran Doddi | 12 Jul 14:15 2016

Need mvn repository info for syslog-ng-core.jar

Hi,

We are using syslog-ng server for receiving logs from different sources 
& we need to configure Java Destination for writing those logs into 
local file. For this,
While implementing, I found syslog-ng-core.jar file, but I could not 
found the same anywhere in maven repo, so that I can add its dependency 
in my pom.xml & avoid putting it manually in jar bundling.

Kindly need your assistance to solve my issues.

Thanks & Regards,
Kiran

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Scheidler, Balázs | 7 Jul 01:41 2016

Re: Transport aux data overflow

We get various data points on the process connecting via /dev/log via recvmsg() that returns this in an aux buffer, statically sized to 1024.

It's interesting that you have more. If you can reproduce it, can you strace syslog-ng to see what is coming aux data?

This can be resized but that needs a recompilation.

On Jul 6, 2016 11:18 PM, "Noémi Ványi" <sitbackandwait <at> gmail.com> wrote:
Hi!

It seems to me that aux buffer size is a hard-coded value. So it cannot be configured using the config file.
I guess the size could be increased, but I am curious what the devs of syslog-ng would say. :)

On 6 July 2016 at 22:09, Evan Rempel <erempel <at> uvic.ca> wrote:
I got this log line out of syslog-ng, but I'm not sure what it really means.

Can anyone shed some slight on it and let me know how I can increase the
aux buffer size?

syslog-ng[29712]: Transport aux data overflow, some fields may not be
associated with the message, please increase aux buffer size;
aux_size='1024'
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Evan Rempel | 6 Jul 22:09 2016
Picon
Picon

Transport aux data overflow

I got this log line out of syslog-ng, but I'm not sure what it really means.

Can anyone shed some slight on it and let me know how I can increase the 
aux buffer size?

syslog-ng[29712]: Transport aux data overflow, some fields may not be 
associated with the message, please increase aux buffer size; 
aux_size='1024'
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Jorge Pereira | 3 Jul 15:32 2016
Picon
Gravatar

extracting jSON from $MESSAGE

Hi,

    I am not sure about the best approach and way to fix my problem, below more information.

1) I receive the below packet sent from a nginx/openresty instance.

2016/07/02 01:17:04 [emerg] 19081#0: *13163 [lua] init.lua:115: [captcha] {"fail_count":"","response_code":200,"client_ip":"192.168.1.22","hostname":"server-lab01","request_id":"2016-07-02T01:17:03Z|9175f93c0c||i0Xb3BuBWV","host":"www.mytest.com","http_request":{"verb":"GET","url":"\/","user-agent":"Mozilla\/5.0 (pc-x86_64-linux-gnu) Siege\/3.0.8","http_version":"1.1","all":"{\"host\":\"www.mytest.com\",\"x-country-code\":\"US\",\"connection\":\"close\",\"accept\":\"*\\\/*\",\"x-client-ip\":\"192.168.1.22\",\"user-agent\":\"Mozilla\\\/5.0 (pc-x86_64-linux-gnu) Siege\\\/3.0.8\",\"accept-encoding\":\"gzip\"}"},"geoip":{"location":"-90.5334,38.6500","city_name":"Chesterfield","country_name":"United States","longitude":-90.5334,"area_code":314,"latitude":38.65,"country_code2":"US","country_code3":"USA"},"got":"","action":"show","expected":"h1szmM","webapp_domain":"www.mytest.com"} while logging request, client: 192.168.1.22, server: www.mytest.com, request: "GET / HTTP/1.1", host: "www.mytest.com"

2) In my server side, I need to save the logs following a value of host: "www.mytest.com" like:

/var/log/syslog-ng/www.mytest.com.log

3) The problem is because the packet received has a part being a jSON, but I can't use the json-parser().

4) What is the best approach? I have used:

# Extracting only the jSON payload
rewrite p_nginx_wb_error_log_clean {
    subst(".*captcha] ", "", value("MESSAGE"), flags("global"));
    subst(" while logging request.*$", "", value("MESSAGE"), flags("global"));
};

parser p_nginx_wb_error_log_json {
    json-parser(
        marker("")
        prefix("j.")
    );  
};

destination d_nginx_wb_error_log {
    file("/var/log/syslog-ng/nginx/${j.webapp_domain:-unknow-payload}_error.log"                                                                                                                                                               
         create_dirs(yes)
         owner("root")
         group("root")
         perm(0644)
         dir_perm(0755)
         template("${MSG}\n")
    );  
};

--
Jorge Pereira
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Attila Szalai | 30 Jun 11:00 2016

getting some information about the sender process

Hi,

Maybe I'm totally wrong, but I have/had the impression that syslog-ng collects some information about the client process if that connects through a unix domain socket. But actually I could not find anything related to this neither in the documentation nor the code. Is this something that only exists in my mind or I just missed something?
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

David Campeau | 28 Jun 18:56 2016

Re: Syslog-ng Multiple Instances

Hello,

I've been using syslog-ng to filter syslog before forwarding on to a log collector. However, I need to spin up a second instance for testing purposes. I've found a little bit of information on-line, but it hasn't completed the entire picture.

This is the command used to start up the 2nd instance. I'm pointing to separate .conf .persist .pid and .ctl files -- However, it's still not working. I suspect the issue is due to OS log sources. How do a change log sources?

syslog-ng --cfgfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.conf --persist-file=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.persist --pidfile=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.pid --control=/usr/local/bin/syslog-ng-Second-Instance/syslog-ng.ctl &

This is the upper part of the syslog-ng.conf file for the 2nd instance I wish to run.

<at> version: 3.3.4
<at> include "scl.conf"
    options {
        time-reap(30);
        mark-freq(10);
        keep-hostname(yes);
        chain-hostnames(no);
        use-dns(no);
##       log-fifo-size(500000);                ## Tuning Options
##      flush_lines(10000);                   ## Tuning Options
##       flush_timeout(10000);                ## Tuning Options
    };

        source s_second_instance {
        syslog(transport("udp") port("518"));       #### Will receive test syslog on port 518
        };

    destination d_syslog_udp {
        syslog("10.X.X.X"
            transport("udp")
            port("514")
            throttle(4000)
        );
    };



I'm hoping someone has experience or has seen information on how to run a 2nd instance on the same box.

Best Regards,

David

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Hirose, Shinsaku | 24 Jun 16:10 2016

a log message is output in the two line

Hello, all,

I use syslog-ng-3.2.5-4.el6.x86_64 on Centos6.
I got it from eple epel repository.

I am troubled in how to use the source file driver.
The touble is that a log message is output in the two lines on remote syslog server.

How to reproduce is followings.

1. Prepare two hosts running syslog-ng.

  Host_A configuration is followings.
  ----------------------------------------
  source test {
    file("/tmp/a.log");
  };
  destination d_remote { udp("192.168.0.2"); };
  log { source(test); destination(d_remote); };
  ----------------------------------------

  Host_B(192.168.0.2) configuration is defaults.

2.Execute following command on Host_A.

  $ seq 8193 | (xargs -i echo -n "a";echo "") >> /tmp/a.log

3.Check the log on Host_B.

  As the result, a log message is output in the two lines on Host_B.

  One line is following. The num of "a" is 8192.
  aaaaaaaaaaaaa....... 

  The other line is following. The num of "a" is 1.
  a

I hope a log message is output in the one line on Host_B.
Is my hope readily achievable?

Please advise me.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Czanik, Péter | 21 Jun 09:11 2016

CzP in Nuremberg

Hi,

For the next couple of days I'll be in Nuremberg for the openSUSE conference: https://events.opensuse.org/conference/oSC16 I'll give a presentation on syslog-ng Friday in the afternoon at 13:30 in the Roter Salon. I'll wear my syslog-ng superhero t-shirt, so I'll be easy to spot and ready to answer syslog-ng questions any time.

If you are in Nuremberg, but not at the conference, I'm happy to talk to you about syslog-ng also outside of the conference floor. You can reach me by e-mail or on twitter as <at> pczanik

See you soon in Nuremberg,

Peter

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


Gmane