Szabó, István | 26 May 16:53 2016

Re: Web Visualize logs (Ivan Adji - Krstev)

Hi,

For using syslog-ng together with elasticsearch and Kibana this is a pretty good description, also giving you a good understanding of what it enables you to do:

https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/

https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-and-simple/

/Istvan
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Ivan Adji - Krstev | 26 May 09:56 2016
Picon

Web Visualize logs

Hi all,
Me again :) !
Is there any other tool, free open source that can show me the log in some kind of graphs and can make some triggers etc. Like LogAnalyzer ? I want to change the LogAnalyzer and try other different web interfaces etc. Any suggestions ?

I see Elasticsearch or Kibana but, as i see it now, they use some different protocols and styles (i still can't understand how is working) and im not sure if i can integrate with syslog-ng, but also i need a free tool :( .

As im using Syslog-NG with MongoDB all i need is something that can present all this log and have options for sending e-mails if we have disaster or something and can query fast all this logs.

LogAnalyzers is ok but i want to see different styles.


Thanks in advanced !

Ivan
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Alexey Vlasov | 24 May 11:46 2016

Transform file path

Hi,

I intend to manage the distribution of the Apache log-files for each
virtualhost using syslog-ng.

I write the following in Apache vhost configue:

<VirtualHost *>
    SetEnv V3WUSER w_test-l26-apache-_b8649b
    LogFormat "%{V3WUSER}e %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" xcombined
    CustomLog /var/log/apache_aux2_worker2/access_pipe.log xcombined
</IfModule>

as a result I get this kind of format of the log file: 

w_test-l26-apache-_b8649b 10.0.2.24 - - [24/May/2016:12:41:33 +0300]
"GET / HTTP/1.1" 200 - "http://example.com/" "ELinks (0.11.7; Linux 3.14.46-1gb-csm x86_64; 158x45)"

where the first field is the unique identifier of the virtualhost.

Next step I write the following in syslog-ng: 

source src_apache_piped_logs {
    pipe("/var/log/apache_aux2_worker2/access_pipe.log"
    keep_timestamp(no));
};

destination dst_apache_piped_logs {
    file("/var/log/virtwww/${V3WUSER}/access.log"
    template("$MSGONLY\n") template-escape(no));
};

log {
    source(src_apache_piped_logs); destination(dst_apache_piped_logs);
};

Сonsequently I would like to have each log file of the virtualhost
placed in its own catalog and the path should contain the first field of
the message  (${V3WUSER} = w_test-l26-apache-_b8649b)

Is it actually possible?
Thanks in advance for the help.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Raymund Ács | 23 May 19:21 2016
Picon

[GSoC]Finding Feature owner for "Web-based syslog-ng configuration editor"

Hi,


I could not write here before because I had some issues with subscribing. I already subscribed and unsubscribed few months ago, and I was not able to subscribe again. Finally I got my confirmation mail, so I can join to the community! :)

In the summer I will work on the Web-based syslog-ng configuration editor project in GSoC 2016. Here is my proposal link: 

This project will provide a Web Application which can create and save syslog-ng configuration files by drag&drop technique. I would like to follow the trends of 2016 in design, and give a detailed documentation. 

If you are interested in being the feature owner of this project, please reply in this thread. Advices from you will help us greatly and make the final project more user friendly.

Best Regards,
Raymund Acs
University of Debrecen
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Yilin Li | 22 May 17:03 2016
Picon

[GSOC]Finding Feature owner for WebSocket for syslog ng

Hi
I'm working on the "WebSocket for syslog ng" project in GSOC 2016. 

This project will provide 
- a websocket plugin for syslog-ng source and destination.
- user-friendly and detailed documents   and  examples.

Any  suggestions will be appreciated  :) 

If you are interested in being the  feature owner of the project, please reply in this thread.
Advices  on user's perspective from you will help us greatly and make the final deliverables more user friendly.
Thanks in advance.

--
Best wishes,
Yilin  Li
Institute of Software Chinese Academy of Sciences
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Vithulan MV | 21 May 17:03 2016
Picon
Picon

Finding feature owner for Kafka source project

Hi all,

As you may have all known already, We are going to develop a new feature, Kafka source for syslog-ng product during this GSoC 2016 under the guidance of Viktor Juhasz.

A simple abstract idea of this project is,
The syslog-ng application can read messages from the sources. It processes them with filters, rewrite rules, parsers and finally sends messages to their destinations. The syslog-ng application already has a Kafka destination that is implemented in Java. The Kafka source will allow syslog-ng to read messages from Kafka, for an example this can be used for example as a queue between several syslog-ng instances.

Kafka source project will allow you to,
-Use Kafka consumer which uses the high level Kafka group consuming API.
-Continue reading from the last message from which syslog-ng reloaded/restarted.
-Avoid message loosing or message duplication as much as possible whenever there is multiple syslog-ng reading the same kafka input.

If you are interested to become a feature owner for Kafka source project and give us guidance on user's perspective, it will be very easy for us to build a effective  component. Appreciate if you can contact us by sending mail to us or to this thread.

Thanks,
Best regards,
Vithulan.

--
Vithulan MV.

Undergraduate, 
Department of Computer Science & Engineering,
University of Moratuwa,
Sri lanka.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Noémi Ványi | 20 May 14:10 2016
Picon

WANTED: feature owner of syslog-ng command line tool :)

Hello!

A new feature is going to be developed, as a part of GSOC, so syslog-ng can be started as a command line tool.

What could the CLI do for you?
  • pipelines of complex filtering, rewriting could be tested more easily, thus configuration could become simpler than before
  • existing logs could be transformed
The format of the new command line options is not decided yet. So your help and insight is needed to design the interface and the usage of the tool that you and other people around the world would use. :)

If you are intereted in contributing to the CLI, thus contributing to the syslog-ng community, please send an email to this thread. :) I am available at gitter for further info. :)

Regards
kvch
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Ankush Sharma | 19 May 10:55 2016
Picon
Gravatar

[Release Automation Project] Finding feature owners

Hi,

I am taking this as an opportunity to introduce myself to everyone in the community. I got a bit late as I was busy in relocating back and finishing college paper work. Sorry for it ;-)

I am Ankush Sharma (black-perl) from India. I contributed to Mailman last year. So, I will be working on the Automation Release project for syslog-ng this time. The project will mostly involve Python usage and Flask/Django later on. 

The potential areas where any feature owner can look into are:

1) Web interface: An experimental web interface to track things during the release automation cycle would be handy. I am considering to use Flask/Django (web frameworks of Python) for it. You can suggest integrations to it or information you would like to see on the interface.

2) Overall architecture: We can discuss the flow from user perspective as well as the architectural view of the tool and you can suggest changes to it ;-)

Here is the link to the intro post : http://black-perl.in/black-perl-gsoc-with-syslog-ng/ . You can find a brief introduction to the project here and links to the detailed proposal.

I would love to talk to other guys who will be working throughout the summers as part of the GSoC program: <at> litterbear, <at> kvch, <at> PoOwAa, <at> Vithulan. It would be great if we can have chat sometime and get to know about each others' work ! 

Thanks !

Ankush Sharma
ECE IV
IIT-BHU
Varanasi-221005
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Ivan Adji - Krstev | 17 May 13:44 2016
Picon

Re: Syslog-NG RFC



On 05/17/2016 01:22 PM, Ivan Adji - Krstev wrote:
Hi Robert,
Everything is placed as it is. I have put the server and the client in the networ() part the flags and still nothing. No template in use.

Here is the configuration of the server:

options {
    flush_lines (0);
    time_reopen (10);
    log_fifo_size (1000);
    chain_hostnames (off);
    use_dns (no);
    use_fqdn (no);
    create_dirs (no);
    keep_hostname (yes);
};

source s_sys {
        system();
        internal();
        network(ip(0.0.0.0) port(6514)
        flags(syslog-protocol)
        transport("tls")
        tls(key_file("/etc/syslog-ng/cert.d/serverkey.pem")
        cert_file("/etc/syslog-ng/cert.d/servercert.pem")
        ca_dir("/etc/syslog-ng/ca.d")
        ) );

};

destination d_mongodb {
        mongodb(
        servers("localhost:27017")
        database("syslog")
        username("test")
        password("test123")
        collection("messages")
        value-pairs(
        scope("selected-macros" "nv-pairs" "sdata")
                )
                        );
                        };


And here it is on the client site:

options {
    flush_lines (0);
    time_reopen (10);
    log_fifo_size (1000);
    chain_hostnames (off);
    use_dns (no);
    use_fqdn (no);
    create_dirs (no);
    keep_hostname (yes);
};

source s_sys {
    system();
    internal();
    # udp(ip(0.0.0.0) port(514));
};


destination tls_destination {
    network("x.x.x.x" port(6514)
        flags(syslog-protocol)
    transport("tls")
    tls( ca_dir("/etc/syslog-ng/ca.d")
         key_file("/etc/syslog-ng/cert.d/clientkey.pem")
         cert_file("/etc/syslog-ng/cert.d/clientcert.pem") )
);
     };


And i have use the same configuration with MySQL and works perfect now i have problems with MongoDB.

Kind regards
Ivan


On 05/17/2016 01:10 PM, Fekete, Róbert wrote:
Hi, 

The protocols used in the syslog-ng clients and the syslog-ng server should match.
You posted a source that uses the network() driver - I take this is from your server.
The destination on your client should also use the network() driver, and that's where you need the flags(syslog-protocol).

Also check your client config to see if it uses a custom template that messes with the message format.

Robert



On Tue, May 17, 2016 at 10:49 AM, Ivan Adji - Krstev <akivanradix <at> gmail.com> wrote:
Any way i put it where i can and on a client but still nothing.

Any other hints ?

Ivan

On 05/17/2016 10:31 AM, Fabien Wernli wrote:
Hi, On Tue, May 17, 2016 at 10:11:27AM +0200, Ivan Adji - Krstev wrote:
some RFC model 5424. Is there an option to configure the syslog-ng to send this messages in that RFC format ?
Add `flags(syslog-protocol)` to the network destination ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq





______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq


______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Ivan Adji - Krstev | 17 May 10:11 2016
Picon

Syslog-NG RFC

Hi all,
I have configure the Syslog-NG with MongoDB and LogAnalyzer, and as output i have the messages but i have no Date, Facility Severity Syslogtag and other information from it. If i go on a message itself i have some additional information. I have found out that i should use some RFC model 5424. Is there an option to configure the syslog-ng to send this messages in that RFC format ?

Kind regards
Ivan
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

Ivan Adji - Krstev | 16 May 13:06 2016
Picon

Syslog-NG with MongoDB

Hi all,

What is the best practice for storing all those logs in one central environment. I have one Linux Box running Syslog-NG with LogAnalyzer and MongoDB ( for now ), and is the best way to configure and use it with MongoDB or with MariaDB ( MySQL ) ? I have once install MySQL but it was getting very slow as the logs getting bigger and bigger ( for one week ).
Now i have done with MongoDB ( still testing ) but i have problem as LogAnalyzer does not show me the real pictures, i have no Date info, no Facility, no serverity, Hosts, syslogtag, i just have ProcessID.

Any hints on this ?

I have the following configuration on the syslog-ng.cfg:

destination d_mongodb {
    mongodb(
    servers("localhost:27017")
        database("logs")
#    uri('mongodb://localhost/syslog-ng')
    collection("syslog")
    value-pairs(
    scope("selected-macros" "nv-pairs" "sdata")
        )
            );
            };

Kind regards
Ivan
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


Gmane