headers
Matt Sisk | 14 Oct 2011 20:01
Favicon

Net::Nmsg 0.07 released

There's a new version of perl extension for libnmsg. Updates of note:

   - added typemap support for the newer double and bool fields
   - fixed some issues on 64-bit architectures
   - fixed some edge-cases in multi-threaded environments

It can be downloaded either here:

     https://tools.netsa.cert.org/confluence/display/tt/Net-Nmsg

or (eventually) from CPAN:

     http://search.cpan.org/dist/Net-Ncap/

Please contact me with questions or bug reports.

Thanks,
Matt

______________
Matt Sisk
Member of the Technical Staff
CERT Software Engineering Institute
Carnegie Mellon University
sisk <at> cert.org
Permalink | Reply | 3 comments |
headers
Paul Vixie | 14 Oct 2011 20:29

Re: Net::Nmsg 0.07 released

On Fri, 14 Oct 2011 14:01:52 -0400
Matt Sisk <sisk <at> cert.org> wrote:

> There's a new version of perl extension for libnmsg. Updates of note:
> 
>    - added typemap support for the newer double and bool fields
>    - fixed some issues on 64-bit architectures
>    - fixed some edge-cases in multi-threaded environments
> 
> It can be downloaded either here:
> 
>      https://tools.netsa.cert.org/confluence/display/tt/Net-Nmsg

kewl!  it's nice to not be the only person using Perl for this.

> or (eventually) from CPAN:
> 
>      http://search.cpan.org/dist/Net-Ncap/

do you mean Net-Nmsg here?

--

-- 
Paul Vixie
Permalink | Reply | 2 comments |
headers
Dave Plonka | 14 Oct 2011 20:44
Picon
Favicon

Re: Net::Nmsg 0.07 released

On Fri, Oct 14, 2011 at 06:29:31PM +0000, Paul Vixie wrote:
> On Fri, 14 Oct 2011 14:01:52 -0400
> Matt Sisk <sisk <at> cert.org> wrote:
> 
> > There's a new version of perl extension for libnmsg. Updates of note:
> > 
> >    - added typemap support for the newer double and bool fields
> >    - fixed some issues on 64-bit architectures
> >    - fixed some edge-cases in multi-threaded environments
> > 
> > It can be downloaded either here:
> > 
> >      https://tools.netsa.cert.org/confluence/display/tt/Net-Nmsg
> 
> kewl!  it's nice to not be the only person using Perl for this.

I'm using it as well. :-)
Yes, thanks for the fixes Matt.

I'm currently using it process dnsqr messages, to passively identify
DNS-based services that have both IPv4 and IPv6 answers.

Dave

--
plonka <at> cs.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI
Permalink | Reply | 0 comments |
headers
Dave Plonka | 14 Oct 2011 20:51
Picon
Favicon

nmsg for [net]flow data?

Hi folks,

Is there an existing nmsg-based tool or set that puts netflow data
in nmsg format?

I'm trying to decide whether to develop my tools to use a hybrid
of nmsg format for DNS and nfdump format for flow data, based on
whether or not someone else has already tackled this.

Barry mentioned (in April) to me something about SIE flow data support,
but I never got a link to the presentation or documentation for it.

Thanks!
Dave

--

-- 
plonka <at> cs.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI
Permalink | Reply | 8 comments |
headers
Robert Edmonds | 14 Oct 2011 21:09

Re: nmsg for [net]flow data?

Dave Plonka wrote:
> Is there an existing nmsg-based tool or set that puts netflow data
> in nmsg format?
> 
> I'm trying to decide whether to develop my tools to use a hybrid
> of nmsg format for DNS and nfdump format for flow data, based on
> whether or not someone else has already tackled this.

this issue is brought up occasionally by those who work with netflow,
although usually in the other direction (i.e. somehow embedding NMSG
data in netflow/ipfix).

afaik there are currently no nmsg message modules that implement a
message type for netflow data.  it wouldn't be that hard to create one.

> Barry mentioned (in April) to me something about SIE flow data support,
> but I never got a link to the presentation or documentation for it.

he must have been mistaken.  we don't have any support for flow data.

--

-- 
Robert Edmonds
edmonds <at> isc.org
Permalink | Reply | 1 comment |
headers
Dave Plonka | 14 Oct 2011 21:28
Picon
Favicon

Re: nmsg for [net]flow data?


Hi Robert,

On Fri, Oct 14, 2011 at 03:09:13PM -0400, Robert Edmonds wrote:
> Dave Plonka wrote:
> > Is there an existing nmsg-based tool or set that puts netflow data
> > in nmsg format?
> > 
> > I'm trying to decide whether to develop my tools to use a hybrid
> > of nmsg format for DNS and nfdump format for flow data, based on
> > whether or not someone else has already tackled this.
> 
> this issue is brought up occasionally by those who work with netflow,
> although usually in the other direction (i.e. somehow embedding NMSG
> data in netflow/ipfix).

Yes, I see how that would similarly be useful since those are also
extensible protocols with enterprise-specific extensions and such.

> afaik there are currently no nmsg message modules that implement a
> message type for netflow data.  it wouldn't be that hard to create one.
> 
> > Barry mentioned (in April) to me something about SIE flow data support,
> > but I never got a link to the presentation or documentation for it.
> 
> he must have been mistaken.  we don't have any support for flow data.

Ah, I looked back at the email; I misread it... he said that "netflow
channels" where in the work queue for SIE, and I mistook it to mean
that the recent webinar he mentioned was about that.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Eric Ziegast | 15 Oct 2011 04:12

Re: nmsg for [net]flow data?

On 10/14/11 11:51 AM, Dave Plonka wrote:
> Is there an existing nmsg-based tool or set that puts netflow data
> in nmsg format?

To add to Robert's short answer ("no")...

He's correct, there is no current NMSG data type for representing data
you'd typically find in netflow.

People who developed netflow put some thought in how to aggregate data
that from a large number of sensor points (routers) over UDP back to a
centralized collector or analysis tool.  Tools like nfdump and
nfreplay have some of the message-relaying features one might find in
nmsgtool.  With a slight programming tweak to nfreplay, it's possible
to have it send messages out in a broadcast manner (setsockopt +
SO_BROADCAST).  I prototyped this tweak in ISC SIE before NMSG was
developed and it worked well enough, so we didn't need to create our
own encapsulation for broadcast transport across a switch to multiple
listeners.  We learned that an entire framework surrounding
high-volume netflow processing already existed (eg: FlowTools) whose
wheel ISC didn't need to reinvent.

There are other database-driven technologies for low-volume processing
for the problem of, "How do I best classify threats based in part on
analysis of netflow data and other data?"

Let's look, though, at a different scenario:

 - Imagine getting a high-volume real-time stream of raw packet
   darknet probes and backscatter.
(Continue reading)

Permalink | Reply | 2 comments |
headers
Jose Nazario | 15 Oct 2011 18:07

Re: nmsg for [net]flow data?

meant to send this to everyone, not just ericz, with a few post-coffee fixes.

netflow is a well defined structure (in fact cisco has the structs available openly in a .h file). so, it
could fit easily into nmsg with the right protobuf definitions, no need to stuff arbitrary UDP packets
into an nmsg. with some effort you could even get netflow v9 with flexible templating working in nmsg, as
long as the nmsg receivers know how to interpret the template record and react accordingly. all nmsg does
is change the transport mechanism from raw UDP PDUs into protobufs over UDP. no big deal.

the benefit is that you can have one type of input handler for your setup that you described where you listen
to events and correlate. in fact this is the basis for the whole work of stream databases (aka continuous
event processing or continuous stream query setups). nmsg just becomes the transport layer between
nodes. what you do there - emit, aggregate, correlate, etc - is up to you. 

- j
Permalink | Reply | 1 comment |
headers
Wes Young | 17 Oct 2011 16:27
Favicon

Re: nmsg for [net]flow data?


we've been thinking down this route, at first the idea was:

+ encapsulating the binary flow data in a protocol buffer

but then you run up against v5 vs v9 stuff, so then we thought:

+ encap the binary flow data in a buffer and document the version (for easier parsing)

but then it's harder to correlate across other flow type stuff (ids logs, fw logs, etc which we've found are
really just other types of flow data when you're in SEM land)

which is what we really want.

so (fwiw), we're testing the mapping flows (v5 and v9) to a binary (encapsulated) idmef structure up front
and streaming that into a database. That way, when we wanna do cross-correlation everything's been
normalized up to that point (for say, a map-reduce jobber). The warehouse only needs to cache the
data-structures as it see's them and they're already ready to go for correlation.

ymmv.

On Oct 15, 2011, at 12:07 PM, Jose Nazario wrote:

> the benefit is that you can have one type of input handler for your setup that you described where you listen
to events and correlate. in fact this is the basis for the whole work of stream databases (aka continuous
event processing or continuous stream query setups). nmsg just becomes the transport layer between
nodes. what you do there - emit, aggregate, correlate, etc - is up to you. 

--
Wes
(Continue reading)

Permalink | Reply | 0 comments |
headers
Matt Sisk | 17 Oct 2011 16:34
Favicon

Re: Net::Nmsg 0.07 released

On 10/14/2011 02:29 PM, Paul Vixie wrote:
>> or (eventually) from CPAN:
>>
>>       http://search.cpan.org/dist/Net-Ncap/
>
> do you mean Net-Nmsg here?

Indeed! I must have been copying and pasting back when I originally 
built the wiki entry.

Thanks,
Matt
Permalink | Reply | 0 comments |

Gmane