Adding additional sources

Hello Peter,

I am trying to add an addition Netflow source to my existing
nfsen/nfdump deployment. I have followed your recommendations listed
below:

" There is an ugly work around - but no warranty whatsoever. Do it at
your own risk:
 1. Be sure, that you really want to do this.
 2. Accept, that all graphic data in your graphs get lost. Graph will
be rebuilt only
    from data existing in your data directory.
 3. Add the new source in nfsen.conf
 4. Stop NfSen.
 5. remove all rrd files in profile live directory.
 6. Re-run ./install.pl /your/nfsen.conf
 7. Start NfSen.
 6. Run ./nfsen -r live"

I even took it a few steps further. I actually blew away all
directories (BASE DIR and HTML DIR) and backed up existing raw netflow
data to a different file system. When I  run ./install.pl
/your/nfsen.conf it actually creates the directories and files for the
newly added source. Permissions are set up correctly. I import my
existing nfcapd data back into their directories and rebuild the
profile with ./nfsen -r live. But, when I run nfsen.rc start, a
process for nfcapd for the new source is not started. Nfcapd is only
collecting for the same sources and is not collecting for the newly
added source. Am I missing something?

Re: Adding additional sources

Hi Matt,
Latest NfSen snapshot on sourceforge supports adding/deleting new netflow sources to profile live.
If you do not want to upgrade for any reason have a look into the file etc/nfsen-shell-param.
There is a line SOURCES, which list each source:port which is needed for nfcapd startup. This should do the job.

	- Peter

-------- Original Message  --------
From: "Matt McBride" <0xmatt@...>
To: nfsen-discuss@...
Subject: [Nfsen-discuss] Adding additional sources
Date: Mon Oct 02 2006 17:28:36 GMT+0200 (CEST)

> Hello Peter,
> 
> I am trying to add an addition Netflow source to my existing
> nfsen/nfdump deployment. I have followed your recommendations listed
> below:
> 
> " There is an ugly work around - but no warranty whatsoever. Do it at
> your own risk:
>  1. Be sure, that you really want to do this.
>  2. Accept, that all graphic data in your graphs get lost. Graph will
> be rebuilt only
>     from data existing in your data directory.
>  3. Add the new source in nfsen.conf
>  4. Stop NfSen.
>  5. remove all rrd files in profile live directory.
>  6. Re-run ./install.pl /your/nfsen.conf

sflows and sampling

This maybe a silly question, but does anyone know if its possible not
to have sflow data sampled, in otherwards have Foundry and other gear
report all flows?

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: sflows and sampling

On 10/2/06, Mr. Jones <worldsense@...> wrote:
> This maybe a silly question, but does anyone know if its possible not
> to have sflow data sampled, in otherwards have Foundry and other gear
> report all flows?

sFlow uses sampling exclusively - unless Foundry or other gear have a
non-standard sFlow agent implementation. I'm not familiar with newer
Foundry gear.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

web interface not seein any profiles

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@...
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss

Give a man enough rope...

Hello,
You probably know the saying: Give a man enough rope and he will hang himself. Well, open-source software is sometimes known to 'give enough rope'. I'm sorry to say that the same thing can't be said about nfdump. 

I tried to do a top 10000 (yes, it's big, but that's my problem, right?), and nfdump wouldn't let me because a valid topN is between 0 and 1000. So, I set out to remove this barrier, and I managed to do it.

in nfdump.c (version 1.5.2), at line 1144 I modfied from:
if ((aggregate || flow_stat)  && ( topN > 1000 || topN == 0) ) {
to
 if ((aggregate || flow_stat)  && ( topN > 10000 || topN == 0) ) {

I recompiled everything, and as magic, it works! (I admit, I was very surprised that it was so easy to do...)  So, you can say that I manufactured my extra rope... :)

My question would be: why is this limitation imposed? Because everything seems to work fine with a greater top...

Sorry for my ranting,
Adrian Popa

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@...
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss

Re: Give a man enough rope...

Hi Adrian,

-------- Original Message  --------
From: Adrian Popa <adrian_gh.popa@...>
To: nfsen-discuss@...
Subject: [Nfsen-discuss] Give a man enough rope...
Date: Wed Oct 04 2006 10:37:17 GMT+0200 (CEST)

> Hello,
> You probably know the saying: Give a man enough rope and he will hang
> himself. Well, open-source software is sometimes known to 'give enough
> rope'. I'm sorry to say that the same thing can't be said about nfdump. 
> 
> 
> I tried to do a top 10000 (yes, it's big, but that's my problem,
> right?), and nfdump wouldn't let me because a valid topN is between 0
> and 1000. So, I set out to remove this barrier, and I managed to do it.
> 
> in nfdump.c (version 1.5.2), at line 1144 I modfied from:
> if ((aggregate || flow_stat)  && ( topN > 1000 || topN == 0) ) {
> to
> if ((aggregate || flow_stat)  && ( topN > 10000 || topN == 0) ) {
> 
> I recompiled everything, and as magic, it works! (I admit, I was very
> surprised that it was so easy to do...)   So, you can say that I
> manufactured my extra rope... :)
> 
> My question would be: why is this limitation imposed? Because everything
> seems to work fine with a greater top...

You're right, that the only limit is that line. You may do top 'what_ever_you_want' from the code
perspective, as the code does everything dynamically. So way this limitation? This magic boundary was
introduced because of speed reason. nfdump uses 2 different sort algorithms for record statistics and
record element statistics. You may have noticed, that you can to -n 0 for -s ip, which results in
a fully sorted list of IP addresses from your flows. Sorting records uses another algorithm, which is
faster for average top N usage ( typically 10 - 100 ), but slower > 1000 and can become very slow
for a full sort ( -n 0 ). That's why the limit was set to 1000. So the rope was limited - typically long enough,
but too short to hang yourself :) So you see I'm afraid of you health.
You may remove any restriction - but don't complain, if the rope gets long enough.

Maybe one day I will change that to dynamically selecting the proper faster algorithm, according to -n .

Hope this helps.

Take care.

	- Peter

> 
> Sorry for my ranting,
> Adrian Popa
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Nfsen-discuss mailing list
> Nfsen-discuss@...
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss

--
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag,  Security Engineer,  Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA  FB 84 CA 94 AB FC 5D D7
SWITCH,  Limmatquai 138,  CH-8001 Zurich,  Switzerland
E-mail: peter.haag@... Web: http://www.switch.ch/security

Re: no graphs in the PortTracker plugin

Hello,

I was experiencing an issue where my RRD graphs worked as expected
with the exception of those located under the "Details" tab and the
"PortTracker" tab. The graphs were empty but the tickers at the bottom
of both pages displayed correctly. After a recomplile of rrdtools
(v1.2.15) with the '--enable-perl-site-install' option run with
./configure, my graphs are working great now. My problem was related
to a previous rrd install that did not allow the site usage of the
rrds.pm module.

I hope this helps.

-Matt

On 7/15/06, Lambert Hoogeveen <lamberth@...> wrote:
> Chelo Malagon wrote:
>
> > Hello,
> > I have checked the mailing list trying to find tha solution to the
> > problem I have and I saw a thread about it but without saying how to
> > solve it (thread "porttracker not producing graphs"). In my case I'm
> > using Red Hat Enterprise 4, nfsen 1.2.4, nfdump  1.5-1 and
> > PortTracker-v.e-1.5.
> > I can not get PortTracker to produce graphs. The rrd files are generated
> > and properly updated as well as the "ports" and "portstatxx" files. The
> > plugin tab displays info in the botton table, but no GIFs are shown.
> > I have checked permissions and the configiration of the pugin (and also
> > reinstalled the plugin) and everything seems to be fine.
> > The only error I get is in the  error_log (the same error people in the
> > list commented before):
> > [client X.X.X.X] PHP Notice:  Undefined index:  0_skip in
> > /var/www/html/nfsen/nfsen.php/plugins/PortTracker.php on line 95,
> > referer: http://fraguelrock.rediris.es:8080/nfsen/nfsen.php/nfsen.php
> > [client X.X.X.X] PHP Notice:  Undefined index:  0_skip in
> > /var/www/html/nfsen/nfsen.php/plugins/PortTracker.php on line 99,
> > referer: http://fraguelrock.rediris.es:8080/nfsen/nfsen.php/nfsen.php
> > [client X.X.X.X] PHP Notice:  Undefined index:  0_skip in
> > /var/www/html/nfsen/nfsen.php/plugins/PortTracker.php on line 266,
> > referer: http://fraguelrock.rediris.es:8080/nfsen/nfsen.php/nfsen.php
> >
> > I was testing the pluing in other machine (Fedora Core 4) and the same
> > versions of nfsen/nfdump/PortTracker without problems.
> >
> > Thank in advance,
> > Chelo
>
> Hello Chelo,
> Yes, your problem is very familiar.
> I had the exact same issue, see thread "porttracker not producing graphs".
> I never found the answer to the problem for that particular installation,
> hence the answer isn't in the thread. Looks like I wasn't the only one.
> I can only put the problem down to some compatibility issue between the
> Porttracker code and RRDTool. I'm not a coder so don't know where to look.
> As Peter has stated on numerous occasions, the Porttracker code is
> experimental
> so I guess we can not expect it to work in all situations.
>
> I do have Porttracker1.5 working now though on Fedora Core 5 and RRDTool
> 1.2.13.
> The previous install was on the same box but then FC4 with RRDTool 1.2.12.
> Did get a different problem this time, with nftrack complaining that it
> could not
> find the RRD library files (librrd.so.2). Not sure why at the time but
> fixed it by
> using a FC5 RPM install for RRD rather than the tarball. Have
> subsequently noticed a thread
> in which a fix is given for this problem (thread "nftrack can't find
> librrd.so.2").
>
> I have noticed a new problem with Porttracker now that it has been
> running for a while.
> I may report this problem in a new thread to see if anyone else has seen
> this.
> When I reboot the box that Porttracker is running on (the box is a devel
> machine
> so these reboots are a common occurrence), once Porttracker starts
> running again it just
> thrashes the hell out of the hard disk and never seems to "catch up"
> again and does
> not update the RRD files.
> I have left it for several hours but it just thrashes the hard disk and
> the only way
> to stop it is to kill the process ID and then stop NFSEN.
> I also saw this behaviour on the previous install of Porttracker on FC4.
>
> Sorry that I can not be of more help with your particular problem, but
> hope that this
> information is of some use to you.
>
> Regards,
>
> Lambert Hoogeveen
>
>
>
>
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Nfsen-discuss mailing list
> Nfsen-discuss@...
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

upgrade to nfsen-snapshot-20060810 = historical data lost?

Hello,

I'm sure I have done this a couple of times before. Upgrading an
existing installation of nfdump and nfsen to the latest available
versions.

Yesterday I upgraded another installation having only two sources.
Our old and new (current) network. The upgrade from nfdump 1.5 /
nfsen 1.2.4 to nfdump 1.5.2 / nfsen snapshot-20060810 went fine.
Everything worked great. But I noticed that before the upgrade
I had the Flows, Packets and Traffic graphs starting at October
last year, after the upgrade all those graphs started at early
September of this year. 

I was able to restore the data using another source and a dirty
script doing a dump and restore. So not too much was lost. But
since I still need to do a couple of upgrades I'm curious if the
transformation script from separate RRDs for flows, packets and
traffic to a single RRD for every source is a 100% fool proof.

Cheers,

-Wim -/- SURFnet

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: upgrade to nfsen-snapshot-20060810 = historical data lost?

Hi Wim,

-------- Original Message  --------
From: Wim Biemolt <Wim.Biemolt@...>
To: nfsen-discuss@...
Subject: [Nfsen-discuss] upgrade to nfsen-snapshot-20060810 = historical	data lost?
Date: Mon Oct 09 2006 14:21:30 GMT+0200 (CEST)

> Hello,
> 
> I'm sure I have done this a couple of times before. Upgrading an
> existing installation of nfdump and nfsen to the latest available
> versions.
> 
> Yesterday I upgraded another installation having only two sources.
> Our old and new (current) network. The upgrade from nfdump 1.5 /
> nfsen 1.2.4 to nfdump 1.5.2 / nfsen snapshot-20060810 went fine.
> Everything worked great. But I noticed that before the upgrade
> I had the Flows, Packets and Traffic graphs starting at October
> last year, after the upgrade all those graphs started at early
> September of this year. 
> 
> I was able to restore the data using another source and a dirty
> script doing a dump and restore. So not too much was lost. But
> since I still need to do a couple of upgrades I'm curious if the
> transformation script from separate RRDs for flows, packets and
> traffic to a single RRD for every source is a 100% fool proof.

I'm sorry for the trouble. As what I can say, the entire RRD conversion logic is the most tested part of the
snapshot, as there are no standard RRD tools available for that. Therefore, I've tested it
on a lot of profiles without any problems. Could it be, that the RRD files where somewhat corrupt? Anyway, if
you can reproduce that, and it turns out a bug in the converter, let me know. No software
is bug free - unfortunately.

	- Peter
> 
> Cheers,
> 
> -Wim -/- SURFnet
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Nfsen-discuss mailing list
> Nfsen-discuss@...
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
> 

--
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag,  Security Engineer,  Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA  FB 84 CA 94 AB FC 5D D7
SWITCH,  Limmatquai 138,  CH-8001 Zurich,  Switzerland
E-mail: peter.haag@... Web: http://www.switch.ch/security