Re: nfprofile takes more than 5 min to complete
> -----Original Message-----
> From: Werner Schram [mailto:Werner.Schram@...]
> Sent: Monday, September 08, 2008 5:17 AM
> To: Brown, Robin
> Cc: dirkjan@...; nfsen-discuss@...
> Subject: Re: [Nfsen-discuss] nfprofile takes more than 5 min
> to complete
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Brown, Robin wrote:
> | This is not working for us, it seems to spawn hundreds of
> nfsend and
> | nfprofile processes and logs these to syslog:
> |
> | Sep 7 13:35:41 munchies nfsen[21090]: 348 channels/alerts
> to profile
> | Sep 7 13:35:45 munchies nfsen[21090]: nfprofile failed:
> Too many open
> | files Sep 7 13:35:45 munchies nfsen[23128]: Use of uninitialized
> | value in string ne at /var/local/nfsen/bin/nfsend line 303.
> | Sep 7 13:35:45 munchies nfsen[23128]: Use of uninitialized
> value in
> | print at /var/local/nfsen/bin/nfsend line 304.
> | Sep 7 13:35:45 munchies nfsen[23128]: Use of uninitialized
> value in
> | concatenation (.) or string at /var/local/nfsen/bin /nfsend
> line 305.
> |
> | Using nfsen-1.3b-20070720 and we're exceeding the 5 minute
> run time,
> | this would help a lot if we could get it to work.
> |
> | -Robin
>
> Hi Robin,
>
> The reason for the amount of processes is most likely because
> your system still doesn't have enough resources to run the
> periodic updates in time. What you should start with is
> looking at how long it actually takes for the periodic
> updates to complete. In your debug log, you can see when the
> profiling stages starts and ends, and when the plugin stage
> starts and ends. Another important thing to look at if the
> bottleneck is either your cpu capacity or harddisk
> throughput. I am not an expert in assessing the performance
> of a system, but the amount of processing time going to user
> and to the system could give some pointers. Possible ways of
> speeding things are:
>
> - - Do not set the $PROFILERS variable higher than the number
> of available processing cores. All instances of nfprofile run
> independently, so they all read the complete live profile
> separately. So it only makes sense to run multiple instances
> of nfprofile if they don't have to fight each other for
> processing time.
> - - Use compression. Peter has told me that there are reports
> of increased performance when using compression. This is
> likely because compression moves the strain from your storage
> hardware to your cpu.
> - - Use a ramdisk (tmpfs in linux or freebsd 7). If you are
> not actively using your live profile, you could consider
> moving it to a tmpfs partition, this decreases the load on
> your storage hardware during the profiling stage and a tmpfs
> partition has a considerably higher throughput. But at the
> cost of having a very short and non-persistent live profile.
> - - Reconsider the used plugins. If the bottleneck is the
> plugin phase, you could consider disabling some of them or
> look into ways of increasing their performance.
> - - Upgrade your harddisks. If the bottleneck is your
> harddisk troughput, this can help solving your problem.
> - - Upgrade your cpu hardware. see above.
> - - Consider moving part of your profiles to another machine.
> If everything fails, just buy more hardware :)
>
> I hope this helps.
>
> Regards,
> Werner
>
Thanks Werner, compression did not help. I'll try deleting some
profiles next. Then if I can get this in under 5 minutes I'll try the
patch once more to see if it spawns over 1000 processes again.
But should I be getting the 'Use of uninitialized value' errors in
syslog when the patch is applied?
-Robin
>
> |
> |
> |
> | -----Original Message-----
> | From: nfsen-discuss-bounces@...
> | [mailto:nfsen-discuss-bounces@...] On Behalf
Of
> | Werner Schram
> | Sent: Wednesday, August 13, 2008 8:27 AM
> | To: dirkjan@...
> | Cc: nfsen-discuss@...
> | Subject: Re: [Nfsen-discuss] nfprofile takes more than 5 min to
> | complete
> |
> | Hi Dirk-Jan,
> |
> | Dirk-Jan van Helmond wrote:
> | | The nfprofile process takes more than 5 minutes to complete.
> | | I have 20 profiles, and about 90G/sec of data to be graphed.
> | |
> | | The machine used is a quad-core Xeon 2.4G, with 2GB of memory.
> | |
> | |
> | | Is there any possible way to increase the performance of
> nfprofile?
> |
> | We also had this problem. I solved it by patching nfsen to run
> | multiple instances of nfprofile parallel. It is a simple patch as
> | nfprofile can actually run in parallel very well by design.
> I actually
> | promised to submit this patch on several occasions, but I
> never got to
> | it. I posted it in the patches section of the nfsen project
> on sourceforge:
> |
> |
> https://sourceforge.net/tracker/index.php?func=detail&aid=2049518&grou
> | p_
> | id=134525&atid=730182
> |
> | If you apply this patch, you can add a $PROFILERS=x
> variable in your
> | nfsen.conf, where you can set x to the amount of nfprofile
> processes
> | you want to use. We keep this number at least one below the
> number of
> | available cores, because this allows the OS to take control of the
> | remaining one to process the IO. We haven't tested if this
> is actually
> | necessary, as this gives us enough processing time either way.
> |
> | regards,
> | Werner Schram
>
> -
> --------------------------------------------------------------
> ----------
> - -
> This SF.Net email is sponsored by the Moblin Your Move
> Developer's challenge Build the coolest Linux based
> applications with Moblin SDK & win great prizes Grand prize
> is a trip for two to an Open Source event anywhere in the
> world http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Nfsen-discuss mailing list
> Nfsen-discuss@...
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkjE7XkACgkQ3ULkMS4OADnlGgCfclPweV2uBKc+Ypk/n3pMf1zo
> fy4AoKExd/hMMghVelF6/ZHcfrfLGEdq
> =0XrQ
> -----END PGP SIGNATURE-----
>
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
.
El Tuesday 02 September 2008 17:28:00 Donnelly, Michael (OFT) escribió:
> I was wondering if anyone has any creative or interesting alerts
>
> they would be willing to share with the list ?
> --------------------------------------------------------
> This e-mail, including any attachments, may be confidential, privileged or
> otherwise legally protected. It is intended only for the addressee. If you
> received this e-mail in error or from someone who was not authorized to
> send it to you, do not disseminate, copy or otherwise use this e-mail or
> its attachments. Please notify the sender immediately by reply e-mail and
> delete the e-mail from your system.
RSS Feed