Re: sfcapd problem with 3com 4800g router

It seems to only happen when nfsen startup. The box is running Centos 6.2 all dependencies is installed via
package manager the perl version on the box is 5.10.1 but it doesn't seem to affect the collector or the
nfsen interface. The box has been running for almost a week now. 

nfsend[6032]: segfault at 65c ip 0018b26a sp bfdea6e0 error 4 in libperl.so[101000+161000] 
-//-
nfsend[12666]: segfault at 65c ip 0018b26a sp bfdea6b0 error 4 in libperl.so[101000+161000]

-----Opprinnelig melding-----
Fra: Peter Haag [mailto:phaag@...] 
Sendt: 30. april 2012 09:54
Til: Adrian Popa
Kopi: nfsen-discuss@...
Emne: Re: [Nfsen-discuss] sfcapd problem with 3com 4800g router

On 4/30/12 7:30, Adrian Popa wrote:
> 
> 
> On Fri, Apr 27, 2012 at 2:59 PM, Peter Haag
<phaag@...
<mailto:phaag@...>> wrote:
> 
>     .. as for the segfaults: This seems to be a Perl bug, which also applies to other Linux distros.
> 
> 
> I've noticed it too on perl 5.10 on some older Fedora Core 
> installations (with other perl scripts). Is it related to threads by any chance?
> http://stackoverflow.com/questions/5038337/why-does-my-threaded-perl-s
> cript-segfault

Re: NfSen & RedHat

On 25/4/2012 10:20 πμ, Nikolaos Milas wrote:

> I decided to:
>
> 1. Update CentOS so that it can compile with most recent software
> 2. Upgrade to nfsen 1.3.6p1 and nfdump 1.6.6
>
> So I compiled latest versions and upgraded. I now run the latest
> versions for about 12 hours and it works OK.
>
> I will let you know how it goes.

I am afraid the problem still occurs (but less frequently). I installed 
debugging symbols, and below follows a backtrace from a hang.

Can you please help in resolving the issue? Does this backtrace help?

Thanks,
Nick

=============================================

# gdb /data/nfsen/bin/nfsend 13540
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.

Re: NfSen & RedHat

Thanks for the info.

May I ask you a few questions:
o how long is nfsend running before it segfaults?
o does the crash occur, while working with NfSen (using a web browser)?
o does the crash occur over night. ( nobody is working )

It seems as a signal is not handled properly. I will check, if I can change
some Perl code.

	- Peter

On 5/8/12 13:35, Nikolaos Milas wrote:
> On 25/4/2012 10:20 πμ, Nikolaos Milas wrote:
> 
>> I decided to:
>>
>> 1. Update CentOS so that it can compile with most recent software
>> 2. Upgrade to nfsen 1.3.6p1 and nfdump 1.6.6
>>
>> So I compiled latest versions and upgraded. I now run the latest
>> versions for about 12 hours and it works OK.
>>
>> I will let you know how it goes.
> 
> I am afraid the problem still occurs (but less frequently). I installed 
> debugging symbols, and below follows a backtrace from a hang.
> 
> Can you please help in resolving the issue? Does this backtrace help?
> 

Re: NfSen & RedHat

On 8/5/2012 5:46 μμ, Peter Haag wrote:

> May I ask you a few questions:
> o how long is nfsend running before it segfaults?
> o does the crash occur, while working with NfSen (using a web browser)?
> o does the crash occur over night. ( nobody is working )
>
> It seems as a signal is not handled properly. I will check, if I can change
> some Perl code.

Thank you Peter,

It can happen any time; nfsen will run from 12 hours to 6 days (max) and 
then it will hang unexpectedly. (nfdump works flawlessly all the time.)

I think there were cases where it happened when there was a user 
connected to the web interface (but not actively using it); yet it has 
*never* happened when the interface was being actively used for queries. 
It mostly (if not always) happens when no user is connected to the web 
interface. My experiments (for several weeks) do not show any 
association between browser use/connection/exit and appearance of the 
hang in nfsen. In short, it happens when nfsen is unattended.

But I am wondering why there are no other reports of this issue by other 
administrators.

If you want me to follow any specialized debugging process, I could 
follow your instructions.

Thank you again for examining the hang.

Re: NfSen & RedHat

On 8/5/12 5:22 PM, Nikolaos Milas wrote:
> On 8/5/2012 5:46 μμ, Peter Haag wrote:
> 
>> May I ask you a few questions:
>> o how long is nfsend running before it segfaults?
>> o does the crash occur, while working with NfSen (using a web browser)?
>> o does the crash occur over night. ( nobody is working )
>>
>> It seems as a signal is not handled properly. I will check, if I can change
>> some Perl code.
> 
> Thank you Peter,
> 
> It can happen any time; nfsen will run from 12 hours to 6 days (max) and 
> then it will hang unexpectedly. (nfdump works flawlessly all the time.)

So it hangs and does not die? or did I misunderstood something?
If it dies does it produce a core, you could backtrace?

Your backtrace seem to occur on a send socket command, which is deep in Perl
anyway. This connection is never supposed to hang as an alarm fires after
some time.
if it hangs, do you have any syslog messages from nfsen?
Can you connect to the socket:

socat UNIX-CONNECT:/path/to/nfsen.comm -

You should see something like:
220 1336497630 nfsend v0.1 ready

Re: NfSen & RedHat

On 8/5/2012 8:26 μμ, Peter Haag wrote:

> So it hangs and does not die? or did I misunderstood something?
>

Sorry, my wording is wrong; it dies: Here is a new "death", I just 
received:

=========================================================
Program received signal SIGPIPE, Broken pipe.
0x00000036e040df15 in __libc_send (fd=4, buf=0x12e47250, n=12, flags=0) 
at ../sysdeps/unix/sysv/linux/x86_64/send.c:28
28          return INLINE_SYSCALL (sendto, 6, fd, buf, n, flags, NULL, 
NULL);
(gdb) backtrace full
#0  0x00000036e040df15 in __libc_send (fd=4, buf=0x12e47250, n=12, 
flags=0) at ../sysdeps/unix/sysv/linux/x86_64/send.c:28
         oldtype = <value optimized out>
         result = <value optimized out>
#1  0x00000036e18cf063 in Perl_pp_send (my_perl=0x122cc010) at pp_sys.c:1934
         sp = <value optimized out>
         mark = 0x123b2498
         origmark = 0
         targ = 0x12d94000
         gv = 0x12e2ffa0
         io = 0x12e1f980
         bufsv = 0x12d93fd0
         buffer = 0x12e47250 ".timeout=10\n"
         length = <value optimized out>
         retval = <value optimized out>

Busted PMs with Ubuntu dist upgrade

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@...
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss

Time Window issue

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@...
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss

Re: Time Window issue

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@...
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss

Re: Time Window issue

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@...
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss