headers
R.H.Hoek | 1 Jul 2005 10:34
Picon
Picon
Favicon

netflow and missing packets

Dear all,

Still I am trying to solve 'my' problem with missing netflow-packets by nTop
send by our Cisco6509.

In one of the last messages Burton said: "It could be that ntop isn't
pulling them fast enough from the interface, but that's just a pretty
simple select() recvfrom() call pair."

I had a look into netflowPlugin.c and found that part of the code:

if((rc = select(maxSock+1, &netflowMask, NULL, NULL, NULL)) > 0) {
  if(FD_ISSET(myGlobals.device[deviceId].netflowGlobals->netFlowInSocket, &netflowMask)){
    len = sizeof(fromHost);
    rc = recvfrom(myGlobals.device[deviceId].netflowGlobals->netFlowInSocket,
                  (char*)&buffer, sizeof(buffer),
                  0, (struct sockaddr*)&fromHost, (socklen_t*)&len);
  }
.
.
.

I have no experience/knowlegde changing this code. I wil try to find someone that can
do this for me.... On the otherhand, if someone on this list can do this and want
to test, I can send our netflowstream to you. With flow-tools-fanout I can
simply fanout the stream to other hosts.

Thanks........

--

--
(Continue reading)

Permalink | Reply | 1 comment |
headers
Tamas Kovacshazy | 1 Jul 2005 13:57
Picon
Favicon

URL problem++


Hi,

It seems like the URL problem I reported previously is more complex.

The problem is checkURLsecurity() in http.c. The checks for // (and 
for some other fishy stuff like &&, ??) in the URL, and returns with 
an error.

After that an HTTP error response is returned to the client and the 
built in HTTP server freezes, not answering requests. NTOP must be 
restarted.

It is clearly a BUG, at least on my setup (current CVS version), in 
addition, it makes way for a DoS attack.

Best Regards,

khazy

PS: Apache substitutes '//' in URLs with '/' if '//' appears in the 
server local part of the URL, and answers the request without problem. 
Just try it on an Apache server. 
_______________________________________________________________________________
  Tamas Kovacshazy  E-mail: khazy <at> mit.bme.hu  WWW: http://www.mit.bme.hu/~khazy
 		Budapest University of Technology and Economics
  Department of Measurement and Information Systems  WWW: http://www.mit.bme.hu
Permalink | Reply | 3 comments |
headers
Burton Strauss | 1 Jul 2005 14:08

RE: netflow and missing packets

If you have the stream in a file, you can probably hack at flow-tools-fanout
to slow down the rate (or does it have a rate parm - there's a tickle about
that in the back of my head).  A simple nanosleep of 0.1s after each packet
send would eliminate an question about whether it's rate limited.

You did check the netFlow plugin statistics, right?  Often 'lost' flows are
actually port zero ...

-----Burton 

-----Original Message-----
From: ntop-dev-bounces <at> unipi.it [mailto:ntop-dev-bounces <at> unipi.it] On Behalf
Of R.H.Hoek
Sent: Friday, July 01, 2005 3:35 AM
To: ntop-dev <at> Unipi.IT
Subject: [Ntop-dev] netflow and missing packets

Dear all,

Still I am trying to solve 'my' problem with missing netflow-packets by nTop
send by our Cisco6509.

In one of the last messages Burton said: "It could be that ntop isn't
pulling them fast enough from the interface, but that's just a pretty simple
select() recvfrom() call pair."

I had a look into netflowPlugin.c and found that part of the code:

if((rc = select(maxSock+1, &netflowMask, NULL, NULL, NULL)) > 0) {
  if(FD_ISSET(myGlobals.device[deviceId].netflowGlobals->netFlowInSocket,
(Continue reading)

Permalink | Reply | 0 comments |
headers
Burton Strauss | 1 Jul 2005 16:04

RE: URL problem++

Well, Apache is accepting an illegal (per the RFC) request:

http://www.w3.org/Protocols/rfc1945/rfc1945

       HTTP-message   = Simple-Request           ; HTTP/0.9 messages
                      | Simple-Response
                      | Full-Request             ; HTTP/1.0 messages
                      | Full-Response

where

       Simple-Request  = "GET" SP Request-URI CRLF

and

       Request-URI    = absoluteURI | abs_path

       absoluteURI    = scheme ":" *( uchar | reserved )

       abs_path       = "/" rel_path

       rel_path       = [ path ] [ ";" params ] [ "?" query ]

       path           = fsegment *( "/" segment )
etc.

Note that a rel_path must have characters before it's 1st /.

// is valid only as:
(Continue reading)

Permalink | Reply | 0 comments |
headers
cvs-commit | 2 Jul 2005 23:21

New ntop commit (author burton)

Update of /export/home/ntop/ntop
In directory unknown:/tmp/cvs-serv17955

Modified Files:
	configure configure.in 
Log Message:
Enhance ./configure for FreeBSD 4.x build problem.  
Now automatically appends the -pthreads option.
This appears to be required for 4.10 and 4.11...

-----Burton (ref 607)
Permalink | Reply | 0 comments |
headers
Tamas Kovacshazy | 3 Jul 2005 15:51
Picon
Favicon

Re: URL problem++


Hi,

I was not patient enough to wait for your "BadGuy" timeout nor I 
identified it in http.c (spent approximately 10 minutes reading it, I must 
accept). Sorry for your time.

Anyway, I would simple return a "Not Found", and do not bother to maintain 
a BadGuy timeout for these strange URL-s. Most of them are caused by 
simple config errors or badly written scripts. These errors are corrected 
fast, as in my case (I triggered the code by some badly constructed 
reverse proxy config).

Best regards,

khazy

_______________________________________________________________________________
  Tamas Kovacshazy  E-mail: khazy <at> mit.bme.hu  WWW: http://www.mit.bme.hu/~khazy
 		Budapest University of Technology and Economics
  Department of Measurement and Information Systems  WWW: http://www.mit.bme.hu
Permalink | Reply | 1 comment |
headers
Burton Strauss | 3 Jul 2005 16:25

RE: Re: URL problem++

Try actually reading the code ... you can disable the whole feature via a
simple #define.  

/*
 * This is the size of the table that holds IP addresses we don't want to
 * talk to (due to sending an invalid - i.e. hostile) URL
 *
 * Valid values are 0 (disables) up to whatever.
 */
#define MAX_NUM_BAD_IP_ADDRESSES            3

and

#if defined(MAX_NUM_BAD_IP_ADDRESSES) && (MAX_NUM_BAD_IP_ADDRESSES > 0)
   /* Note if the size of the table is zero, we simply nullify all of this
      code (why bother wasting the work effort)
      Burton M. Strauss III <Burton <at> ntopsupport.com>, June 2002
    */

  for(i=0; i<MAX_NUM_BAD_IP_ADDRESSES; i++) {
    if(addrcmp(&myGlobals.weDontWantToTalkWithYou[i].addr,&from) == 0) {
       if((myGlobals.weDontWantToTalkWithYou[i].lastBadAccess +
           PARM_WEDONTWANTTOTALKWITHYOU_INTERVAL) < myGlobals.actTime) {
         /*
          * We 'forget' the address of this nasty guy after 5 minutes
          * since its last bad access as we hope that he will be nicer
          * with ntop in the future.
          */
         memset(&myGlobals.weDontWantToTalkWithYou[i], 0,
sizeof(BadGuysAddr));
(Continue reading)

Permalink | Reply | 0 comments |
headers
cvs-commit | 4 Jul 2005 05:34

New ntop commit (author burton)

Update of /export/home/ntop/ntop
In directory unknown:/tmp/cvs-serv20503

Modified Files:
	address.c globals-core.c globals-core.h globals-defines.h 
	globals-structtypes.h hash.c http.c initialize.c leaks.c 
	main.c ntop.c ntop_win32.c pbuf.c plugin.c prefs.c 
	reportUtils.c ssl.c util.c webInterface.c 
Log Message:
Replace the two ntop state indicators, capturePackets and endNtop, with 
a single flag, ntopRunState;

Here's the skinny...

It's your classic state-transition table, tracking from preinit through term 
(so someday we could restart).  Flag values defined as usual in globals-defines.h, 
around lines 1620ff.

There is a routine to trap illegal transitions in globals-core.c, _setRunState().
There is a one-time load of the permitted transition change table in there too.

_setRunState() also has (dummy) places - switch() statements - where we can 
(and should) move a lot of the setup/cleanup logic.  The 1st one is executed 
based on the old state, the second switch based on the new state.  So someday, 
we could just cycle through the states and the setup/cleanup could happen 
automatically.

The single state var allow a new routine, ntopSleepWhileSameState(), which 
allows all of the sleep() loops to interrupt themselves if the state changes 
(i.e. shutdown request).  ntop won't shutdown immediately (that would take a
(Continue reading)

Permalink | Reply | 0 comments |
headers
cvs-commit | 4 Jul 2005 05:35

New ntop commit (author burton)

Update of /export/home/ntop/ntop/plugins
In directory unknown:/tmp/cvs-serv20503/plugins

Modified Files:
	netflowPlugin.c rrdPlugin.c sflowPlugin.c 
Log Message:
Replace the two ntop state indicators, capturePackets and endNtop, with 
a single flag, ntopRunState;

Here's the skinny...

It's your classic state-transition table, tracking from preinit through term 
(so someday we could restart).  Flag values defined as usual in globals-defines.h, 
around lines 1620ff.

There is a routine to trap illegal transitions in globals-core.c, _setRunState().
There is a one-time load of the permitted transition change table in there too.

_setRunState() also has (dummy) places - switch() statements - where we can 
(and should) move a lot of the setup/cleanup logic.  The 1st one is executed 
based on the old state, the second switch based on the new state.  So someday, 
we could just cycle through the states and the setup/cleanup could happen 
automatically.

The single state var allow a new routine, ntopSleepWhileSameState(), which 
allows all of the sleep() loops to interrupt themselves if the state changes 
(i.e. shutdown request).  ntop won't shutdown immediately (that would take a 
LOT of POSIX rework so each thread could be individually addressed by signals), 
but rather at the next short (10s) interval, defined via #define PARM_SLEEP_LIMIT.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Burton Strauss | 4 Jul 2005 05:38

3.2 update

I've just committed the last planned fix/enhancement - this combines the two
state flags into one and does a lot of cleanup in the setup/shutdown of
ntop's threads.  I've tested in Linux and FreeBSD (4.10, 4.11, 5.3 and 5.4).

The plan is to let Luca fix whatever I've broken in the Win32 world, and
then put out an RC1 for people to test.

Right now, the only problem I'm definitely looking into is the dump.html
issue.

-----Burton
Permalink | Reply | 0 comments |

Gmane