select vs fork implementation of sndrcv() in scapy-com

G'day scapy,

Currently the scapy-com repository head includes my change that totally
replaces sndrcv() to use select loop driven send/recv instead of forking
off a sender process. I did this because I needed more accurate
timestamps on packets to get accurate RTT measurements, and this looked
like the easiest way. 

Since my application relies on this change, I'd like to know the chances
that this will make it into the official release. I don't want to be
forever running with a local fork of scapy, and would rather have my
application monkeypatch this change into a vanilla scapy if I have to.

My change is non-trivial so I can understand if it never makes it into
release. A quick summary of what I consider the pros and cons might
help;

Pros:
  * accurate timestamps and RTT time measurement
  * faster on single-core systems
  * simpler and smaller code
  * easier to profile
Cons:
  * big change that could introduce breakage
  * slower on multi-core systems

I'm thinking right now that I will go down the path of making my app
monkey-patch scapy, because it is something I can always remove later
when/if scapy accepts this change.

Re: select vs fork implementation of sndrcv() in scapy-com

Hi,

On Mon, 14 May 2007, Donovan Baarda wrote:

> G'day scapy,
>
> Currently the scapy-com repository head includes my change that totally
> replaces sndrcv() to use select loop driven send/recv instead of forking
> off a sender process. I did this because I needed more accurate
> timestamps on packets to get accurate RTT measurements, and this looked
> like the easiest way.
>
> Since my application relies on this change, I'd like to know the chances
> that this will make it into the official release. I don't want to be
> forever running with a local fork of scapy, and would rather have my
> application monkeypatch this change into a vanilla scapy if I have to.

Your application doesn't rely on fork() or not fork(), but on precise 
timestamping, afaik. Whatever happens, I will provide reliable 
timestamps, so will not need your app do this instead of scapy.

Regarding the select() vs fork(), I've not decided yet. select() is nice 
to avoid link full problems I sometimes encounter on slow links, but 
managing emission at a given rate is non-trivial with select while it is 
in a fork... Maybe scapy could propose both and choose the best one 
according to the machine it is running on (UP vs SMP)... Or maybe I would 
come up with an hybrid approach.

--

-- 
Philippe Biondi <phil <at>  secdev.org>      SecDev.org

Re: select vs fork implementation of sndrcv() in scapy-com

On Mon, 2007-05-14 at 20:17 +0200, Philippe Biondi wrote:
> Hi,
> 
> On Mon, 14 May 2007, Donovan Baarda wrote:
> 
> > G'day scapy,
> >
> > Currently the scapy-com repository head includes my change that totally
> > replaces sndrcv() to use select loop driven send/recv instead of forking
> > off a sender process. I did this because I needed more accurate
> > timestamps on packets to get accurate RTT measurements, and this looked
> > like the easiest way.
> >
> > Since my application relies on this change, I'd like to know the chances
> > that this will make it into the official release. I don't want to be
> > forever running with a local fork of scapy, and would rather have my
> > application monkeypatch this change into a vanilla scapy if I have to.
> 
> Your application doesn't rely on fork() or not fork(), but on precise 
> timestamping, afaik. Whatever happens, I will provide reliable 
> timestamps, so will not need your app do this instead of scapy.

Yeah. Getting accurate timestamps in a forking implementation is tougher
because you need to send the timestamp information back along the pipe
from the sending process.

> Regarding the select() vs fork(), I've not decided yet. select() is nice 
> to avoid link full problems I sometimes encounter on slow links, but 
> managing emission at a given rate is non-trivial with select while it is 
> in a fork... Maybe scapy could propose both and choose the best one 

VoIP protocols and scapy

Load Problems with scapy

Hi all,

we're using scapy for a small python sniffer which only takes tcp syn packages
and saves the source ip into a database.

Our problem is, that the load of the system on which the sniffer is beeing used
increases after the sniffer is running for several hours. If the sniffer is
running about 1 or more days, it is using almost all of the available memory.

We're assuming that this is a sort of bug inside scapy.

Does anyone else has monitored this behavior?

We are using Python 2.4.4 on a Debian Etch Linux System. As Database we are
using a Postgres SQL Database and the psycopg and the dnspython modules.

Thanks in advance.

Regards
Sebastian
--
Sebastian Ganschow
Königsberger Str. 17
45770 Marl
Germany

Phone:  +49 2365 9 24 96 76
Mobile: +49 172 2 47 41 44
Mail:   sebastian <at> ganschow.name

Re: Load Problems with scapy

Hi,

On Tue, 22 May 2007, Sebastian Ganschow wrote:

> we're using scapy for a small python sniffer which only takes tcp syn packages
> and saves the source ip into a database.
>
> Our problem is, that the load of the system on which the sniffer is beeing used
> increases after the sniffer is running for several hours. If the sniffer is
> running about 1 or more days, it is using almost all of the available memory.
>
> We're assuming that this is a sort of bug inside scapy.
>
> Does anyone else has monitored this behavior?
>
> We are using Python 2.4.4 on a Debian Etch Linux System. As Database we are
> using a Postgres SQL Database and the psycopg and the dnspython modules.

What command do you use for sniffing ? If it is sniff() which parameters 
to you use ?

--

-- 
Philippe Biondi <phil <at>  secdev.org>      SecDev.org
Computer Security/R&D                   http://www.secdev.org
PGP KeyID:3D9A43E2  FingerPrint:C40A772533730E39330DC0985EE8FF5F3D9A43E2

---------------------------------------------------------------------
Desinscription: envoyez un message a: scapy.ml-unsubscribe <at> secdev.org
Pour obtenir de l'aide, ecrivez a: scapy.ml-help <at> secdev.org

Re: Load Problems with scapy

On Tue, 22 May 2007, Sebastian Ganschow wrote:
> Philippe Biondi schrieb:
>> What command do you use for sniffing ? If it is sniff() which parameters
>> to you use ?
>
> We're using the following command:
>
> sniff(iface="eth1", prn=analysePackage, filter="tcp dst port 25")

All packets are stored in memory, that's why you experience problems after 
some time. You need to provide the store=0 parameter to the sniff 
function.

See latest example I just added on:
http://www.secdev.org/projects/scapy/build_your_own_tools.html

--

-- 
Philippe Biondi <phil <at>  secdev.org>      SecDev.org
Computer Security/R&D                   http://www.secdev.org
PGP KeyID:3D9A43E2  FingerPrint:C40A772533730E39330DC0985EE8FF5F3D9A43E2

---------------------------------------------------------------------
Desinscription: envoyez un message a: scapy.ml-unsubscribe <at> secdev.org
Pour obtenir de l'aide, ecrivez a: scapy.ml-help <at> secdev.org

Re: Load Problems with scapy

Philippe Biondi schrieb:
>> sniff(iface="eth1", prn=analysePackage, filter="tcp dst port 25")
> 
> All packets are stored in memory, that's why you experience problems
> after some time. You need to provide the store=0 parameter to the sniff
> function.
> 
> See latest example I just added on:
> http://www.secdev.org/projects/scapy/build_your_own_tools.html

Thanks, i'll try it.

--
Sebastian Ganschow
Königsberger Str. 17
45770 Marl
Germany

Phone:  +49 2365 9 24 96 76
Mobile: +49 172 2 47 41 44
Mail:   sebastian <at> ganschow.name

Re: Load Problems with scapy

Sebastian Ganschow schrieb:
> 
> 
> Philippe Biondi schrieb:
>>> sniff(iface="eth1", prn=analysePackage, filter="tcp dst port 25")
>> All packets are stored in memory, that's why you experience problems
>> after some time. You need to provide the store=0 parameter to the sniff
>> function.
> 
>> See latest example I just added on:
>> http://www.secdev.org/projects/scapy/build_your_own_tools.html
> 
> Thanks, i'll try it.
> 

It seems to work. Thanks a lot.

--
Sebastian Ganschow
Königsberger Str. 17
45770 Marl
Germany

Phone:  +49 2365 9 24 96 76
Mobile: +49 172 2 47 41 44
Mail:   sebastian <at> ganschow.name

invitation philippe biondi

http://journees.afpy.org/

il reste un slot pour les lightning talks le samedi en fin d'apres midi !!

---------------------------------------------------------------------
Desinscription: envoyez un message a: scapy.ml-unsubscribe <at> secdev.org
Pour obtenir de l'aide, ecrivez a: scapy.ml-help <at> secdev.org