headers
Ernest Wypierowski | 6 Dec 2007 10:30
Picon

thread memory stack size consumption in clamav-milter

Good morning,

I had dive into the maillist's archive but I was unsuccessful in finding
the answer how to calculate memory stack size for thread in  clamav-mil-
ter.  I am running clamav-milter in 32 bit mode operating system (linux)
and I am wonder if there will be a problem in fixing 2 MB per thread? In
that way I will be able to size daemon for at least more the 1000 simul-
taneous connections.

Best regards,
Ernest Wypierowski
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Permalink | Reply | 0 comments |
headers
Ernest Wypierowski | 6 Dec 2007 18:03
Picon

thread memory stack size consumption in clamav-milter

Good morning,

I had dive into the maillist's archive but I was unsuccessful in finding
the answer how to calculate memory stack size for thread in  clamav-mil-
ter.  I am running clamav-milter in 32 bit mode operating system (linux)
and I am wonder if there will be a problem in fixing 2 MB per thread? In
that way I will be able to size daemon for at least more the 1000 simul-
taneous connections.

Best regards,
Ernest Wypierowski
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Permalink | Reply | 1 comment |
headers
Tomasz Kojm | 6 Dec 2007 19:13
Favicon

Re: thread memory stack size consumption in clamav-milter

On Thu, 06 Dec 2007 18:03:12 +0100
Ernest Wypierowski <ew-reg <at> mailbox.com.pl> wrote:

> I had dive into the maillist's archive but I was unsuccessful in finding
> the answer how to calculate memory stack size for thread in  clamav-mil-
> ter.  I am running clamav-milter in 32 bit mode operating system (linux)
> and I am wonder if there will be a problem in fixing 2 MB per thread?

That mostly depends on your scan settings. If you want to limit the stack
size, you'll definitely need to limit the level of recursion (eg.
MailMaxRecursion, ArchiveMaxRecursion). 2 MB should be enough to keep sane
limits, but you will need to determine proper settings empirically.

--

-- 
   oo    .....         Tomasz Kojm <tkojm <at> clamav.net>
  (\/)\.........         http://www.ClamAV.net/gpg/tkojm.gpg
     \..........._         0DCA5A08407D5288279DB43454822DC8985A444B
       //\   /\              Thu Dec  6 18:56:31 CET 2007
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Permalink | Reply | 0 comments |
headers
Ernest Wypierowski | 15 Dec 2007 23:37
Picon

libmilter's workers pool in clamav-milter

Good evening,

version:
libmilter - sendmail-8.14.2
clamav-milter - clamav-0.91.2

If   you  configure  libmilter library  with  using  poll()  rather than
select() and use workers pool you might hit problem related with locking
up your  clamav-milter.

To  reproduce  this  kind of behavior you should ran clamav-milter in 32
bit mode, lower max-children and set timeout to some rather high  value.
When  number  of simultaneous connections exceed max children then every
new connection will equal one thread. Going step further,  when  threads
allocate  all  virtual  memory  available for clamav-milter workers from
libmilter will not have space to execute  (clamav-milter:  LAUNCH_WORKER
error: Cannot allocate memory).

Daemon   clamav-milter   will  not  answer with temporary error but will
keep going with blocking threads/connections.

How to solve this kind of problem? Does not clamav-milter is prepare for
workers from libmilter?

Most  interesing is how to limit clamav-milter's resource based on thre-
ads. In described situation number of threads are not  equal  number  of
connections,  limiting  it  with  dont-wait  threads is not a good idea.
Maybe there is a way to  limit  numbers  of  threads  that  workers  can
consume?
(Continue reading)

Permalink | Reply | 5 comments |
headers
Jose-Marcio Martins da Cruz | 16 Dec 2007 09:49
Picon

Re: libmilter's workers pool in clamav-milter


That's already done inside libmilter :

With sendmail 8.14.0, you shall compile libmilter with :

   APPENDDEF(`conf_libmilter_ENVDEF',`-D_FFR_WORKER_MODEL')

This will switch the libmilter thread model to a pool of workers instead 
of one thread per sendmail connection. And the number of threads in the 
filter will be the number of threads really doing something...

This makes a very big difference on systems handling a big number of 
simultaneous connections.

Here are some old slides (2003) from an old presentation I've done at 
sendmail about that :

   http://j-chkmail.ensmp.fr/papers/workermilter.pdf

Well... I've got an award from sendmail... 8-)

   http://www.j-chkmail.org/wiki/doku.php/award

Ernest Wypierowski wrote:
> Good evening,
> 
> version:
> libmilter - sendmail-8.14.2
> clamav-milter - clamav-0.91.2
>
(Continue reading)

Permalink | Reply | 4 comments |
headers
Jose-Marcio Martins da Cruz | 16 Dec 2007 11:25
Picon

Re: libmilter's workers pool in clamav-milter


Well, I think now I didn't understood your first message.... 8-(

What I can say is that there are a number of servers running this 
without problem since 2004. The biggest one I have some feedback handles 
almost 30.000 messages an hour.

Surely, this thread model can handle much more traffic than original 
threads model, but it to has a limit somewhere.

The problem there is that this new limit shall be evaluated by someone : 
libmilter or the filter. The solution I proposed is OS and filter 
independent.

So there are things which can be done at libmilter level and others by 
the filter.

A filter shall eventually be able to evaluate resources consumption and 
behave adequately. So, if the filter says - sorry, I'm not able to 
handle any other new connection now, this is better than accepting and 
crashing. This is usually what's done in real life with real people.

Limit the number of threads isn't a good idea, unless the filter 
(clamav-milter or other) can limit the amount of time allocated to each 
task (e.g. message content scanning). If you don't do that, you create a 
DoS vulnerability.

If there are new ideas to improve this feature, please let me know.

Regards,
(Continue reading)

Permalink | Reply | 3 comments |
headers
Ernest Wypierowski | 16 Dec 2007 12:53
Picon

Re: libmilter's workers pool in clamav-milter

On Sun, 16.12.2007 at 11:25:47, Jose-Marcio Martins da Cruz wrote:
> A filter shall eventually be able to evaluate resources consumption and 
> behave adequately. So, if the filter says - sorry, I'm not able to 
> handle any other new connection now, this is better than accepting and 
> crashing. This is usually what's done in real life with real people.

I entirely agree. However the filter, clamav-milter, when libmilter/wor-
kers.c could not spawn new thread freeze waiting for answer.

Limiting the connection number to clamav-milter in my opinion is not the
best approach. I can image  a  event  when  smtp's  client  connects  to
server  and   send mail from/rcpt to and wait. Then connection to filter
(clamav- milter) would be set but nothing will not load cpu.

I  would like to see the better way to limit the resources, workers from
libmilter inform clamav-milter that they reach the MAX_WORKERS
and clamav-milter sends: 451 4.3.2 AV system temporarily  overloaded
to smtp server.

> Limit the number of threads isn't a good idea, unless the filter 
> (clamav-milter or other) can limit the amount of time allocated to each 
> task (e.g. message content scanning). If you don't do that, you create a 
> DoS vulnerability.

So the only limit is to set the max number of connection?

Best regards,
Ernest Wypierowski
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
(Continue reading)

Permalink | Reply | 2 comments |
headers
Jose-Marcio Martins da Cruz | 16 Dec 2007 14:05
Picon

Re: libmilter's workers pool in clamav-milter

Ernest Wypierowski wrote:
> On Sun, 16.12.2007 at 11:25:47, Jose-Marcio Martins da Cruz wrote:

> 
> Limiting the connection number to clamav-milter in my opinion is not the
> best approach. I can image  a  event  when  smtp's  client  connects  to
> server  and   send mail from/rcpt to and wait. Then connection to filter
> (clamav- milter) would be set but nothing will not load cpu.
> 
> I  would like to see the better way to limit the resources, workers from
> libmilter inform clamav-milter that they reach the MAX_WORKERS
> and clamav-milter sends: 451 4.3.2 AV system temporarily  overloaded
> to smtp server.

I agree with you, but this kind of thing is better done inside the 
filter than inside libmilter.

This is what I do inside my filter j-chkmail. Already, it evaluate a 
"resource usage" which depends on many things, mainly CPU load and the 
number of file descriptors in use.

> 
>> Limit the number of threads isn't a good idea, unless the filter 
>> (clamav-milter or other) can limit the amount of time allocated to each 
>> task (e.g. message content scanning). If you don't do that, you create a 
>> DoS vulnerability.
> 
> So the only limit is to set the max number of connection?

j-chkmail begins limiting the connection rate per IP address and the
(Continue reading)

Permalink | Reply | 0 comments |
headers
karthik Ramamoorthy | 18 Dec 2007 07:16
Picon
Favicon

how to make contribution to ClamAV

Hi all,

     I have been looking into the source code ClamAV for some times and found it quite interesting. I got a very
vague idea. Is CLAMUKO optional and when it will be used(I am not sure, but i guess it will be if dazuko module
is there, correct me if not). 

    I am really interested in ClamAV and would like to contribute to it. So can anybody please guide me in which
area i should start looking into and how to proceed and related things. I am not a clamAV expert, just a
started who would like to get involved in ClamAV and contribute to it something.

Karthik.

---------------------------------
Looking for last minute shopping deals?  Find them fast with Yahoo! Search.
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Permalink | Reply | 0 comments |
headers
Renato Botelho | 18 Dec 2007 17:18
Picon
Gravatar

Error building Mail::ClamAV after clamav ABI change

Hello,

After last clamav ABI change on 0.92, i'm testing all related packages
on FreeBSD ports to make the update. But one of clamav dependant ports
didn't build. I've contacted maintainer and author, but, anyway, i'm
sending it here, maybe it's easy to fix and i can make a patch to fix
the port.

It's about a perl modules Mail::ClamAV, here is the buildlog:

http://tb3.droso.net/errors/7-garga/p5-Mail-ClamAV-0.20_1.log

Any idea to fix it?
--

-- 
Renato Botelho
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net
Permalink | Reply | 3 comments |

Gmane