Sunny K | 6 Feb 2009 19:49
Picon

Offloading pattern matching

Hi,

I am new to clamav and yet to understand the implementation details.
However, before I dive in,
how difficult is it to offload pattern matching/reg-ex operations to a
dedicated hardware? Any
pointers?

Thanks much,
Sam
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

Stephen Gran | 20 Feb 2009 01:12

use system libtommath when present

Hopefully this will be a useful patch.

Cheers,

 [PATCH] Modify build system to conditionally use system install of libtommath -
 it will be used if it is installed, or if it is explicitly enabled in
 configure
 Signed-off-by: Stephen Gran <steve <at> lobefin.net>

---
 configure.in             |   13 +++++++++++++
 libclamav/Makefile.am    |   19 ++++++++++++++-----
 libclamav/bignum.c       |    6 ++++++
 libclamav/bignum.h       |    6 ++++++
 libclamav/bignum_class.h |    6 ++++++
 libclamav/dsig.c         |    4 ++++
 libclamav/dsig.h         |    4 ++++
 7 files changed, 53 insertions(+), 5 deletions(-)

diff --git a/configure.in b/configure.in
index 3454abf..e400644 100644
--- a/configure.in
+++ b/configure.in
 <at>  <at>  -505,6 +505,19  <at>  <at>  fi
 AC_MSG_RESULT([$ZLIB_HOME])
 ])

+AC_ARG_ENABLE(system_tommath,
+[  --enable-system-tommath           Enable use of system libtommath (default=auto)],
enable_system_tommath=$enableval, enable_system_tommath="auto" )
(Continue reading)

Stephen Gran | 20 Feb 2009 01:57

Re: use system libtommath when present

On Fri, Feb 20, 2009 at 12:12:11AM +0000, Stephen Gran said:
> Hopefully this will be a useful patch.

And if you add this one, it even compiles with the bundled tommath.
sorry about that - autotools after midnight is never a good idea.

 [PATCH] and fix up my various errors
 Signed-off-by: Stephen Gran <steve <at> lobefin.net>

---
 configure.in          |    4 ++--
 libclamav/Makefile.am |    9 +++------
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/configure.in b/configure.in
index e400644..3a22ff9 100644
--- a/configure.in
+++ b/configure.in
 <at>  <at>  -511,12 +511,12  <at>  <at>  if test "x$enable_system_tommath" = "xauto"; then
     AC_CHECK_HEADER([tommath.h],[have_tommath_h=yes],)
 fi

-if test "x$have_tommath_h" = "xyes" || "x$enable_system_tommath" = "xyes"; then
+if test "x$have_tommath_h" = "xyes" || test "x$enable_system_tommath" = "xyes"; then
     AC_DEFINE([USE_SYSTEM_TOMMATH], 1, [use system lintommath])
     TOMMATH_LIBS="-ltommath"
     AC_SUBST([TOMMATH_LIBS])
 fi
-AM_CONDITIONAL([SYSTEM_TOMMATH], test "x$ac_cv_use_system_tommath" = "xyes")
+AM_CONDITIONAL([SYSTEM_TOMMATH], [test "x$ac_cv_use_system_tommath" = "xyes"])
(Continue reading)

Stephen Gran | 20 Feb 2009 02:13

Re: [fabio.ped <at> libero.it: [Pkg-clamav-devel] Bug#512776: License incompatibility with libgmp ("GPLv2 only" linking to "LGPLv3 or later")]

On Sat, Jan 24, 2009 at 11:17:02AM +0100, Tomasz Kojm said:
> On Sat, 24 Jan 2009 01:21:34 +0000
> Stephen Gran <steve <at> lobefin.net> wrote:
> 
> > I've just received the bug report below via the Debian bug tracking
> > system.  There's obviously not a whole lot I can do about it myself, but
> > I just thought I'd bring it to your attention.
> 
> Yeah, we got this information as well:
> 
> https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1366

Now that I've been doing some looking, I see that there is also a
bundled -ltdl in the source tree.  I'm not aware of any license issues
with libtool (although as it's an FSF project, I wouldn't be surprised
to see it move to v3 in the future).  Is this just a convenience copy,
or is because of a license problem I'm unaware of?

If it's just a convenience copy, can we give it the same treatment, and
preferentially use the system copy?  It'll probably mean patching the
version of the ltdl m4 that you have in favor of something that doesn't
produce totally broken make targets when you try to get it to use the
system libtool.

Currently, if you pass --with-ltdl-dir=/usr/lib, it add -L/usr/lib as
a preprequisite, and if you pass --with-ltdl-dir=/usr/lib/libltdl.la,
configure goes off and looks for /usr/lib/libltdl.la/libltdl.la and
doesn't find it and exits.

Grrr.
(Continue reading)

Török Edwin | 20 Feb 2009 10:52
Picon

Re: [fabio.ped <at> libero.it: [Pkg-clamav-devel] Bug#512776: License incompatibility with libgmp ("GPLv2 only" linking to "LGPLv3 or later")]

On 2009-02-20 03:13, Stephen Gran wrote:
> Now that I've been doing some looking, I see that there is also a
> bundled -ltdl in the source tree.  I'm not aware of any license issues
> with libtool (although as it's an FSF project, I wouldn't be surprised
> to see it move to v3 in the future).  Is this just a convenience copy,
> or is because of a license problem I'm unaware of?
>
> If it's just a convenience copy, can we give it the same treatment, and
> preferentially use the system copy?  It'll probably mean patching the
> version of the ltdl m4 that you have in favor of something that doesn't
> produce totally broken make targets when you try to get it to use the
> system libtool.
>
> Currently, if you pass --with-ltdl-dir=/usr/lib, it add -L/usr/lib as
> a preprequisite, and if you pass --with-ltdl-dir=/usr/lib/libltdl.la,
> configure goes off and looks for /usr/lib/libltdl.la/libltdl.la and
> doesn't find it and exits.
>   

It already picks the system ltdl, *IF* you have libltdl7-dev installed
(which is currently in experimental, waiting to be uploaded in sid).
It won't work with an older ltdl, and I don't think it would be wise to
downgrade libtool.

ldd clamscan/.libs/clamscan
        linux-vdso.so.1 =>  (0x00007fff0c5fe000)
        libclamav.so.5 => /usr/local/lib/libclamav.so.5 (0x0000003ed4200000)
        libltdl.so.7 => /usr/lib/libltdl.so.7 (0x00007f78041a9000)
        libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x0000003ee7800000)
        libz.so.1 => /usr/lib/libz.so.1 (0x0000003ed6000000)
(Continue reading)

Török Edwin | 20 Feb 2009 10:56
Picon

Re: use system libtommath when present

On 2009-02-20 02:12, Stephen Gran wrote:
> Hopefully this will be a useful patch.
>
> Cheers,
>
>  [PATCH] Modify build system to conditionally use system install of libtommath -
>  it will be used if it is installed, or if it is explicitly enabled in
>  configure
>  Signed-off-by: Stephen Gran <steve <at> lobefin.net>
>
>   

Does this prefer the included tommath over the system one?

Using the system one by default would be bad, because we had to make
some changes to workaround compiler problems.

For example OpenBSD 4.3 comes with gcc3.3, that has a broken __udivti3
in libgcc2 on amd64 (see
https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1409),
so MP64_BIT is turned off, to avoid using  __attribute__((mode(TI))).

Best regards,
--Edwin
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

Stephen Gran | 20 Feb 2009 12:02

Re: [fabio.ped <at> libero.it: [Pkg-clamav-devel] Bug#512776: License incompatibility with libgmp ("GPLv2 only" linking to "LGPLv3 or later")]

On Fri, Feb 20, 2009 at 11:52:17AM +0200, Török Edwin said:
> On 2009-02-20 03:13, Stephen Gran wrote:
> > Now that I've been doing some looking, I see that there is also a
> > bundled -ltdl in the source tree.  I'm not aware of any license issues
> > with libtool (although as it's an FSF project, I wouldn't be surprised
> > to see it move to v3 in the future).  Is this just a convenience copy,
> > or is because of a license problem I'm unaware of?
> >
> > If it's just a convenience copy, can we give it the same treatment, and
> > preferentially use the system copy?  It'll probably mean patching the
> > version of the ltdl m4 that you have in favor of something that doesn't
> > produce totally broken make targets when you try to get it to use the
> > system libtool.
> >
> > Currently, if you pass --with-ltdl-dir=/usr/lib, it add -L/usr/lib as
> > a preprequisite, and if you pass --with-ltdl-dir=/usr/lib/libltdl.la,
> > configure goes off and looks for /usr/lib/libltdl.la/libltdl.la and
> > doesn't find it and exits.
> >   
> 
> It already picks the system ltdl, *IF* you have libltdl7-dev installed
> (which is currently in experimental, waiting to be uploaded in sid).
> It won't work with an older ltdl, and I don't think it would be wise to
> downgrade libtool.

Ah, that's all right then - I can just wait on that making it to
unstable to develop against it.

Cheers,
--

-- 
(Continue reading)

Stephen Gran | 20 Feb 2009 12:03

Re: use system libtommath when present

On Fri, Feb 20, 2009 at 11:56:30AM +0200, Török Edwin said:
> On 2009-02-20 02:12, Stephen Gran wrote:
> > Hopefully this will be a useful patch.
> >
> > Cheers,
> >
> >  [PATCH] Modify build system to conditionally use system install of libtommath -
> >  it will be used if it is installed, or if it is explicitly enabled in
> >  configure
> >  Signed-off-by: Stephen Gran <steve <at> lobefin.net>
> >
> >   
> 
> Does this prefer the included tommath over the system one?

This patch prefers the system one (since that's generally my preference
:), but it's easy to reverse the order of the logic.  I won't be able to
look at the issue until late tonight or tomorrow sometime, but if you'd
prefer it that way around, I'm happy to send a patch for that behavior.

Cheers,
--

-- 
 --------------------------------------------------------------------------
|  Stephen Gran                  | He's just a politician trying to save   |
|  steve <at> lobefin.net             | both his faces...                       |
|  http://www.lobefin.net/~steve |                                         |
 --------------------------------------------------------------------------
_______________________________________________
(Continue reading)

Eugene Crosser | 2 Mar 2009 10:39
Favicon
Gravatar

New 0.95 API concerns

Gentlemen,

I have a couple of concerns about the new libclamav API introduced in
0.95 (rc1). I understand the reason to remove cl_limits structure, but I
think that the way it was done is, hmm, suboptimal.

cl_engine_set() and cl_engine_get() accessors have void* for the
argument, which may point to different type of variables: uint32, uint64
or char. The type of expected argument is dependent on the value of
cl_engine_field, and there is no type check of any kind, i.e. nothing
that prevents passing of e.g. a char pointer where in32 pointer was
expected. If, by chance, the types of arguments change in a future
release, the user program will recompile cleanly, and the change won't
be noticed. It's actually worse than it was when cl_limits was exposed:
when you assigned a value to a field of cl_limits structure, at least
basic type checking (and/or automatic conversion) was performed.

To mitigate this problem (if you *really* want to get rid of cl_limits
structure exposed to the user), you might introduce separate pairs of
accessor functions for different types of arguments, e.g.:

cl_engine_{get|set}_size(...,uint64_t *val)
cl_engine_{get|set}_int(...,uint32_t *val)
cl_engine_{get|set}_str(...,char *val)

This way, there will be no chance to pass the argument of wrong type.

And here we are coming to my second concern. By requiring the the user
to use bit-size-specific types (uint32_t, uint64_t), you force them to
deploy all the dark magic of having these types defined portably on
(Continue reading)

Chris Moules | 2 Mar 2009 17:04
Picon
Favicon

New ClamAV Milter + Patch

Hi,

I have been looking at the RC1 for ClamAV. With the rewrite of the Milter it has removed one important (for us)
feature. That is 
the ability to notify clients that there mail has been intercepted.

As an ISP here in Luxembourg we stand liable if we intercept and remove mail without informing the client.
Our current system 
informs the recipient about filtered Virus mail and delivers UCE tagged (or delivered to a Spam folder).

I have had a look at the source to the new Milter and have seen that it has simplified the program and made it
much simpler. 
This is great. I looked at the possibility for re-adding the notification possibility and have had initial
success with only a 
small patch. The style of the patch, I believe, stays within the spirit of the re-write.

Before doing any further development, I wanted to get an idea if this might make it into the upstream code or
if something like 
this would end up being an in-house patch.

Regards

Chris
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

Gmane