Samba 3.0 / 3.2 heap overflow on AIX?
<yaberger <at> ca.ibm.com>
2008-05-01 19:50:57 GMT
I'm trying to find a possible heap overflow which first seemed to be in
AIX 5.3 with Samba (3.0 ou 3.2).
With the AIX support, we've been able to use some debugging utility, debug
libc, etc. on AIX that allow the support to think the problem might be in
Actually, without any modification to Samba code and using a local (AIX)
userid, Samba is working #1.
But when I add a few lines to allow AIX authentication on using the LAM
(Loadable Authentication Module) and authenticating with a DCE userid
belonging to more than 32 groups, it coredump.
The same patch/tests is working #1 without this problem on AIX 5.2 btw.
Before telling me that the problem might come from my patch or the DCE
client or AIX, please take the time to read what's below
With this scenario, Samba was coredumping when I was doing the
connection/authentication from a windows workstation with the DCE userid
(belonging to more than DCE groups). By looking at the stack/traces of the
core file, it pointed me that it was coredumping in initgroups(), which is
an OS syscall. By replacing the AIX syscall with the rep_initgroups()
you're providing to system without an initgroups() implementation, I'm
able to authenticate with this user correctly (no coredump)
I've opened a PMR (Problem Management Request) to the AIX team because I
thought the overflow was in initgroups() or a subroutine called by it
which wasn't in debug mode. After some collaboration with them, we've
ended with the following 2 tests that let us think the heap overflow might
be in Samba and possibly in rbtree.c (explaination in the test below).