A flaw was found in the way Samba handles the file descriptor sets (fd_set)
datastructure.
The Samba codebase uses file descriptor sets in various places. The fd_set
structure is a fixed size defined by the FD_SETSIZE variable. If a file
descriptor with a value greater than or equal to FD_SETSIZE is added to a
set, it can set a single bit on the stack to a '1'.
In Red Hat Enterprise Linux, all samba processes except for smbd have a
limit set which prevents a process from allocating more than 1024 file
descriptors by default. 1024 is the value of FD_SETSIZE on Red Hat
Enterprise Linux.
smbd does not cap the maximum allowed file descriptors below 1024. This
means that if a remote attacker has the ability to open files on a Samba
server, they may be able to flip arbitrary stack bits to a '1'. It is not
currently believed that this flaw can be used for arbitrary code execution,
but the possibility should not be ruled out.
Acknowledgements:
Red Hat would like to thank the Samba team for reporting this issue. Upstream acknowledges Volker Lendecke of SerNet as the original reporter.
This issue has been addressed in following products:
Red Hat Enterprise Linux 4
Red Hat Enterprise Linux 5
Red Hat Enterprise Linux 6
Via RHSA-2011:0305 https://rhn.redhat.com/errata/RHSA-2011-0305.html