It was discovered, that OpenOffice.org memory allocator is not 64bit clean. rtl_allocateMemory() function in sal/rtl/source/alloc_global.c accepts one argument - sal_Size n. On 64bit platforms such as x86_64, sal_Size is defined as unsigned long int. This requested memory chunk size is later memory aligned as size (type sal_Size). size is later used to calculate int index to g_alloc_table[] array: int index = (size - 1) >> RTL_MEMALIGN_SHIFT; However, as sizeof(int) == 4 and sizeof(sal_Size) == 8 on 64bit platforms, the calculated value may not fit into index (this can happen when rtl_allocateMemory() is called with large argument, e.g. when some other flaw causes OpenOffice to attempt to allocate chunk of memory with negative size, which wraps to large positive value during signed -> unsigned type conversion, such as [1]). Value stored in the index is wrapped / truncated, possibly resulting in the index being negative. Before index is used, it is checked not to exceed fixed upper limit, but it's not checked whether its value is >= 0: if (index < RTL_MEMORY_CACHED_LIMIT >> RTL_MEMALIGN_SHIFT) Negative index used in g_alloc_table[index] will cause OpenOffice to access memory outside of the g_alloc_table[] array. This may result it crash, or if that points to an attacker controlled memory, attacker may possibly be able to use this flaw to run arbitrary code. [1] http://www.openoffice.org/issues/show_bug.cgi?id=91818 http://scary.beasts.org/security/CESA-2008-006.html
OpenOffice.org upstream is aware of this issue, but they do not plan to release any security advisory for it, as they do not ship any 64bit pre-built packages. Issue was fixed upstream shortly after we reported it. Upstream bug report: http://www.openoffice.org/issues/show_bug.cgi?id=92217 This issue does not affect openoffice.org and openoffice.org2 packages as shipped in Red Hat Enterprise Linux 3 and 4, as only 32bit builds are shipped in those versions.
Rene Engelhard of Debian pointed out that 64bit builds that use system memory allocator instead of the OpenOffice.org custom one (i.e. compiled with --with-alloc=system) are also unaffected by this problem.
Lifting embargo.
*** Bug 455867 has been marked as a duplicate of this bug. ***
openoffice.org-2.4.1-17.6.fc9 has been pushed to the Fedora 9 stable repository. If problems still persist, please make note of it in this bug report.
openoffice.org-2.3.0-6.16.fc8 has been pushed to the Fedora 8 stable repository. If problems still persist, please make note of it in this bug report.
This issue was addressed in: Red Hat Enterprise Linux: http://rhn.redhat.com/errata/RHSA-2008-0835.html Fedora: https://admin.fedoraproject.org/updates/F8/FEDORA-2008-7531 https://admin.fedoraproject.org/updates/F9/FEDORA-2008-7680