A memory corruption flaw was discovered in Xorg X server's Render extension. Problem originally reported as Firefox crashing X server issue was tracked down to an incorrect calculation issue in mod() macro by Olivier Fourdan: https://bugzilla.redhat.com/show_bug.cgi?id=495733#c15 This issue could cause mod() to return value greater than its second argument (the divisor), resulting in excessive read and write in Render composite operation, causing heap or video memory corruption.
mod() macro has the same definition in Xorg and XFree86 in Red Hat Enterprise Linux 4 and Red Hat Enterprise Linux 3 respectively, but in those versions, no expression is used as the first operand (the dividend). The expression causing incorrect calculation was added upstream via: http://cgit.freedesktop.org/xorg/xserver/commit/?id=7c84189ea1 https://bugs.freedesktop.org/show_bug.cgi?id=3566
In current X server versions, there is similar issue in fbWalkCompositeRegion. However, it does not seem to be called in the current code. It was previous called from fbComposite: http://cgit.freedesktop.org/xorg/xserver/commit/?id=d2f813f7db but it does not call it any more.
Removing embargo
This issue has been addressed in following products: Red Hat Enterprise Linux 5 Via RHSA-2010:0382 https://rhn.redhat.com/errata/RHSA-2010-0382.html