A memory corruption flaw was discovered in Xorg X server's Render extension. Problem originally reported as Firefox crashing X server issue was tracked down to an incorrect calculation issue in mod() macro by Olivier Fourdan:
This issue could cause mod() to return value greater than its second argument (the divisor), resulting in excessive read and write in Render composite operation, causing heap or video memory corruption.
mod() macro has the same definition in Xorg and XFree86 in Red Hat Enterprise Linux 4 and Red Hat Enterprise Linux 3 respectively, but in those versions, no expression is used as the first operand (the dividend).
The expression causing incorrect calculation was added upstream via:
In current X server versions, there is similar issue in fbWalkCompositeRegion. However, it does not seem to be called in the current code. It was previous called from fbComposite:
but it does not call it any more.
This issue has been addressed in following products:
Red Hat Enterprise Linux 5
Via RHSA-2010:0382 https://rhn.redhat.com/errata/RHSA-2010-0382.html