page_count is copied from userspace. agp_allocate_memory() tries to check whether this number is too big, but doesn't take into account the wrap case. Also agp_create_user_memory() doesn't check whether alloc_size is calculated from num_agp_pages variable without overflow. This may lead to allocation of too small buffer with following buffer overflow. Reference and patch: https://lkml.org/lkml/2011/4/14/294
Upstream commit: http://git.kernel.org/linus/b522f02184b413955f3bc952e3776ce41edc6355
Statement: This issue affects the versions of Linux kernel as shipped with Red Hat Enterprise 4, 5, 6, and Red Hat Enterprise MRG. This has been addressed in Red Hat Enterprise Linux 5, 6, and Red Hat Enterprise MRG via https://rhn.redhat.com/errata/RHSA-2011-0927.html, https://rhn.redhat.com/errata/RHSA-2011-1350.html, and https://rhn.redhat.com/errata/RHSA-2011-1253.html. Red Hat Enterprise Linux 4 is now in Production 3 of the maintenance life-cycle, https://access.redhat.com/support/policy/updates/errata/, therefore the fix for this issue is not currently planned to be included in the future updates.
This issue has been addressed in following products: Red Hat Enterprise Linux 5 Via RHSA-2011:0927 https://rhn.redhat.com/errata/RHSA-2011-0927.html
Acknowledgements: Red Hat would like to thank Vasiliy Kulikov of Openwall for reporting this issue.
This issue has been addressed in following products: MRG for RHEL-6 v.2 Via RHSA-2011:1253 https://rhn.redhat.com/errata/RHSA-2011-1253.html
This issue has been addressed in following products: Red Hat Enterprise Linux 6 Via RHSA-2011:1350 https://rhn.redhat.com/errata/RHSA-2011-1350.html