There is a problem in agp code - kernel memory exhaustion (AGPIOC_RESERVE and
AGPIOC_ALLOCATE ioctls). It is not checked whether requested pid is a pid of
the caller (no check in agpioc_reserve_wrap()). Each allocation is limited to
16KB, though, there is no per-process limit. This might lead to OOM situation,
which is not even solved in case of the caller death by OOM killer - the memory
is allocated for another (faked) process.
Red Hat would like to thank Vasiliy Kulikov of Openwall for reporting this issue.
This issue was mentioned in https://lkml.org/lkml/2011/4/14/294 and linux-2.6:b522f02184b413955f3bc952e3776ce41edc6355 for the patch fixing CVE-2011-1746 because the patch tries to fix a similar problem - OOM.
CVE-2011-1747 is not fixed yet.
Red Hat Security Response team does not consider this bug to be security relevant one due to the privileges (CAP_SYS_RAWIO) required to exploit this issue.