Rafal Wojtczuk reported that X server has a number of properties that allow unprivileged users to exploit stack overflow in X server process by forcing it to allocate excessive amount of memory and place user-writeable shared memory segment close to the stack memory area boundary. This way, local unprivileged attacker may be able to overwrite memory that is used by privileged X server process as stack memory and execute code with server privileges. This type of flaws was covered in more detail in Gael Delalleu's "Large memory management vulnerabilities" presentation on CanSecWest 2005: http://cansecwest.com/core05/memory_vulns_delalleau.pdf Two possible approaches to address this problem were proposed: - Modify Linux kernel to always use heap-stack guard page to reduce probably of successful jump over the stack gap. Such patch was proposed in 2004 by Andrea Arcangeli and is used by some vendors' kernels, but is not upstreamed yet. http://lkml.org/lkml/2004/9/25/89 Separate bug #606611 tracks this proposed change. Note: Such protection may be bypassed if stack pointer is adjusted by more than the size of the guard area. This can happen in case of alloca() use, or in case of functions with large amount of local variables. - Modify X.org server to RLIMIT_AS limit by default to avoid the situation where kernel needs to place memory pages for requested allocation close to the stack memory. Proposed limits are: 1.5GB for 32bit systems and 10GB for 64bit systems.
Created attachment 432396 [details] Proposed X.org patch (from Matthieu Herrb) Adds default RLIMIT_AS limits as described above. Also adds new server command line option to override default limits.
Created attachment 434600 [details] Patch limiting window tree depth (from Keith Packard) This patch enforces limits on window tree depth to block user's ability to trigger deep recursion in X server.
Making this bug public too, as the issue is public for some time now. As noted elsewhere (see e.g. http://lwn.net/Articles/400746/), proposed X server changes were not deemed complete fixes for the problem, kernel-side fix adding guard page was added in upstream kernel an is being backported to vendor kernels (check bug #606611 to follow Red Hat Enterprise Linux and Fedora kernel updates adding this protection).