Christian Borntraeger found a security relevant problem with the Linux system calls argument passing for at least s390, powerpc, sparc64 and mips:
The ABI for some architectures defines that the caller of a function has to sign extend each parameter to full register width. This is a problem in Linux system call handling.
For example with this system call on 64 bit:
asmlinkage long sys_example(unsigned int index)
if (index > 5)
It would mean that the caller has to sign extend index to 64 bit. In this case the caller is userspace. So we cannot rely on a correct sign extension.
But actually we just pass userspace delivered parameters unmodified to the system call function.
The above example could break like this:
The compiler can create code that will only test the lower 32 bits of the register that contain index to make sure its value is <= 5. However to access the memory location to get the value out of the array it can use the same unmodified (64 bit) register as index register, since the caller had to make sure that the upper 32 bits of the register contain zeroes. But since the value came from user space this isn't guaranteed and can lead to an addressing exception. Or userspace reads or even writes from/to memory locations it is not allowed to have access to.
Please note that this has nothing to do with compat system calls. This is an issue with 64 bit kernel and 64 bit userspace but 32 bit arguments.
Unfortunately the issue must be considered to be already public:
Common Vulnerabilities and Exposures assigned an identifier CVE-2009-0029 to
the following vulnerability:
The ABI in the Linux kernel 2.6.28 and earlier on s390, powerpc,
sparc64, and mips 64-bit platforms requires that a 32-bit argument in
a 64-bit register was properly sign extended when sent from a
user-mode application, but cannot verify this, which allows local
users to cause a denial of service (crash) or possibly gain privileges
via a crafted system call.
Created attachment 329100 [details]
the whole patch series that went upstream as tarball
This flaw affects most 64-bit architectures, including IBM S/390 and 64-bit PowerPC, but it does not affect x86_64 or Intel Itanium. The risks associated with fixing this flaw are greater than the security risk. We therefore currently have no plans to fix this flaw in Red Hat Enterprise Linux 3, 4, or 5. Red Hat Enterprise MRG is not affected as it is not supported on 64-bit architectures other than x86_64.