+++ This bug was initially created as a clone of Bug #481682 +++

Created an attachment (id=330051)
proposed fix

in kernel-2.6.18-128.el5 and prior versions, arch/x86_64/ia32/ia32entry.S hunk of the linux-2.6-misc-utrace-update.patch contains incorrect optimization.
As result, out-of-table 32-bit syscalls on the x86_64 kernel do not return ENOSYS (unless the caller is being ptraced).

For example, glibc-2.9+ popen() goes mad when pipe2 syscall returns its number 331 instead of fail with ENOSYS.
As result, FC10+ 32-bit processes on RHEL5 x86_64 kernel break once popen(3) is called.

--- Additional comment from kir on 2009-01-27 04:29:59 EDT ---

The issue was found while running Fedora 10 containers on an RHEL5+OpenVZ kernel.

Relative OpenVZ bug: http://bugzilla.openvz.org/show_bug.cgi?id=1150

--- Additional comment from roland on 2009-02-06 20:59:52 EDT ---

Created an attachment (id=331182)
test case source

Simple test case, compile with -m32 and run on x86-64 kernel.

The RHEL5 code is the same (broken) as upstream.  I'll fix it upstream and then RHEL5 should backport the change so it continues to match upstream.

--- Additional comment from roland on 2009-02-06 21:02:48 EDT ---

Created an attachment (id=331183)
test case source

Test case fixed to exit 0 for correct and nonzero for bug.

Also, I forgot to note that to reproduce you have to have auditd disabled:
/sbin/chkconfig auditd off; reboot

--- Additional comment from roland on 2009-02-06 21:34:13 EDT ---

Created an attachment (id=331187)
fix posted upstream

Wait to see if upstream takes this as is, but almost certain they will.
With s,/x86/,/x86_64/,g this same patch applies to RHEL5 fine.

--- Additional comment from vmayatsk on 2009-02-07 04:57:47 EDT ---

> The RHEL5 code is the same (broken) as upstream.

RHEL5 is not broken, I wasn't able to reproduce it. As I understood,  this is due to this check:

        testl $(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SECCOMP),threadinfo_flags(%r10)
        jnz ia32_tracesys

And we have TIF_SYSCALL_AUDIT set.

Also, it's possible to make the patch shorter:

--- a/arch/x86/ia32/ia32entry.S
+++ b/arch/x86/ia32/ia32entry.S
@@ -417,6 +417,7 @@ ENTRY(ia32_syscall)
        GET_THREAD_INFO(%r10)
        orl   $TS_COMPAT,TI_status(%r10)
        testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags(%r10)
+       movq $-ENOSYS,RAX-ARGOFFSET(%rsp)
        jnz ia32_tracesys
 ia32_do_syscall:
        cmpl $(IA32_NR_syscalls-1),%eax

I've tested it yesterday, but didn't post upstream yet.

--- Additional comment from roland on 2009-02-07 17:21:41 EDT ---

It is broken.  See comment#3 on how to reproduce it.
Shorter patch does not mean shorter code path, which is what matters.
Anyway, I've already posted upstream.

--- Additional comment from roland on 2009-02-07 17:25:55 EDT ---

Upstream fix was merged: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c09249f8d1b84344eca882547afdbffee8c09d14