Bug 633940

Description of problem:

When an SMP RH5 kernel boots on a single cpu, there is code which
runs to replace the "lock",<operation> sequences with "nop",<operation>.
This in general works fine and supposedly it offers a benefit performance
wise by avoiding the expense of locked bus operations when there is no
other cpu requiring the operation to be interlocked.

However, the futex code [specifically the PI futex code] does atomic
handoffs with user addresses complete with exception handling in case
the user page is not resident and needs to be faulted in to complete
the operation. In general this works because there is a preceding
get_user(uval, uaddr) [before locking down mmap_sem] to acquire the
initial value of the futex. If however acquiring the mmap_sem for read
blocks, it is possible that the page containing the futex word will fault
and if so, the fixup table addresses are off-by-1 if the UP lock => nop
replacement has been done. In short, the exception address is the original
address+1 and never being found causes the kernel to not handle the fault
and OOPS on the page fault (as if the kernel were doing something bad).

Here is an example of what that looks like:

<1>Unable to handle kernel paging request at 000000000bdd99b8 RIP: 
<1> [<ffffffff80060b93>] do_futex+0x1333/0x1490
<4>PGD 346146067 PUD 32bbcc067 PMD 341c73067 PTE 800000032e5a2065
<0>Oops: 0003 [1] SMP 
<1>last sysfs file: /class/scsi_host/host0/proc_name

[0]kdb> rd
     r15 = 0x0000000000000000      r14 = 0x000000000bdd99b8 
     r13 = 0x000000008000591d      r12 = 0x0000000000000000 
     rbp = 0xffffffff8060fc78      rbx = 0x0000000000000000 
     r11 = 0x0000000000000000      r10 = 0x00000000ffffffff 
      r9 = 0x0000000000000015       r8 = 0xffff81032e5ca000 
     rax = 0x000000008000591d      rcx = 0x0000000000000000 
     rdx = 0x00000000fffffff2      rsi = 0x00000000fdfbd800 
     rdi = 0xffffffff8060fc70 orig_rax = 0xffffffffffffffff 
     rip = 0xffffffff80060b93       cs = 0x0000000000000010 
  eflags = 0x0000000000210246      rsp = 0xffff81032e5cbc90 
      ss = 0x0000000000000000 &regs = 0xffff81032e5cbbf8
[0]kdb> bt
Stack traceback for pid 22813
0xffff81032e5af100    22813    22675  1    0   R  0xffff81032e5af390 *process
rsp                rip                Function (args)
0xffff81032e5cbc78 0xffffffff80060b93 do_futex+0x1333
0xffff81032e5cbca0 0xffffffff80060a14 do_futex+0x11b4
0xffff81032e5cbce0 0xffffffff800797be unlock_page+0x2e
0xffff81032e5cbd30 0xffffffff8007cb1b filemap_nopage+0x19b
0xffff81032e5cbe38 0xffffffff8003cab0 default_wake_function
0xffff81032e5cbe70 0xffffffff8002e607 do_page_fault+0x4b7
0xffff81032e5cbe80 0xffffffff8033d0d8 thread_return+0x62
0xffff81032e5cbec0 0xffffffff8000f4c3 do_gettimeofday+0x43
0xffff81032e5cbf10 0xffffffff80061349 compat_sys_futex+0x119
0xffff81032e5cbf80 0xffffffff80030c72 ia32_sysret


Version-Release number of selected component (if applicable): Any RHEL5 kernel.


How reproducible:

It is non-trivial to reproduce this, but it should be clear that
futex_atomic_cmpxchg_inatomic() generates the exception address
where the "lock","cmpxchg ..." sequence is and AFAIK as long as
the "lock" prefix is there, it will fault on the saved exception
address. If the "lock" is replaced with a "nop", then the fault
address is the address of the cmpexch instruction and the exception
lookup will fail if it faults [which is the hard part]. I suspect
that a thread doing something to the address space of the process
requiring write lock on mmap_sem is the culprit. All that has to
happen is for the page to fault when the atomic update of the futex
takes place and it will fault [as above].


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

I am working on a piece of code that for i386/x86_64 would adjust the
exception table addresses dynamically [after they have been sorted].
Alternatively, it would be easy to augment the exception table lookup
with another table that is computed from the intersection of the nop
addresses and the exception table for the kernel. Doing that would add
slight overhead for exception lookup, but would only happen if the
main exception table lookup fails. We are talking about < 16 additional
exception table entries if we augment the search (after failing the
main kernel and module lookup).