Red Hat Bugzilla – Bug 154221
Thread exits siliently via __RESTORE_ALL exeception for iret
Last modified: 2007-11-30 17:07:17 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0
Description of problem:
So related to 152012, I instrumented the kernel to find out how our thread "disappeared" when running a Java thread test with JRockit. That is the thread did not complete, but was destroyed by the OS.
The last thing the thread was doing was trying to execute an illegal instruction, for which we have a signal handler for (SA_RESTART|SA_SIGINFO|SA_ONSTACK).
Sure enough the thread went through do_exit, but did not originate from the system call or a signal. After exhausting all other possibilities I instrumented the one last candidate, entry.S macro __RESTORE_ALL has a kernel exception fix up (333,666) for a fail iret (will attach patch with instrumentation).
We are basically faced with two problems:
1) __RESTORE_ALL fixup code simply calls "do_exit(11)", causing my thread to EXIT SILENTLY. This can't be right ? We need some notification/explanation: warning in syslog, core dump, kernel dump ?!
2) We have no idea why the iret instruction is failing, further more we have no simple 1 page repro case, currently running a JVM for repro.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Run Java thread tests.
2. Start causing general protection faults, via use of an illegal instruction.
Actual Results: Signal stack is completely unused, the thread has exited, and it's last instruction is 0x6f (outs).
Expected Results: Expecting to catch SIGSEGV in a signal handler, and handle the illegal instruction.
Created attachment 112859 [details]
syslog results of instrumentation
Created attachment 112860 [details]
Added this to the kernel-2.6 spec and rpmbuild -bb --target i686
It is possible for us to supply a repro case involving the JVM.
The exception happens because of trying to restore a bad value into a segment
register, %ds or %es. From the registers in your trace output, it looks like
%ds and %es both have bogus values (normally both should match %ss, 0x7b). If
you clobbered the %ds before the syscall, I think that would cause a trap at the
offending segment register load--and it's unlikely you do any segment register
loads anyway (unless you are running garbage instructions). If the syscall is a
sigreturn, then that will try to load whatever %ds value is in the struct
sigcontext you passed it, so that would explain it if you clobbered the struct
sigcontext on the stack, either before the call or potentially in a race where a
different thread actually clobbers part of your stack.
I'd certainly call it a bug that the process just silently dies here.
It ought to give you the same signal it would if you tried the same bad segment
register load in your own code. The upstream kernel still does this the same
way, I'll look into making it give a signal instead.
Oh sorry, I misread the code--you are certainly hitting a fault in the iret
itself rather than the segment register loads. I believe there will be a trap
at the iret instruction if the %cs or %ss being restored is invalid, or if the
PC being returned to is outside the segment limits. In the Red Hat kernels, %cs
has a segment limit to implement the exec-shield functionality when the
processor does not support the NX page table bit (which is new in processors in
the last year or less). So, a bad PC value here can get you a GP fault in the
iret rather than just a page fault after the iret returns to user mode, but with
the same meaning as the SIGBUS you'll get there if the PC is invalid but below
the segment limit (which all PCs are when the NX page table protection is
available). This still means the likelihood is a clobbered sigcontext being
passed to sigreturn--but it's the cs, ss, and eip fields that are what get to
this failure mode.
Created attachment 112875 [details]
This test case reproduces the dying with report of SIGSEGV, but not actually
generating the signal. You can tell (after ulimit -c unlimited) because it
doesn't dump a core. Compile the test with -DWHATSIG=SIGSEGV, and you can tell
because it doesn't iterate back into the signal handler after returning the
On x86_64, with this program built either 32-bit or 64-bit, the same problem
exists but the wait status is totally bogus (113, which is -9999&0xff) rather
than 11 (SIGSEGV).
Created attachment 112876 [details]
second reproducer using ptrace instead of sigreturn
This is another way to reproduce the same bug, which can also manifest if the
bogus state is poked in by ptrace rather than by a signal handler returning
after clobbering the sigcontext.
Created attachment 112881 [details]
replacement for ptrace reproducer, also bites on native x86_64
This one produces a fault in iret (I think) on x86_64--the failure mode is even
Just a note about the signal reproducer...in our complex case we have
sigaltstack for SIGSEGV, and we know we are trying to cause SIGSEGV via "outs"
instruction generating GP fault, yet the sig stack seems completely unused,
instrumenting our user code showed us that particular thread never entered any
signal handler (none that we know of)...worried is wasn't a simple sigcontext
Which brings me to "part 2" of our problem: what really caused the iret fault ?
Assuming fixing the first part of the problem; providing some kind of
diaganostics (signal/panic) showing the real problem, we'd be happy to test such
a patch or suggestions/request for further information.
Created attachment 112989 [details]
RHEL4 kernel patch
This patch makes the RHEL4 i686 kernel report a proper signal for this
so it can be debugged via ptrace or produce a core dump.
I can't much speculate on the underlying problem with the information at hand.
The patch I've attached should make it easier to diagnose what's going on.
Created attachment 113101 [details]
gdb session with crashed process
Still running the JVM repro, spawning and waiting on threads, performing
illegal outs and handling the signals...seems we get SIGSEGV trapno 13 in
Created attachment 113142 [details]
Repro source, executable (RHAS21 compiled) and core file
Finally managed to create a "simple reproducer". The magic here was to compile
the code on RHAS2.1 and then run on RHEL4. Crashes within the minute (included
example core file from iretfault patch RHEL4). I get the same SIGSEGV
(trapno=13) on my illegal code.
Running test the program on the standard kernel (unpatched) reproduce the
previous problem off a thread "missing", ie do_exit from iret fault.
I can reproduce it using the 2.1 built binary, and will look into what's going on.
But I'd like to move that case into a separate bug report. This bug 154221 is
for the failure to produce a proper core dump, which we now have a fix for.
There should be a separate report for the comment #15 problem, which is still
under investigation. If that leads to a kernel fix, it will be a separate one
from the 154221 fix.
Sure, no problem, please let us know the new bug number.
Thanks for your work on the iret fault btw.
See bug 154972 looking at the "mtbadcode" test case.
This bug will track the fix for properly dumping core when all such faults happen.
Created attachment 113589 [details]
RHEL4 kernel patch
This patch for RHEL4 kernels fixes this problem in a way that interacts well
with the exec-shield support code not in upstream kernels.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.