Description of problem: oops/wedge while running nptl/tst-eintr1 from glibc test suite. Version-Release number of selected component (if applicable): 2.6.33.2-41.fc13.x86_64 How reproducible: 99% Steps to Reproduce: 1.Run the test program. Hits almost every time. 2. 3. Actual results: oops text attached Expected results: Additional info: Vanilla Fedora 13ish installation; auditd is running. The test program is in the glibc sources (nptl/tst-eintr1.c). It is not immediately handy to compile it separately, though not too hard. I can supply a binary if that helps. The test does fast iterations of pthread_create/pthread_join and sending signals to threads, so the bug is probably just some race purely in the task creation/destruction paths.
Created attachment 407687 [details] oops text
In another instance, the oops had only this part. Note that this is for "ld-linux-x86-64.so.2", which is how libc's "make check" runs tst-eintr1. You get the same kernel failures running just ./tst-eintr1. ------------[ cut here ]------------ kernel BUG at kernel/cred.c:843! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:0b:00.0/irq CPU 2 Pid: 6145, comm: ld-linux-x86-64 Not tainted 2.6.33.2-41.fc13.x86_64 #1 0GU083/Precision WorkStation 490 RIP: 0010:[<ffffffff810724f3>] [<ffffffff810724f3>] __invalid_creds+0x53/0x57 RSP: 0018:ffff880006a03da0 EFLAGS: 00010286 RAX: 000000000000e870 RBX: ffff880128a11000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000246 RBP: ffff880006a03dc0 R08: 000000000000e9b2 R09: ffffffff8107ca88 R10: ffffffff81a569f8 R11: 0000000000000000 R12: ffff880128a11000 R13: ffff880006bd07d0 R14: ffff880127b59448 R15: ffff880127b6dd68 FS: 00007f1e9cdfa710(0000) GS:ffff880006a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f1e9f5fdff8 CR3: 000000010f2c3000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process ld-linux-x86-64 (pid: 6145, threadinfo ffff88011fc98000, task ffff88011fc94920) Stack: ffffea00000000c2 ffffffff8177950e ffff880006a03dd0 ffff880128a11000 <0> ffff880006a03df0 ffffffff8107267b ffff8800000000c2 ffffffff8177950e <0> ffff880006a03e30 ffff880127b58000 ffff880006a03e10 ffffffff81072d63 Call Trace: <IRQ> [<ffffffff8107267b>] __validate_creds+0x30/0x37 [<ffffffff81072d63>] exit_creds+0x41/0xbd [<ffffffff81050c43>] __put_task_struct+0x6a/0xa6 [<ffffffff81052f11>] put_task_struct+0x19/0x1b [<ffffffff8105310b>] delayed_put_task_struct+0x6d/0x72 [<ffffffff810b1b8d>] __rcu_process_callbacks+0x1b1/0x2df [<ffffffff81072268>] ? cpu_clock+0x43/0x5e [<ffffffff810b1ce6>] rcu_process_callbacks+0x2b/0x4f [<ffffffff81057b2f>] __do_softirq+0xf8/0x1cd [<ffffffff81079d67>] ? tick_program_event+0x2a/0x2c [<ffffffff8100abdc>] call_softirq+0x1c/0x30 [<ffffffff8100c3e3>] do_softirq+0x4b/0xa3 [<ffffffff8105771a>] irq_exit+0x4a/0x8c [<ffffffff8147fc11>] smp_apic_timer_interrupt+0x8e/0x9c [<ffffffff8100a693>] apic_timer_interrupt+0x13/0x20 <EOI> [<ffffffff8147a8f5>] ? _raw_spin_unlock_irq+0x37/0x3c [<ffffffff8147a8ee>] ? _raw_spin_unlock_irq+0x30/0x3c [<ffffffff81048b85>] finish_task_switch+0x74/0xdb [<ffffffff81048b11>] ? finish_task_switch+0x0/0xdb [<ffffffff81478127>] schedule+0x865/0x921 [<ffffffff8107cabe>] ? lock_release_holdtime+0x34/0xe3 In all cases I've observed, it's wedged hard after this (sysrq does not work).
Can I see the two lines before the cut-here line, please? And, also, can I have a copy of the test program?
There is no console output before that (the preceding line in the console log is a login prompt). On my latest attempt, the first run worked OK and the second run produced this different crash: ------------[ cut here ]------------ kernel BUG at kernel/cred.c:875! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1c.0/0000:0b:00.0/irq CPU 3 Pid: 1749, comm: tst-eintr1 Not tainted 2.6.33.2-41.fc13.x86_64 #1 0GU083/Precision WorkStation 490 RIP: 0010:[<ffffffff810725f7>] [<ffffffff810725f7>] __validate_process_creds+0xb0/0xdc RSP: 0018:ffff8800b9a2fea8 EFLAGS: 00010292 RAX: 000000000000002a RBX: ffff8800bf788000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000246 RBP: ffff8800b9a2fec8 R08: 000000000000ec20 R09: ffffffff8107ca88 R10: ffffffff81a569f8 R11: 0000000000000000 R12: ffffffff8177950e R13: 0000000000000379 R14: 0000000000000000 R15: ffff8800bf768000 FS: 00007fe9cd7fb710(0000) GS:ffff880006c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fe9afffeff8 CR3: 00000000bf7f6000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process tst-eintr1 (pid: 1749, threadinfo ffff8800b9a2e000, task ffff8800bf788000) Stack: ffff8800b9a2feb8 ffff8800bf788000 0000000000000000 0000000000000000 <0> ffff8800b9a2fed8 ffffffff81072649 ffff8800b9a2ff68 ffffffff8105543d <0> ffff8800bf788000 ffffffff81009caa ffff8800bf788300 ffff8800bf788300 Call Trace: [<ffffffff81072649>] validate_creds_for_do_exit+0x26/0x28 [<ffffffff8105543d>] do_exit+0x78a/0x7a5 [<ffffffff81009caa>] ? sysret_check+0x2e/0x69 [<ffffffff81479d29>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff8105553a>] complete_and_exit+0x0/0x26 [<ffffffff81009c72>] system_call_fastpath+0x16/0x1b Code: 04 00 00 48 3b bb 80 04 00 00 74 11 48 89 da 48 c7 c6 47 95 77 81 e8 b5 fd ff ff eb 0e 48 c7 c7 51 95 77 81 31 c0 e8 b3 50 40 00 <0f> 0b eb fe 48 89 c7 e8 f4 fe ff ff 84 c0 75 89 48 8b bb 88 04 RIP [<ffffffff810725f7>] __validate_process_creds+0xb0/0xdc RSP <ffff8800b9a2fea8> ---[ end trace bed8fa0cbb47d4fb ]--- Fixing recursive fault but reboot is needed! CRED: Invalid credentials CRED: At kernel/cred.c:200 CRED: Specified credentials: ffff88010f772d00 [real][eff] CRED: ->magic=43736564, put_addr=(null) CRED: ->usage=671, subscr=671 CRED: ->*uid = { 5281,5281,5281,5281 } CRED: ->*gid = { 5281,5281,5281,5281 } CRED: ->security is ffff88012851d9c0 CRED: ->security {60d, 60d} BUG: scheduling while atomic: tst-eintr1/12203/0x10000100 INFO: lockdep is turned off. Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss autofs4 sunrpc ipv6 cpufreq_ondemand acpi_cpufreq freq_table uinput snd_hda_codec_atihdmi snd_hda_codec_idt i5k_amb iTCO_wdt snd_hda_intel iTCO_vendor_support snd_seq hwmon snd_hda_codec snd_seq_device tg3 snd_hwdep snd_pcm
Created attachment 407915 [details] test program binary
(In reply to comment #4) > > [<ffffffff81072649>] validate_creds_for_do_exit+0x26/0x28 > [<ffffffff8105543d>] do_exit+0x78a/0x7a5 > [<ffffffff81009caa>] ? sysret_check+0x2e/0x69 > [<ffffffff81479d29>] ? trace_hardirqs_on_thunk+0x3a/0x3f > [<ffffffff8105553a>] complete_and_exit+0x0/0x26 > [<ffffffff81009c72>] system_call_fastpath+0x16/0x1b So strange... How it is possible that tst-eintr1 calls complete_and_exit ? I failed to install Fedora 13 on the testing machine, tried 3 times... I used Fedora-rawhide-2010xxxx, is it supposed to work?
Kernel backtraces are fuzzy and can have false frames from old bits sitting on the stack reused later. Rawhide is F14 now, not F13. I'm not sure about the installability of F13 right now. I think on my test machine I may have first installed F12 and then upgraded, but I don't recall any more. You can surely get some generic Fedora help in getting your F13 installation done.
Created attachment 408029 [details] Fix for race in validation creds_are_invalid() reads both cred->usage and cred->subscribers and then compares them to make sure the number of processes subscribed to a cred struct never exceeds the refcount of that cred struct. The problem is that this can cause a race with both copy_creds() and exit_creds() as the two counters, whilst they are of atomic_t type, are only atomic with respect to themselves, and not atomic with respect to each other. This means that if creds_are_invalid() can read the values on one CPU whilst they're being modified on another CPU, and so can observe an evolving state in which the subscribers count now is greater than the usage count a moment before. Switching the order in which the counts are read cannot help, so the thing to do is to remove that particular check. Note that this can only happen if CONFIG_DEBUG_CREDENTIALS is enabled. The problem is only likely to occur with multithreaded programs.
(In reply to comment #6) > > I failed to install Fedora 13 on the testing machine, tried 3 times... > I used Fedora-rawhide-2010xxxx, is it supposed to work? Try the F-13 beta.
I added the patch to 2.6.33.2-60.fc13 and it seems to work. Please make sure this patch gets upstream into -stable trees.
kernel-2.6.33.3-72.fc13 has been submitted as an update for Fedora 13. http://admin.fedoraproject.org/updates/kernel-2.6.33.3-72.fc13