Fedora Account System
Red Hat Associate
Red Hat Customer
The current and the one-prior versions of the utrace patch in fedora kernels both suffer from a BUG_ON problem that can lead to a subsequent kernel crash. See reports at: http://sourceware.org/bugzilla/show_bug.cgi?id=6964 http://www.kerneloops.org/search.php?search=utrace_get_signal&btnG=Function+Search
Can you test 2.6.26.6-79? It's in updates-testing now.
I may be mistaken, but I see no changes to the linux-2.6-utrace.patch in the fedora cvs, so this code is unlikely to have been touched by that update. Are you suggesting testing it just for completeness, or do you have reason to believe that this bug may have been fixed?
If you tested 2.6.26.6-75 or later then it's not fixed. BTW where did you get a working systemtap package for Fedora 9? I was told at the kernel summit that it's been broken for a long time.
> If you tested 2.6.26.6-75 or later then it's not fixed. http://sourceware.org/bugzilla/show_bug.cgi?id=6964#c5 > BTW where did you get a working systemtap package for Fedora 9? From Fedora updates, or from systemtap git. > I was told at the kernel summit that it's been broken for a long time. Most unfortunate misinformation.
It would be ideal to produce a simple utrace module with human-comprehensible source code, that can reproduce the same bug via pure use of utrace calls in the module and ptrace calls (or simpler tools like strace/gdb) in the test case.
Created attachment 324325 [details] testcase, work in progress
1. this testcase is not ready yet. 2. it requires compiled uprobe module. Create it this way: stap -p4 -e 'probe process.syscall {}' -c ls
Created attachment 324444 [details] Testcase Further work on the testcase. This testcase seems to reproduce the bug. Or a different bug, I am not sure. It oopses the kernel quite happily, that's for sure. Sometimes it's bad enough that I can't save the oops. It may need some polishing, but is in usable shape already.
Created attachment 324484 [details] Testcase v.2 Further work. I can't make it work without oopsing even without ptrace usage. This might be my error (improper usage of uprobe interface), however I don't think oopses are ok anyway. I do believe the testcase does the same steps as staprun would do. Build wiht "gcc -Os -DTALKATIVE utrace_plus_DETACH3.c" for verbosity.
with WANT_PTRACE off and WANT_UPROBE on, on 2.6.27.5-37.fc9.x86_64, all oopses are the same (tried 5 times): stap_9034400b1b15b3f5e322794b4679b8d1_184: systemtap: 0.8/0.137, base: ffffffffa05ec000, memory: 33460+24540+496+14000 data+text+ctx+net, probes: 1 ------------[ cut here ]------------ kernel BUG at kernel/utrace.c:1427! invalid opcode: 0000 [1] SMP CPU 0 Modules linked in: stap_9034400b1b15b3f5e322794b4679b8d1_184 nfs lockd nfs_acl ipt_MASQUERADE iptable_nat nf_nat bridge stp bnep rfcomm l2cap bluetooth ib_iser rdma_cm ib_cm i w_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi fuse sunrpc ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath scsi_dh kvm snd_hda_intel snd_seq_dummy snd_seq_oss s nd_seq_midi_event snd_seq arc4 snd_seq_device ecb snd_pcm_oss crypto_blkcipher snd_mixer_oss iwl3945 thinkpad_acpi snd_pcm snd_timer sr_mod cdrom snd_page_alloc rfkill firewir e_ohci snd_hwdep firewire_core joydev snd ppdev mac80211 pcspkr hwmon ata_generic e1000e sdhci_pci sdhci ricoh_mmc i2c_i801 ata_piix yenta_socket cfg80211 mmc_core soundcore p ata_acpi rsrc_nonstatic sg iTCO_wdt crc_itu_t iTCO_vendor_support video battery output ac parport_pc bay parport wmi dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci libata sd _mod scsi_mod crc_t10dif ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd i915 drm i2c_algo_bit i2c_core [last unloaded: microcode] Pid: 4827, comm: a.out Tainted: G W 2.6.27.5-37.fc9.x86_64 #1 RIP: 0010:[<ffffffff8107376d>] [<ffffffff8107376d>] start_report+0x13/0x65 RSP: 0018:ffff8800609cdd08 EFLAGS: 00010202 RAX: ffff8800609cc001 RBX: ffff880060a68870 RCX: ffff8800609cc000 RDX: ffff8800609cdfd8 RSI: ffff8800609cdf58 RDI: ffff880060a68870 RBP: ffff8800609cdd18 R08: ffff8800609cdc78 R09: ffff8800609cdd28 R10: 0000001eedbe90f0 R11: ffffffff81010349 R12: ffff880060a68870 R13: ffff88005c505b80 R14: ffff880060a68880 R15: ffff8800609cdd38 FS: 00007fd4e654b6f0(0000) GS:ffffffff814b0100(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000cf6000 CR3: 00000000665c2000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process a.out (pid: 4827, threadinfo ffff8800609cc000, task ffff88005c505b80) Stack: ffff8800609cde78 0000000000000000 ffff8800609cdd78 ffffffff81074ab2 ffff8800609cdd78 ffffffff8104cc11 0000000000000005 ffff880060000000 ffff88005c4af388 0000000000000000 0000000000400af0 ffff8800609cdf58 Call Trace: [<ffffffff81074ab2>] utrace_resume+0xaf/0x152 [<ffffffff8104cc11>] ? get_signal_to_deliver+0x2c3/0x2dc [<ffffffff8100fcaa>] do_notify_resume+0x90b/0x93f [<ffffffff81037938>] ? wake_up_state+0xb/0xd [<ffffffff8104b009>] ? signal_wake_up+0x38/0x3d [<ffffffff8104b246>] ? send_signal+0xea/0x23e [<ffffffff8104b145>] ? complete_signal+0x137/0x14e [<ffffffff8104b37d>] ? send_signal+0x221/0x23e [<ffffffff810c5bc7>] ? path_put+0x1d/0x21 [<ffffffff811523cd>] ? __put_user_4+0x1d/0x30 [<ffffffff8107ce4b>] ? audit_syscall_entry+0x101/0x135 [<ffffffff810105f4>] int_signal+0x12/0x17 Code: 49 83 7d 08 00 ba f5 ff ff ff 74 02 31 d2 59 5b 41 5c 89 d0 41 5d c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 8a 47 40 a8 01 74 04 <0f> 0b eb fe a8 02 74 46 48 8d 7f 30 e8 cb f8 24 00 48 8b 4b 20 RIP [<ffffffff8107376d>] start_report+0x13/0x65 RSP <ffff8800609cdd08> ---[ end trace a7919e7f17c0a725 ]--- BUG: unable to handle kernel paging request at ffffffffa05f6770 NB1: kernel/utrace.c:1427 is here: 1425 static void start_report(struct utrace *utrace) 1426 { 1427 BUG_ON(utrace->stopped); 1428 if (utrace->report) { 1429 spin_lock(&utrace->lock); 1430 utrace->report = 0; 1431 splice_attaching(utrace); 1432 spin_unlock(&utrace->lock); 1433 } 1434 } NB2: "Tainted: W" is due to harmless sysfs warning at boot: WARNING: at fs/sysfs/dir.c:463 sysfs_add_one+0x35/0x3d() sysfs: duplicate filename '1' can not be created
I tried: stap -e 'probe kernel.function("utrace_*").call { printf("%s %s\n",pp(),$$parms) } probe kernel.statement("*@kernel/utrace.c:1427") { printf("%s\n", pp()) }' -c ./xxx where ./xxx is Denys's test program. It produced some coherent output. Maybe more hacks like this (even try "*@kernel/utrace.c:*"???) along with maybe 'probe syscall.* if (target() == pid())' and the like would deliver a sensical picture of the important sequence of operations. (I didn't try return value probes on those utrace functions, which might help too.)
Created attachment 325844 [details] Testcase v.4 (there is no v.3 :) This testcase still crashes, now on 2.6.27.5-41.fc9.x86_64. I tried to document it better (comments in the code), but otherwise it is not much changed.
Oops on 2.6.27.5-41.fc9.x86_64 with the following settings in testcase: //#define WANT_PTRACE //#define WANT_USR1 #define WANT_UPROBE looks like this: stap_c3d7b62680ece872df42d5d174d0b5fd_184: systemtap: 0.8/0.137, base: ffffffffa05ea000, memory: 33460+24540+496+14000 data+text+ctx+net, probes: 1 ------------[ cut here ]------------ kernel BUG at kernel/utrace.c:1427! invalid opcode: 0000 [1] SMP CPU 1 Modules linked in: stap_c3d7b62680ece872df42d5d174d0b5fd_184 nfs lockd nfs_acl ipt_MASQUERADE iptable_nat nf_nat bridge stp bnep rfcomm l2cap bluetooth ib_iser rdma_cm ib_cm i w_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi fuse sunrpc ipt_REJECT nf_conntrack_ipv4 iptable_filter ip_tables ip6t_REJECT xt_tcpudp nf_conntrack_ ipv6 xt_state nf_conntrack ip6table_filter ip6_tables x_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath scsi_dh kvm snd_hda_intel snd_seq_dummy snd_seq_oss s nd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss arc4 snd_mixer_oss ecb sr_mod crypto_blkcipher cdrom thinkpad_acpi snd_pcm firewire_ohci ppdev iwl3945 snd_timer rfkill sn d_page_alloc snd_hwdep snd hwmon e1000e mac80211 sdhci_pci sdhci ricoh_mmc ata_generic firewire_core mmc_core pcspkr joydev soundcore i2c_i801 cfg80211 iTCO_wdt yenta_socket a ta_piix crc_itu_t iTCO_vendor_support sg pata_acpi rsrc_nonstatic ac video battery parport_pc output bay parport wmi dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci libata sd _mod scsi_mod crc_t10dif ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd i915 drm i2c_algo_bit i2c_core [last unloaded: microcode] Pid: 18004, comm: a.out Tainted: G W 2.6.27.5-41.fc9.x86_64 #1 RIP: 0010:[<ffffffff810736d5>] [<ffffffff810736d5>] start_report+0x13/0x65 RSP: 0018:ffff88005fd19d08 EFLAGS: 00010202 RAX: ffff88005fd18001 RBX: ffff88005fce1af8 RCX: ffff88005fd18000 RDX: ffff88005fd19fd8 RSI: ffff88005fd19f58 RDI: ffff88005fce1af8 RBP: ffff88005fd19d18 R08: ffff88005fd19c78 R09: ffff88005fd19d28 R10: 000000fed93267fc R11: ffffffff81010349 R12: ffff88005fce1af8 R13: ffff88004f8badc0 R14: ffff88005fce1b08 R15: ffff88005fd19d38 FS: 00007f41934c56f0(0000) GS:ffff88007e004880(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000038248dd390 CR3: 000000005fdc8000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process a.out (pid: 18004, threadinfo ffff88005fd18000, task ffff88004f8badc0) Stack: ffff88005fd19e78 0000000000000000 ffff88005fd19d78 ffffffff81074a1a ffff88005fd19d78 ffffffff8104cc11 0000000000000005 ffff88005f000000 ffff880060810008 0000000000000000 0000000000400af0 ffff88005fd19f58 Call Trace: [<ffffffff81074a1a>] utrace_resume+0xaf/0x152 [<ffffffff8104cc11>] ? get_signal_to_deliver+0x2c3/0x2dc [<ffffffff8100fcaa>] do_notify_resume+0x90b/0x93f [<ffffffff81037938>] ? wake_up_state+0xb/0xd [<ffffffff8104b009>] ? signal_wake_up+0x38/0x3d [<ffffffff8104b145>] ? complete_signal+0x137/0x14e [<ffffffff8104b37d>] ? send_signal+0x221/0x23e [<ffffffff810c5b2f>] ? path_put+0x1d/0x21 [<ffffffff8115232d>] ? __put_user_4+0x1d/0x30 [<ffffffff8107cdb3>] ? audit_syscall_entry+0x101/0x135 [<ffffffff810105f4>] int_signal+0x12/0x17 Code: 49 83 7d 08 00 ba f5 ff ff ff 74 02 31 d2 59 5b 41 5c 89 d0 41 5d c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 8a 47 40 a8 01 74 04 <0f> 0b eb fe a8 02 74 46 48 8d 7f 30 e8 c3 f8 24 00 48 8b 4b 20 RIP [<ffffffff810736d5>] start_report+0x13/0x65 RSP <ffff88005fd19d08>
This message is a reminder that Fedora 9 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 9. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '9'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 9's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 9 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
This is systemtap bug http://sourceware.org/bugzilla/show_bug.cgi?id=6964 Systemtap contains a workaround in runtime/staprun/mainloop for this issue. Removing that workaround still seems to crash/freeze my kernel 2.6.27.24-170.2.68.fc10.i686
Raising severity as it appears to be basic userspace operations that kill the kernel.
This bug persists in 2.6.31-0.204.rc9.fc12.i686. I have not tried to reproduce it from a pure user-space environment. It may not be as bad ad I thought though because ptrace is the only (other) utrace client in fedora kernels, and ptrace locks out other concurrent ptrace users.
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle. Changing version to '12'. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
This message is a reminder that Fedora 12 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 12. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '12'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 12's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 12 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 12 changed to end-of-life (EOL) status on 2010-12-02. Fedora 12 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.