Description of problem: This is on a Thinkpad T420s with "ThinkPad USB Keyboard with TrackPoint" connected via USB, with F19 installed [but using rawhide kernels] 1. boot with kernel-3.11.0-0.rc7.git4.1.fc21.x86_64 2. Notice the trackpoint on the external usb thinkpad keyboard does not work. 2. [just so as to see any console messages I switch to VT2 with Alt-Ctl-F2] 3. unplug the usb cable [to the thinkpad keyboard/mouse] and then reconnect. 4. observe the stack trace in /var/log/messages. Note: I noticed this issue with kernel-3.11.0-1.fc20.x86_64. However with this kernel - I see hard lockup when I reconnect the thinkpad keyboard. [with no entries in /var/log/messages] - So I installed rc7.git1-5 kernels. kernel-3.11.0-0.rc7.git3.1.fc21.x86_64 worked fine - but kernel-3.11.0-0.rc7.git4.1.fc21.x86_64 gave the stack trace. Additional info: reporter: libreport-2.1.6 ODEBUG: free active (active state 0) object type: timer_list hint: hid_retry_timeout+0x0/0x70 Modules linked in: ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle tun bridge stp llc rfcomm bnep ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ip6table_filter ip6_tables snd_hda_codec_hdmi arc4 snd_hda_codec_conexant iwldvm mac80211 snd_hda_intel snd_hda_codec snd_hwdep iwlwifi iTCO_wdt cfg80211 snd_seq sdhci_pci sdhci iTCO_vendor_support snd_seq_device btusb snd_pcm i2c_i801 hid_lenovo_tpkbd mmc_core lpc_ich mfd_core bluetooth thinkpad_acpi snd_page_alloc snd_timer snd e1000e rfkill ptp pps_core wmi mperf soundcore nfsd auth_rpcgss nfs_acl lockd sunrpc binfmt_misc uinput i915 i2c_algo_bit drm_kms_helper drm i2c_core video CPU: 3 PID: 34 Comm: khubd Not tainted 3.11.0-0.rc7.git4.1.fc21.x86_64 #1 Hardware name: LENOVO 417152U/417152U, BIOS 8CET54WW (1.34 ) 07/11/2012 0000000000000009 ffff880210c4b9a0 ffffffff817221a6 ffff880210c4b9e8 ffff880210c4b9d8 ffffffff8107464d ffff88020ba1f9e0 ffffffff81c3ee60 ffffffff81a33edc ffffffff82bc1c50 0000000000000002 ffff880210c4ba38 Call Trace: [<ffffffff817221a6>] dump_stack+0x54/0x74 [<ffffffff8107464d>] warn_slowpath_common+0x7d/0xa0 [<ffffffff810746bc>] warn_slowpath_fmt+0x4c/0x50 [<ffffffff81383113>] debug_print_object+0x83/0xa0 [<ffffffff815b1000>] ? usbhid_open+0xf0/0xf0 [<ffffffff8138426b>] debug_check_no_obj_freed+0x20b/0x250 [<ffffffff8117e4e5>] free_pages_prepare+0x1b5/0x2b0 [<ffffffff815b0b0f>] ? usbhid_disconnect+0x2f/0x50 [<ffffffff8117f684>] __free_pages+0x34/0x80 [<ffffffff8117f8b2>] __free_memcg_kmem_pages+0x22/0x50 [<ffffffff811cfe10>] kfree+0x2a0/0x2f0 [<ffffffff815b0b0f>] usbhid_disconnect+0x2f/0x50 [<ffffffff81507427>] usb_unbind_interface+0x67/0x1d0 [<ffffffff8148616f>] __device_release_driver+0x7f/0xf0 [<ffffffff81486205>] device_release_driver+0x25/0x40 [<ffffffff814859a8>] bus_remove_device+0x108/0x190 [<ffffffff814822c2>] device_del+0x142/0x1e0 [<ffffffff81504e30>] usb_disable_device+0xb0/0x270 [<ffffffff814fae65>] usb_disconnect+0xb5/0x1d0 [<ffffffff814fccea>] hub_thread+0x74a/0x1710 [<ffffffff810a1370>] ? wake_up_atomic_t+0x30/0x30 [<ffffffff814fc5a0>] ? hub_port_debounce+0x130/0x130 [<ffffffff810a00ad>] kthread+0xed/0x100 [<ffffffff8109ffc0>] ? insert_kthread_work+0x80/0x80 [<ffffffff81734fec>] ret_from_fork+0x7c/0xb0 [<ffffffff8109ffc0>] ? insert_kthread_work+0x80/0x80
Created attachment 793277 [details] File: dmesg
The problem goes away if I rebuild kernel-3.11.0-2.fc21.src.rpm without HID-CVE-fixes.patch
(In reply to Satish Balay from comment #2) > The problem goes away if I rebuild kernel-3.11.0-2.fc21.src.rpm without > HID-CVE-fixes.patch Hm. That's a patch from upstream to fix a large number of CVEs. Apparently something isn't quite right with it. Though to be clear, the trackpoint on the keyboard works fine as well, or you just don't see the ODEBUG messages?
(In reply to Josh Boyer from comment #3) > Though to be clear, the trackpoint on the keyboard works fine as well, or > you just don't see the ODEBUG messages? with the HID-CVE-fixes.patch removed, the trackpoint on the external thinkpad keyboard works fine. [and there is no crash on removing and replugging in the usb cable for the keyboard]. This is the same behavior as rc7.git3
Thanks for the bug report and preliminary testing. This bug is twofold: - the CVE fix seems to be broken at some point - the error path in the tpkbd_probe() function is wrong because it does not clean the input devices if tpkbd_probe_tp() fails (which was raised by the CVE fix). To fix and test this, I would need to have access to the hid-recorder[1] traces of your devices (hidraw2 and 3 I would say), without the CVE fix, and with some events of each hidraw device. [1] http://bentiss.github.io/hid-replay-docs/ PS: I _think_ your kernel will oops if you use hid-replay on the recordings you are about to do. Just make sure some events are recorded, and I'll break my own kernel if it does :)
Created attachment 794885 [details] /dev/hidraw0 [root@asterix ~]# hid-recorder > mydevice1.hid Available devices: /dev/hidraw0: Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint /dev/hidraw1: Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint Select the device event number [0-1]: 0
Created attachment 794887 [details] /dev/hidraw1 [root@asterix ~]# hid-recorder > mydevice2.hid Available devices: /dev/hidraw0: Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint /dev/hidraw1: Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint Select the device event number [0-1]: 1 Note: This is captured 3.11.0-2 rebuilt without HID-CVE-fixes.patch. Since the keyboard/mouse is detected as 2 devices [and hid-recorder is asking me to select one of them] - I'm attaching 2 logs - one for each device. I've attempted to create a few key strokes and a few mouse events for these logs. Just to clarify - [as you suggest] there are perhaps multiple issues here. The first is: - the mouse events are not visible to the kernel from this keyboard. secondly: when I unplug the usb cable [with the rc7.git4 (debug) kernel - I get the stack trace reported on this bugzilla. Thirdly: - if I use the nodebug 3.11.0-2 kernel - I get a freeze/pannic when I reattach the usb cable.[which I couldn't capture in logs.]. Today I noticed that [when I'm in VT2 mode] - I had to do this a few times before the panic would trigger - and during one attempt I noticed a non-panic stack trace - below: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sep 6 11:12:51 asterix kernel: [ 109.764210] usb 1-1.2: new low-speed USB device number 6 using ehci-pci Sep 6 11:12:51 asterix kernel: [ 109.855342] usb 1-1.2: New USB device found, idVendor=17ef, idProduct=6009 Sep 6 11:12:51 asterix kernel: [ 109.855354] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Sep 6 11:12:51 asterix kernel: [ 109.855360] usb 1-1.2: Product: ThinkPad USB Keyboard with TrackPoint Sep 6 11:12:51 asterix kernel: [ 109.855364] usb 1-1.2: Manufacturer: Lite-On Technology Corp. Sep 6 11:12:51 asterix kernel: [ 109.860530] input: Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.0/input/input16 Sep 6 11:12:51 asterix kernel: [ 109.860869] lenovo_tpkbd 0003:17EF:6009.0005: input,hidraw0: USB HID v1.10 Keyboard [Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint] on usb-0000:00:1a.0-1.2/input0 Sep 6 11:12:51 asterix mtp-probe: checking bus 1, device 6: "/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2" Sep 6 11:12:51 asterix mtp-probe: bus: 1, device: 6 was not an MTP device Sep 6 11:12:51 asterix kernel: [ 109.870136] input: Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.1/input/input17 Sep 6 11:12:51 asterix kernel: [ 109.870842] lenovo_tpkbd 0003:17EF:6009.0006: input,hiddev0,hidraw3: USB HID v1.10 Mouse [Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint] on usb-0000:00:1a.0-1.2/input1 Sep 6 11:12:51 asterix kernel: [ 109.870845] lenovo_tpkbd 0003:17EF:6009.0006: missing HID_OUTPUT_REPORT 4 Sep 6 11:12:51 asterix kernel: [ 109.875856] ------------[ cut here ]------------ Sep 6 11:12:51 asterix kernel: [ 109.875872] WARNING: CPU: 0 PID: 9 at lib/list_debug.c:33 __list_add+0xac/0xc0() Sep 6 11:12:51 asterix kernel: [ 109.875875] list_add corruption. prev->next should be next (ffffffff81eb3978), but was (null). (prev=ffff880206766898). Sep 6 11:12:51 asterix kernel: [ 109.875922] Modules linked in: joydev hid_lenovo_tpkbd ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle tun bridge stp llc rfcomm bnep ip6t_REJECT nf_conntrack_ipv4 nf_conntrack_ipv6 nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter xt_conntrack nf_conntrack ip6_tables arc4 tpm_tis tpm iwldvm mac80211 iTCO_wdt iTCO_vendor_support lpc_ich btusb mfd_core snd_hda_codec_hdmi snd_hda_codec_conexant i2c_i801 tpm_bios snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm bluetooth e1000e sdhci_pci ptp sdhci pps_core mmc_core iwlwifi cfg80211 thinkpad_acpi snd_page_alloc snd_timer snd mperf rfkill wmi soundcore nfsd uinput auth_rpcgss nfs_acl lockd binfmt_misc sunrpc i915 i2c_algo_bit drm_kms_helper drm i2c_core video Sep 6 11:12:51 asterix kernel: [ 109.875925] CPU: 0 PID: 9 Comm: rcu_sched Not tainted 3.11.0-2.fc21.x86_64 #1 Sep 6 11:12:51 asterix kernel: [ 109.875928] Hardware name: LENOVO 417152U/417152U, BIOS 8CET54WW (1.34 ) 07/11/2012 Sep 6 11:12:51 asterix kernel: [ 109.875931] 0000000000000009 ffff880212949c90 ffffffff81641d9b ffff880212949cd8 Sep 6 11:12:51 asterix kernel: [ 109.875934] ffff880212949cc8 ffffffff810670dd ffff880212949da0 ffffffff81eb3978 Sep 6 11:12:51 asterix kernel: [ 109.875936] ffff880206766898 ffffffff81eb3280 000000000000e088 ffff880212949d28 Sep 6 11:12:51 asterix kernel: [ 109.875937] Call Trace: Sep 6 11:12:51 asterix kernel: [ 109.875943] [<ffffffff81641d9b>] dump_stack+0x45/0x56 Sep 6 11:12:51 asterix kernel: [ 109.875948] [<ffffffff810670dd>] warn_slowpath_common+0x7d/0xa0 Sep 6 11:12:51 asterix kernel: [ 109.875950] [<ffffffff8106714c>] warn_slowpath_fmt+0x4c/0x50 Sep 6 11:12:51 asterix kernel: [ 109.875953] [<ffffffff8130eaec>] __list_add+0xac/0xc0 Sep 6 11:12:51 asterix kernel: [ 109.875957] [<ffffffff81072a08>] __internal_add_timer+0xc8/0x130 Sep 6 11:12:51 asterix kernel: [ 109.875960] [<ffffffff81072ce7>] internal_add_timer+0x17/0x40 Sep 6 11:12:51 asterix kernel: [ 109.875964] [<ffffffff81644a01>] schedule_timeout+0x161/0x2c0 Sep 6 11:12:51 asterix kernel: [ 109.875967] [<ffffffff81072a70>] ? __internal_add_timer+0x130/0x130 Sep 6 11:12:51 asterix kernel: [ 109.875971] [<ffffffff810890c6>] ? prepare_to_wait+0x56/0x90 Sep 6 11:12:51 asterix kernel: [ 109.875975] [<ffffffff810fe4df>] rcu_gp_kthread+0x2ff/0x5c0 Sep 6 11:12:51 asterix kernel: [ 109.875978] [<ffffffff81089380>] ? wake_up_atomic_t+0x30/0x30 Sep 6 11:12:51 asterix kernel: [ 109.875980] [<ffffffff810fe1e0>] ? rcu_gp_fqs+0x80/0x80 Sep 6 11:12:51 asterix kernel: [ 109.875983] [<ffffffff810885c0>] kthread+0xc0/0xd0 Sep 6 11:12:51 asterix kernel: [ 109.875986] [<ffffffff81088500>] ? insert_kthread_work+0x40/0x40 Sep 6 11:12:51 asterix kernel: [ 109.875989] [<ffffffff81650eac>] ret_from_fork+0x7c/0xb0 Sep 6 11:12:51 asterix kernel: [ 109.875992] [<ffffffff81088500>] ? insert_kthread_work+0x40/0x40 Sep 6 11:12:51 asterix kernel: [ 109.875994] ---[ end trace b4937772633d3a49 ]--- <<<<<<<<<<<<<< If the stack strace from the subsequent panic is useful - I can try reproducing so that the stack trace is visible on VT2 - and caputre with a camera and post.
Thanks for the logs. I have prepared a repo for you to test: https://github.com/bentiss/hid-lenovo-tpkbd just clone, make and run: $> sudo rmmod hid-lenovo-tpkbd ; sudo insmod ./hid_lenovo_tpkbd_backport.ko Just for double checking, un-plug, re-plug the keyboard If it does works, please check that lsmod shows only the backported version of hid-lenovo-tpkbd, and drop a message here. I'll send the modifications upstream.
(In reply to Benjamin Tissoires from comment #8) > Thanks for the logs. > I have prepared a repo for you to test: > https://github.com/bentiss/hid-lenovo-tpkbd > > just clone, make and run: > $> sudo rmmod hid-lenovo-tpkbd ; sudo insmod ./hid_lenovo_tpkbd_backport.ko > > Just for double checking, un-plug, re-plug the keyboard > > If it does works, please check that lsmod shows only the backported version > of hid-lenovo-tpkbd, and drop a message here. I'll send the modifications > upstream. I tried this with kernel-3.11.0-0.rc7.git0.2.fc21.x86_64 [debug + HID-CVE-fixes.patch] The above gave me a kernel panic - so I did the following alternative. 1. reboot with the keyboard unhooked. 2. verified hid-lenovo-tpkbd is not loaded [root@asterix ~]# lsmod |grep hid [root@asterix ~]# 3. loaded the backport driver and verified that its loaded [root@asterix hid-lenovo-tpkbd]# insmod ./hid_lenovo_tpkbd_backport.ko [root@asterix hid-lenovo-tpkbd]# lsmod |grep hid hid_lenovo_tpkbd_backport 13846 0 [root@asterix hid-lenovo-tpkbd]# 4. Now plugged in the USB keyboard. Now I see the mouse works fine. 5. Now I disconnected and reconnected it a few times - and It worked fine! [I had no tracebacks or panics] thanks!
Thanks for the tests. The patches have now been sent to the LKML. Hopefully, they will get pushed to Linus soon, and Josh may be able to do something for Fedora. Josh, a "simple" fix for that can be to change the line 844 of the file HID-CVE-fixes.patch from + if (!hid_validate_report(hdev, HID_OUTPUT_REPORT, 4, 4, 1) || to + if (!hid_validate_report(hdev, HID_FEATURE_REPORT, 4, 4, 1) || And then, you can wait for dropping this when it goes to stable.
OK, great. I'll get this fixed up tomorrow.
Satish, I've done a scratch build with v3 of the fixes from Benjamin included. Could you please test this when it finished building and make sure the issue is resolved with this build? http://koji.fedoraproject.org/koji/taskinfo?taskID=5927471
The x86_64 build is done now: http://koji.fedoraproject.org/koji/taskinfo?taskID=5927473
(In reply to Josh Boyer from comment #13) > The x86_64 build is done now: > > http://koji.fedoraproject.org/koji/taskinfo?taskID=5927473 I've installed and booted to kernel-3.11.0-300.1.fc20.x86_64 from above - and its not showing the problem. The mouse on the usb keyboard does work with this kernel - and I've attempted to disconnect/reconnect a few times -without any stack traces. thanks!
Excellent, thank you for testing. I've pushed that to the Fedora git repo and it should be in the next official build/update.
kernel-3.11.1-300.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/kernel-3.11.1-300.fc20
Package kernel-3.11.1-300.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing kernel-3.11.1-300.fc20' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-16794/kernel-3.11.1-300.fc20 then log in and leave karma (feedback).
kernel-3.11.1-200.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/kernel-3.11.1-200.fc19
kernel-3.10.12-100.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/kernel-3.10.12-100.fc18
*** Bug 1009394 has been marked as a duplicate of this bug. ***
I'm running kernel-3.10.12-100.fc18 and it seems to work fine now. Thanks.
kernel-3.11.1-200.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
*** Bug 1009815 has been marked as a duplicate of this bug. ***
kernel-3.10.12-100.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.
kernel-3.11.1-300.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.