Bug 1003998 - [abrt] ODEBUG: free active (active state 0) object type: timer_list hint: hid_retry_timeout+0x0/0x70
[abrt] ODEBUG: free active (active state 0) object type: timer_list hint: hid...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
rawhide
x86_64 Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Benjamin Tissoires
Fedora Extras Quality Assurance
abrt_hash:2897dea3ab720faa839b503697c...
:
: 1009394 1009815 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-03 12:56 EDT by Satish Balay
Modified: 2013-09-22 20:35 EDT (History)
11 users (show)

See Also:
Fixed In Version: kernel-3.11.1-300.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-18 22:03:51 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
File: dmesg (72.26 KB, text/plain)
2013-09-03 12:56 EDT, Satish Balay
no flags Details
/dev/hidraw0 (2.35 KB, text/plain)
2013-09-06 12:27 EDT, Satish Balay
no flags Details
/dev/hidraw1 (10.18 KB, text/plain)
2013-09-06 12:39 EDT, Satish Balay
no flags Details

  None (edit)
Description Satish Balay 2013-09-03 12:56:15 EDT
Description of problem:
This is on a Thinkpad T420s with "ThinkPad USB Keyboard with TrackPoint" connected via USB, with F19 installed [but using rawhide kernels]

1. boot with kernel-3.11.0-0.rc7.git4.1.fc21.x86_64
2. Notice the trackpoint on the external usb thinkpad keyboard does not work.
2. [just so as to see any console messages I switch to VT2 with Alt-Ctl-F2]
3. unplug the usb cable [to the thinkpad keyboard/mouse] and then reconnect.
4. observe the stack trace in /var/log/messages.

Note: I noticed this issue with kernel-3.11.0-1.fc20.x86_64. However with this kernel - I see hard lockup when I reconnect the thinkpad keyboard. [with no entries in /var/log/messages] - So I installed rc7.git1-5 kernels.

kernel-3.11.0-0.rc7.git3.1.fc21.x86_64 worked fine - but kernel-3.11.0-0.rc7.git4.1.fc21.x86_64 gave the stack trace.

Additional info:
reporter:       libreport-2.1.6
ODEBUG: free active (active state 0) object type: timer_list hint: hid_retry_timeout+0x0/0x70
Modules linked in: ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle tun bridge stp llc rfcomm bnep ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ip6table_filter ip6_tables snd_hda_codec_hdmi arc4 snd_hda_codec_conexant iwldvm mac80211 snd_hda_intel snd_hda_codec snd_hwdep iwlwifi iTCO_wdt cfg80211 snd_seq sdhci_pci sdhci iTCO_vendor_support snd_seq_device btusb snd_pcm i2c_i801 hid_lenovo_tpkbd mmc_core lpc_ich mfd_core bluetooth thinkpad_acpi snd_page_alloc snd_timer snd e1000e rfkill ptp pps_core wmi mperf soundcore nfsd auth_rpcgss nfs_acl lockd sunrpc binfmt_misc uinput i915 i2c_algo_bit drm_kms_helper drm i2c_core video
CPU: 3 PID: 34 Comm: khubd Not tainted 3.11.0-0.rc7.git4.1.fc21.x86_64 #1
Hardware name: LENOVO 417152U/417152U, BIOS 8CET54WW (1.34 ) 07/11/2012
 0000000000000009 ffff880210c4b9a0 ffffffff817221a6 ffff880210c4b9e8
 ffff880210c4b9d8 ffffffff8107464d ffff88020ba1f9e0 ffffffff81c3ee60
 ffffffff81a33edc ffffffff82bc1c50 0000000000000002 ffff880210c4ba38
Call Trace:
 [<ffffffff817221a6>] dump_stack+0x54/0x74
 [<ffffffff8107464d>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff810746bc>] warn_slowpath_fmt+0x4c/0x50
 [<ffffffff81383113>] debug_print_object+0x83/0xa0
 [<ffffffff815b1000>] ? usbhid_open+0xf0/0xf0
 [<ffffffff8138426b>] debug_check_no_obj_freed+0x20b/0x250
 [<ffffffff8117e4e5>] free_pages_prepare+0x1b5/0x2b0
 [<ffffffff815b0b0f>] ? usbhid_disconnect+0x2f/0x50
 [<ffffffff8117f684>] __free_pages+0x34/0x80
 [<ffffffff8117f8b2>] __free_memcg_kmem_pages+0x22/0x50
 [<ffffffff811cfe10>] kfree+0x2a0/0x2f0
 [<ffffffff815b0b0f>] usbhid_disconnect+0x2f/0x50
 [<ffffffff81507427>] usb_unbind_interface+0x67/0x1d0
 [<ffffffff8148616f>] __device_release_driver+0x7f/0xf0
 [<ffffffff81486205>] device_release_driver+0x25/0x40
 [<ffffffff814859a8>] bus_remove_device+0x108/0x190
 [<ffffffff814822c2>] device_del+0x142/0x1e0
 [<ffffffff81504e30>] usb_disable_device+0xb0/0x270
 [<ffffffff814fae65>] usb_disconnect+0xb5/0x1d0
 [<ffffffff814fccea>] hub_thread+0x74a/0x1710
 [<ffffffff810a1370>] ? wake_up_atomic_t+0x30/0x30
 [<ffffffff814fc5a0>] ? hub_port_debounce+0x130/0x130
 [<ffffffff810a00ad>] kthread+0xed/0x100
 [<ffffffff8109ffc0>] ? insert_kthread_work+0x80/0x80
 [<ffffffff81734fec>] ret_from_fork+0x7c/0xb0
 [<ffffffff8109ffc0>] ? insert_kthread_work+0x80/0x80
Comment 1 Satish Balay 2013-09-03 12:56:23 EDT
Created attachment 793277 [details]
File: dmesg
Comment 2 Satish Balay 2013-09-03 18:31:06 EDT
The problem goes away if I rebuild kernel-3.11.0-2.fc21.src.rpm without HID-CVE-fixes.patch
Comment 3 Josh Boyer 2013-09-03 19:32:53 EDT
(In reply to Satish Balay from comment #2)
> The problem goes away if I rebuild kernel-3.11.0-2.fc21.src.rpm without
> HID-CVE-fixes.patch

Hm.  That's a patch from upstream to fix a large number of CVEs.  Apparently something isn't quite right with it.

Though to be clear, the trackpoint on the keyboard works fine as well, or you just don't see the ODEBUG messages?
Comment 4 Satish Balay 2013-09-03 19:42:07 EDT
(In reply to Josh Boyer from comment #3)

> Though to be clear, the trackpoint on the keyboard works fine as well, or
> you just don't see the ODEBUG messages?

with the HID-CVE-fixes.patch removed, the trackpoint on the external thinkpad keyboard works fine. [and there is no crash on removing and replugging in the usb cable for the keyboard]. This is the same behavior as rc7.git3
Comment 5 Benjamin Tissoires 2013-09-06 08:44:27 EDT
Thanks for the bug report and preliminary testing.
This bug is twofold:

- the CVE fix seems to be broken at some point
- the error path in the tpkbd_probe() function is wrong because it does not clean the input devices if tpkbd_probe_tp() fails (which was raised by the CVE fix).

To fix and test this, I would need to have access to the hid-recorder[1] traces of your devices (hidraw2 and 3 I would say), without the CVE fix, and with some events of each hidraw device.

[1] http://bentiss.github.io/hid-replay-docs/

PS: I _think_ your kernel will oops if you use hid-replay on the recordings you are about to do. Just make sure some events are recorded, and I'll break my own kernel if it does :)
Comment 6 Satish Balay 2013-09-06 12:27:31 EDT
Created attachment 794885 [details]
/dev/hidraw0

[root@asterix ~]# hid-recorder > mydevice1.hid
Available devices:
/dev/hidraw0:	Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint
/dev/hidraw1:	Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint
Select the device event number [0-1]: 0
Comment 7 Satish Balay 2013-09-06 12:39:18 EDT
Created attachment 794887 [details]
/dev/hidraw1

[root@asterix ~]# hid-recorder > mydevice2.hid
Available devices:
/dev/hidraw0:	Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint
/dev/hidraw1:	Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint
Select the device event number [0-1]: 1


Note: This is captured 3.11.0-2 rebuilt without HID-CVE-fixes.patch.

Since the keyboard/mouse is detected as 2 devices [and hid-recorder is asking me to select one of them] - I'm attaching 2 logs - one for each device.

I've attempted to create a few key strokes and a few mouse events for these logs.


Just to clarify - [as you suggest] there are perhaps multiple issues here.

The first is: - the mouse events are not visible to the kernel from this keyboard.

secondly: when I unplug the usb cable [with the rc7.git4 (debug) kernel - I get the stack trace reported on this bugzilla.

Thirdly: - if I use the nodebug 3.11.0-2 kernel - I get a freeze/pannic  when I reattach the usb cable.[which I couldn't capture in logs.]. Today I noticed that [when I'm in VT2 mode] - I had to do this a few times before the panic would trigger - and during one attempt I noticed a non-panic stack trace - below:


>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Sep  6 11:12:51 asterix kernel: [  109.764210] usb 1-1.2: new low-speed USB device number 6 using ehci-pci
Sep  6 11:12:51 asterix kernel: [  109.855342] usb 1-1.2: New USB device found, idVendor=17ef, idProduct=6009
Sep  6 11:12:51 asterix kernel: [  109.855354] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Sep  6 11:12:51 asterix kernel: [  109.855360] usb 1-1.2: Product: ThinkPad USB Keyboard with TrackPoint
Sep  6 11:12:51 asterix kernel: [  109.855364] usb 1-1.2: Manufacturer: Lite-On Technology Corp.
Sep  6 11:12:51 asterix kernel: [  109.860530] input: Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.0/input/input16
Sep  6 11:12:51 asterix kernel: [  109.860869] lenovo_tpkbd 0003:17EF:6009.0005: input,hidraw0: USB HID v1.10 Keyboard [Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint] on usb-0000:00:1a.0-1.2/input0
Sep  6 11:12:51 asterix mtp-probe: checking bus 1, device 6: "/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2"
Sep  6 11:12:51 asterix mtp-probe: bus: 1, device: 6 was not an MTP device
Sep  6 11:12:51 asterix kernel: [  109.870136] input: Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.1/input/input17
Sep  6 11:12:51 asterix kernel: [  109.870842] lenovo_tpkbd 0003:17EF:6009.0006: input,hiddev0,hidraw3: USB HID v1.10 Mouse [Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint] on usb-0000:00:1a.0-1.2/input1
Sep  6 11:12:51 asterix kernel: [  109.870845] lenovo_tpkbd 0003:17EF:6009.0006: missing HID_OUTPUT_REPORT 4
Sep  6 11:12:51 asterix kernel: [  109.875856] ------------[ cut here ]------------
Sep  6 11:12:51 asterix kernel: [  109.875872] WARNING: CPU: 0 PID: 9 at lib/list_debug.c:33 __list_add+0xac/0xc0()
Sep  6 11:12:51 asterix kernel: [  109.875875] list_add corruption. prev->next should be next (ffffffff81eb3978), but was           (null). (prev=ffff880206766898).
Sep  6 11:12:51 asterix kernel: [  109.875922] Modules linked in: joydev hid_lenovo_tpkbd ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle tun bridge stp llc rfcomm bnep ip6t_REJECT nf_conntrack_ipv4 nf_conntrack_ipv6 nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter xt_conntrack nf_conntrack ip6_tables arc4 tpm_tis tpm iwldvm mac80211 iTCO_wdt iTCO_vendor_support lpc_ich btusb mfd_core snd_hda_codec_hdmi snd_hda_codec_conexant i2c_i801 tpm_bios snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm bluetooth e1000e sdhci_pci ptp sdhci pps_core mmc_core iwlwifi cfg80211 thinkpad_acpi snd_page_alloc snd_timer snd mperf rfkill wmi soundcore nfsd uinput auth_rpcgss nfs_acl lockd binfmt_misc sunrpc i915 i2c_algo_bit drm_kms_helper drm i2c_core video
Sep  6 11:12:51 asterix kernel: [  109.875925] CPU: 0 PID: 9 Comm: rcu_sched Not tainted 3.11.0-2.fc21.x86_64 #1
Sep  6 11:12:51 asterix kernel: [  109.875928] Hardware name: LENOVO 417152U/417152U, BIOS 8CET54WW (1.34 ) 07/11/2012
Sep  6 11:12:51 asterix kernel: [  109.875931]  0000000000000009 ffff880212949c90 ffffffff81641d9b ffff880212949cd8
Sep  6 11:12:51 asterix kernel: [  109.875934]  ffff880212949cc8 ffffffff810670dd ffff880212949da0 ffffffff81eb3978
Sep  6 11:12:51 asterix kernel: [  109.875936]  ffff880206766898 ffffffff81eb3280 000000000000e088 ffff880212949d28
Sep  6 11:12:51 asterix kernel: [  109.875937] Call Trace:
Sep  6 11:12:51 asterix kernel: [  109.875943]  [<ffffffff81641d9b>] dump_stack+0x45/0x56
Sep  6 11:12:51 asterix kernel: [  109.875948]  [<ffffffff810670dd>] warn_slowpath_common+0x7d/0xa0
Sep  6 11:12:51 asterix kernel: [  109.875950]  [<ffffffff8106714c>] warn_slowpath_fmt+0x4c/0x50
Sep  6 11:12:51 asterix kernel: [  109.875953]  [<ffffffff8130eaec>] __list_add+0xac/0xc0
Sep  6 11:12:51 asterix kernel: [  109.875957]  [<ffffffff81072a08>] __internal_add_timer+0xc8/0x130
Sep  6 11:12:51 asterix kernel: [  109.875960]  [<ffffffff81072ce7>] internal_add_timer+0x17/0x40
Sep  6 11:12:51 asterix kernel: [  109.875964]  [<ffffffff81644a01>] schedule_timeout+0x161/0x2c0
Sep  6 11:12:51 asterix kernel: [  109.875967]  [<ffffffff81072a70>] ? __internal_add_timer+0x130/0x130
Sep  6 11:12:51 asterix kernel: [  109.875971]  [<ffffffff810890c6>] ? prepare_to_wait+0x56/0x90
Sep  6 11:12:51 asterix kernel: [  109.875975]  [<ffffffff810fe4df>] rcu_gp_kthread+0x2ff/0x5c0
Sep  6 11:12:51 asterix kernel: [  109.875978]  [<ffffffff81089380>] ? wake_up_atomic_t+0x30/0x30
Sep  6 11:12:51 asterix kernel: [  109.875980]  [<ffffffff810fe1e0>] ? rcu_gp_fqs+0x80/0x80
Sep  6 11:12:51 asterix kernel: [  109.875983]  [<ffffffff810885c0>] kthread+0xc0/0xd0
Sep  6 11:12:51 asterix kernel: [  109.875986]  [<ffffffff81088500>] ? insert_kthread_work+0x40/0x40
Sep  6 11:12:51 asterix kernel: [  109.875989]  [<ffffffff81650eac>] ret_from_fork+0x7c/0xb0
Sep  6 11:12:51 asterix kernel: [  109.875992]  [<ffffffff81088500>] ? insert_kthread_work+0x40/0x40
Sep  6 11:12:51 asterix kernel: [  109.875994] ---[ end trace b4937772633d3a49 ]---

<<<<<<<<<<<<<<


If the stack strace from the subsequent panic is useful - I can try reproducing so that the stack trace is visible on VT2 - and caputre with a camera and post.
Comment 8 Benjamin Tissoires 2013-09-11 13:08:42 EDT
Thanks for the logs.
I have prepared a repo for you to test:
https://github.com/bentiss/hid-lenovo-tpkbd

just clone, make and run:
$> sudo rmmod hid-lenovo-tpkbd ; sudo insmod ./hid_lenovo_tpkbd_backport.ko

Just for double checking, un-plug, re-plug the keyboard

If it does works, please check that lsmod shows only the backported version of hid-lenovo-tpkbd, and drop a message here. I'll send the modifications upstream.
Comment 9 Satish Balay 2013-09-11 15:13:00 EDT
(In reply to Benjamin Tissoires from comment #8)
> Thanks for the logs.
> I have prepared a repo for you to test:
> https://github.com/bentiss/hid-lenovo-tpkbd
> 
> just clone, make and run:
> $> sudo rmmod hid-lenovo-tpkbd ; sudo insmod ./hid_lenovo_tpkbd_backport.ko
> 
> Just for double checking, un-plug, re-plug the keyboard
> 
> If it does works, please check that lsmod shows only the backported version
> of hid-lenovo-tpkbd, and drop a message here. I'll send the modifications
> upstream.

I tried this with kernel-3.11.0-0.rc7.git0.2.fc21.x86_64 [debug + HID-CVE-fixes.patch]

The above gave me a kernel panic - so I did the following alternative.

1. reboot with the keyboard unhooked.
2. verified hid-lenovo-tpkbd is not loaded

[root@asterix ~]# lsmod |grep hid
[root@asterix ~]#

3. loaded the backport driver and verified that its loaded
[root@asterix hid-lenovo-tpkbd]# insmod  ./hid_lenovo_tpkbd_backport.ko 
[root@asterix hid-lenovo-tpkbd]# lsmod |grep hid
hid_lenovo_tpkbd_backport    13846  0 
[root@asterix hid-lenovo-tpkbd]#

4. Now plugged in the USB keyboard. Now I see the mouse works fine.

5. Now I disconnected and reconnected it a few times - and It worked fine!
[I had no tracebacks or panics]

thanks!
Comment 10 Benjamin Tissoires 2013-09-11 16:23:28 EDT
Thanks for the tests.

The patches have now been sent to the LKML. Hopefully, they will get pushed to Linus soon, and Josh may be able to do something for Fedora.

Josh, a "simple" fix for that can be to change the line 844 of the file HID-CVE-fixes.patch from
+	if (!hid_validate_report(hdev, HID_OUTPUT_REPORT, 4, 4, 1) ||
to
+	if (!hid_validate_report(hdev, HID_FEATURE_REPORT, 4, 4, 1) ||

And then, you can wait for dropping this when it goes to stable.
Comment 11 Josh Boyer 2013-09-11 16:37:29 EDT
OK, great.  I'll get this fixed up tomorrow.
Comment 12 Josh Boyer 2013-09-12 09:17:01 EDT
Satish, I've done a scratch build with v3 of the fixes from Benjamin included.  Could you please test this when it finished building and make sure the issue is resolved with this build?

http://koji.fedoraproject.org/koji/taskinfo?taskID=5927471
Comment 13 Josh Boyer 2013-09-12 11:12:46 EDT
The x86_64 build is done now:

http://koji.fedoraproject.org/koji/taskinfo?taskID=5927473
Comment 14 Satish Balay 2013-09-12 11:52:27 EDT
(In reply to Josh Boyer from comment #13)
> The x86_64 build is done now:
> 
> http://koji.fedoraproject.org/koji/taskinfo?taskID=5927473

I've installed and booted to kernel-3.11.0-300.1.fc20.x86_64 from above - and its not showing the problem.

The mouse on the usb keyboard does work with this kernel - and I've attempted to disconnect/reconnect a few times -without any stack traces.

thanks!
Comment 15 Josh Boyer 2013-09-12 13:11:06 EDT
Excellent, thank you for testing.  I've pushed that to the Fedora git repo and it should be in the next official build/update.
Comment 16 Fedora Update System 2013-09-14 18:32:08 EDT
kernel-3.11.1-300.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.11.1-300.fc20
Comment 17 Fedora Update System 2013-09-15 13:38:02 EDT
Package kernel-3.11.1-300.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.11.1-300.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-16794/kernel-3.11.1-300.fc20
then log in and leave karma (feedback).
Comment 18 Fedora Update System 2013-09-16 14:57:13 EDT
kernel-3.11.1-200.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/kernel-3.11.1-200.fc19
Comment 19 Fedora Update System 2013-09-16 14:59:44 EDT
kernel-3.10.12-100.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/kernel-3.10.12-100.fc18
Comment 20 Josh Boyer 2013-09-18 08:48:08 EDT
*** Bug 1009394 has been marked as a duplicate of this bug. ***
Comment 21 Ken Sugawara 2013-09-18 21:45:14 EDT
I'm running kernel-3.10.12-100.fc18 and it seems to work fine now. Thanks.
Comment 22 Fedora Update System 2013-09-18 22:03:51 EDT
kernel-3.11.1-200.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 23 Josh Boyer 2013-09-19 09:56:41 EDT
*** Bug 1009815 has been marked as a duplicate of this bug. ***
Comment 24 Fedora Update System 2013-09-22 19:59:52 EDT
kernel-3.10.12-100.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 25 Fedora Update System 2013-09-22 20:35:57 EDT
kernel-3.11.1-300.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.