Bug 1653841

Summary: input layer misbehaves when having more than one logical keyboard device
Product: [Fedora] Fedora Reporter: Juha Nikkanen <nikkej>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: airlied, bskeggs, ewk, hdegoede, ichavero, itamar, jarodwilson, jglisse, john.j5live, jonathan, josef, kernel-maint, linville, mchehab, mjg59, steved, y9t7sypezp
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-27 18:09:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
test script to automate collection of various logs
none
dmesg after boot
none
ACPI dump
none
DMI dump
none
dmesg excerpt of single test run
none
evtest output of even node2
none
evtest output of even node7
none
hid-recorder output of hid raw node0
none
hid-recorder output of hid raw node3
none
lsusb -v output none

Description Juha Nikkanen 2018-11-27 18:01:33 UTC
Created attachment 1509001 [details]
test script to automate collection of various logs

Description of problem:

having a keyboard which presents itself as a multiple logical keyboard or having two or more physical keyboards, it is possible to get input layer in to state where random key scan code starts repeat sporadically ad infinitum. 

Version-Release number of selected component (if applicable):

4.19.2
4.18.18
4.18.17

How reproducible:

Have two USB HID keyboards, pressing a key like caps lock or num lock
that causes actual data to be sent from the system to the input device
driver like e.g. EV_LED, random key code starts repeat. Usually this
key scan code is 185 which seems to respond KEY_F15 which is non
physically existent. Sometimes it is KEY_A and sometimes it comes
(repeated a's) through to cli making terminal useless. Repeat interval
seems match with about 330msec. Have tested this without X on multi
user non graphical runlevel and wmi modules black listed without any
effect. Problem persists.

Comment 1 Juha Nikkanen 2018-11-27 18:03:07 UTC
Created attachment 1509003 [details]
dmesg after boot

Comment 2 Juha Nikkanen 2018-11-27 18:04:10 UTC
Created attachment 1509004 [details]
ACPI dump

Comment 3 Juha Nikkanen 2018-11-27 18:05:00 UTC
Created attachment 1509005 [details]
DMI dump

Comment 4 Juha Nikkanen 2018-11-27 18:06:25 UTC
Created attachment 1509007 [details]
dmesg excerpt of single test run

Comment 5 Juha Nikkanen 2018-11-27 18:07:48 UTC
Created attachment 1509008 [details]
evtest output of even node2

Comment 6 Juha Nikkanen 2018-11-27 18:09:02 UTC
Created attachment 1509009 [details]
evtest output of even node7

Comment 7 Juha Nikkanen 2018-11-27 18:10:16 UTC
Created attachment 1509010 [details]
hid-recorder output of hid raw node0

Comment 8 Juha Nikkanen 2018-11-27 18:12:40 UTC
Created attachment 1509011 [details]
hid-recorder output of hid raw node3

Comment 9 Steve 2018-11-28 06:13:40 UTC
Developers might want to see the output from

# lsusb -v

for the configuration you are testing.

The "dmesg" output shows a "Corsair Link TM USB Dongle". Can you reproduce the problem without that connected?

I looked at the ROCCAT web site, but couldn't find a datasheet. However, this web page has some "Techspecs":

https://en.roccat.org/Keyboards/Vulcan

ROCCAT only claims compatibility with: "Windows® 10, Windows® 7, Windows® 8".

Have you tried to reproduce the problem with Windows? Or asked ROCCAT about Linux compatibility?

Comment 10 Steve 2018-11-28 08:35:02 UTC
(In reply to Juha Nikkanen from comment #0)
...
> ... it is possible to get input layer in
> to state where random key scan code starts repeat sporadically ad infinitum. 
...

Why do you believe it is the "input layer" and not the keyboard that is generating those events?

I suggest contacting ROCCAT for tech support:
https://en.roccat.org/Support/Contact

Comment 11 Juha Nikkanen 2018-11-28 10:27:04 UTC
Created attachment 1509429 [details]
lsusb -v output

Comment 12 Juha Nikkanen 2018-11-28 10:56:57 UTC
Steve, I'll drop the Corsair link, re-test and update results. What comes to your  doubt about "input layer", well, how would you see it when every test case I  have run so far does not indicate any extra USB traffic (i.e. any extra key press or release) and still one input device starts run wildly 'ghost repeat? To my (still developing) understanding of this issue, I have found logical match for every physical key press & release, from both USB bus and raw HID events. What I may have miss, I'm keen to learn.

I have triggered this same issue from  another physical keyboard. That time I had Roccat and Logitech K800 connected and running test from K800, event node 7 did start ghost repeat. Maybe I could disconnect Roccat, pair Apple wireless and use that simultaneously with K800. Updating results also with this combo.

Roccat peripheral is business-as-usual. If we users / developers were choosing our peripherals by "Linux compatible" tag were our options really rare. Nevertheless, I do contact Roccat technical support and start gently push them to provide at least firmware binaries and simple specs or tool that safely applies those fw blobs onto keyboard. Just now there does not exist support of any kind. Not reverse engineered or official.

Comment 13 Steve 2018-11-28 13:52:26 UTC
(In reply to Juha Nikkanen from comment #11)
> Created attachment 1509429 [details]
> lsusb -v output

Thanks. IIUC, the ROCCAT keyboard is connected to a USB 1.1 hub:

Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Device Descriptor:
...
  bcdUSB               1.10
...

Bus 005 Device 002: ID 1e7d:3098 ROCCAT 
Device Descriptor:
...
  bcdUSB               2.00
...

Comment 14 Steve 2018-11-28 14:34:42 UTC
According to the ASUS SABERTOOTH 990FX user's manual, there is a BIOS setting that controls the speed of certain USB controllers.

3.5.4 USB Configuration
SB USB Configuration (page 3-20)
https://www.asus.com/Motherboards/SABERTOOTH_990FX/HelpDesk_Manual/

Comment 15 Juha Nikkanen 2018-11-28 16:22:40 UTC
Yes, you understood correctly, Roccat were connected to OHCI hub. I'll check those BIOS settings. Long time ago there were a lot of USB compatibility problems with this particular mobo and then I baked in a configuration that worked most. Honestly I don't remember what had to be as a values for EHCI hand-off and legacy OHCI but I go and check. Thanks for the tip.

I connected Roccat on to external USB3.0 hub and it still wants to be 12M only:
/:  Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
    |__ Port 2: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 3: Dev 7, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 4: Dev 8, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 9, If 0, Class=Human Interface Device, Driver=usbhid, 12M
            |__ Port 4: Dev 9, If 1, Class=Human Interface Device, Driver=usbhid, 12M
            |__ Port 4: Dev 9, If 2, Class=Human Interface Device, Driver=usbhid, 12M
            |__ Port 4: Dev 9, If 3, Class=Human Interface Device, Driver=usbhid, 12M

Now I have checked with two other keyboards without Roccat and I can't trigger this problem with them. Those were that Apple wireless and Logitech K400. Only Roccat alone or with K400 or K800 triggers.

From those keyboards, Roccat is only that spits out at some point while pressing caps or num lock this lengthy HID event from one of it's associated hidraw nodes (payload is 64 bytes, URB total size is 128 bytes):
E: 0.000001 64 03 00 ff 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 6a


And at the moment of receiving this, event node for the secondary logical kbd says:
Event: time 1543421204.970903, type 4 (EV_MSC), code 4 (MSC_SCAN), value 700ff
Event: time 1543421204.970903, type 1 (EV_KEY), code 240 (KEY_UNKNOWN), value 1
Event: time 1543421204.970903, type 4 (EV_MSC), code 4 (MSC_SCAN), value 70004
Event: time 1543421204.970903, type 1 (EV_KEY), code 30 (KEY_A), value 1
Event: time 1543421204.970903, type 4 (EV_MSC), code 4 (MSC_SCAN), value 7006a
Event: time 1543421204.970903, type 1 (EV_KEY), code 185 (KEY_F15), value 1
Event: time 1543421204.970903, -------------- SYN_REPORT ------------
AEvent: time 1543421205.227414, type 1 (EV_KEY), code 185 (KEY_F15), value 2
Event: time 1543421205.227414, -------------- SYN_REPORT ------------
...
See how those misc scan values correlate with event shown above. I start to believe thís is two fold now. If that lengthy event is legitimate HID 1.11 event, will it overflow something?

Comment 16 Steve 2018-11-28 16:43:18 UTC
(In reply to Juha Nikkanen from comment #15)
...
> I connected Roccat on to external USB3.0 hub and it still wants to be 12M
> only:
...

According to the ASUS manual, the mobo has two USB 3.0 ports on the back. What happens when connect the ROCCAT keyboard to one of them?

Comment 17 Juha Nikkanen 2018-12-01 17:35:40 UTC
Steve, I'm completely done messing around with mobo USB settings. This does not lead me to anything.

> According to the ASUS SABERTOOTH 990FX user's manual, there is a BIOS
> setting that controls the speed of certain USB controllers.
> 
> 3.5.4 USB Configuration
> SB USB Configuration (page 3-20)
> https://www.asus.com/Motherboards/SABERTOOTH_990FX/HelpDesk_Manual/

Disabling OHCI support disables access to BIOS and grub menu with ANY USB keyboard. Roccat kbd then funtions only on external USB3.0 hub and problem persists. Also, lsusb tells speed is 12Mbit/s.

> According to the ASUS manual, the mobo has two USB 3.0 ports on the back.
> What happens when connect the ROCCAT keyboard to one of them?
 
No effect, speed is 12Mbit/s.

Not to taken offensive or in any other negative attitude but just for my own motivation to dig this deeper and further I want to say that testing Roccat on FreeBSD 11.2, keyboard functions flawlessly and does not require any quirks or driver for that

Comment 18 Juha Nikkanen 2018-12-15 15:44:49 UTC
Now I'm quite sure I found at least one anomaly. USB HID specification (hid1_11.pdf) reads on page 10 chapter 4.4 Interfaces:
"The Interrupt Out pipe is optional. If a device declares an Interrupt Out endpoint then Output reports are transmitted by the host to the device through the Interrupt Out endpoint. If no Interrupt Out endpoint is declared then Output reports are transmitted to a device through the Control endpoint, using Set_Report(Output) requests."

"Note Endpoint 0 is a Control pipe always present in USB devices. Therefore, only the Interrupt In pipe is described for the Interface descriptor using an Endpoint descriptor. In fact, several Interface descriptors may share Endpoint 0. An Interrupt Out pipe is optional and requires an additional Endpoint descriptor if declared."

And on page 18 chapter 5.6 Reports:
"Note Only Input reports are sent via the Interrupt In pipe. Feature and Output reports must be initiated by the host via the Control pipe or an optional Interrupt Out pipe."

LED state changes are such output reports that are sent to either control0 OR optional interrupt OUT. Vulcan 120 specifically implements dedicated interrupt OUT endpoint. However, as can be seen from tshark capture below (num lock LED state change initiated from a connected PS/2 keyboard, thus only LED reports are seen and captured on USB), LED state change is sent to BOTH endpoints:
    1 17:14:39.306606         host → 8.2.0        USB 65 URB_CONTROL out 
    2 17:14:39.306618         host → 8.2.4        USB 128 URB_INTERRUPT out 
    3 17:14:39.306688        8.2.0 → host         USB 64 URB_CONTROL out 
    4 17:14:39.307221        8.2.4 → host         USB 64 URB_INTERRUPT out 

This has to be wrong. HID driver shall send LED report only to endpoint 4 OUT if I have understood the HID spec correctly.

Comment 19 Justin M. Forbes 2019-01-29 16:25:40 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 28 kernel bugs.

Fedora 28 has now been rebased to 4.20.5-100.fc28.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 29, and are still experiencing this issue, please change the version to Fedora 29.

If you experience different issues, please open a new bug report for those.

Comment 20 Juha Nikkanen 2019-02-18 09:34:34 UTC
Upgraded to Fedora 29, and with kernel 4.20.8-200.fc29.x86_64, still the same issue. Output reports are sent to both endpoints:

   51 11:18:29.109659         host → 4.3.0        USB 65 URB_CONTROL out
   52 11:18:29.109732         host → 4.3.4        USB 128 URB_INTERRUPT out
   53 11:18:29.111437        4.3.0 → host         USB 64 URB_CONTROL out
   54 11:18:29.111453        4.3.4 → host         USB 64 URB_INTERRUPT out

Comment 21 Justin M. Forbes 2019-08-20 17:42:56 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs.

Fedora 29 has now been rebased to 5.2.9-100.fc29.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30.

If you experience different issues, please open a new bug report for those.

Comment 22 Juha Nikkanen 2019-09-04 11:46:53 UTC
I've tested this using kernel 5.2.11-100.fc29.x86_64 on f29 and issue persists. I'll get back as soon as I can since I'm still using some time to investigate this further.

Comment 23 Ben Cotton 2019-10-31 18:52:41 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 24 Juha Nikkanen 2019-11-15 16:44:41 UTC
I finally were able to upgrade my keyboard firmware to v1.33 without native windows installation and now situation has got much better. Roccat keyboard does not send that 128 byte not-standard HID message on the USB bus anymore and no ghost repeats are seen after host sending LED status reports.

The only thing which remains is the fact that input subsystem still sends LED reports to both endpoints, interrupt out and control out which is against HID spec. I think this was the main difference when compared Linux USB behavior against e.g. FreeBSD and Windows virtual guest as those two did sent output reports only to interrupt out.

Comment 25 Ben Cotton 2019-11-27 18:09:30 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.