Bug 1879442

Created attachment 1715051 [details]
dmesg of kernel 5.7.8-100.fc31

Created attachment 1715052 [details]
dmesg of kernel 5.6.19-200, the one that is OK

To be more specific about the dock. It is a:

Lenovo USB-C Dock
Model NO. DK1633
Type: 40A9

This is the product page of the exact model: https://support.lenovo.com/nl/en/solutions/acc100348

It took some effort but after a first bisect between v5.6..v5.7, which ended up finding an EFI boot bug that is later fixed that came down to disabling EFI early PCI DMA in order to have them boot.

I found the commit responsible for my issue with a second round of bisecting; range (bad) 50a5de895dbe5df947b3a695777db5b2c313e065 down to (good) 29d9f30d4ce6c7a38745a54a8cddface10013490. Finding commit 0f8839f5f323da04a800e6ced1136e4b1e1689a9 to be the first bad commit, containing:

commit 0f8839f5f323da04a800e6ced1136e4b1e1689a9
Author: Ville Syrjälä <ville.syrjala.com>
Date:   Thu Feb 13 16:04:12 2020 +0200

    drm/i915: Force state->modeset=true when distrust_bios_wm==true
..snip..

This commit is not present in the v5.6 stable tree, but is merged in v5.7 and onwards.
Linux 5.5.0-rc7-02468-ga4277aa398d7 is the last kernel of the master branch to work for me. The commit after it, which is the one mentioned above, resulting in Linux 5.5.0-rc7-02469-g0f8839f5f323 breaks dock display sync on boot.

I have now built Linux v5.8.9 with commit 0f8839f5f323 reverted: it works splendid again.

I am considering contacting the maintainer/developer responsible to see what we can do about this.
Perhaps later I can also generate a signed patch to revert this commit in case the developer agrees.
(Or would you guys rather perform the communication?)

I have added to the bug report:
- the commit message including the responsible code (output of "git log 0f8839f5f323da04a800e6ced1136e4b1e1689a9 -1 -p") 
- my bisect.log file

For now I can at least upgrade again knowing I can resolve the issue manually by building my own kernel.

Created attachment 1717473 [details]
Commit responsible for my issue

This is the output of "git log 0f8839f5f323da04a800e6ced1136e4b1e1689a9 -1 -p" showing the commit and code responsible for my issue.

Created attachment 1717474 [details]
bisect log

Log of linux git bisect, good||bad, starting with:

git bisect start
git bisect bad 50a5de895dbe5df947b3a695777db5b2c313e065
git bisect good 29d9f30d4ce6c7a38745a54a8cddface10013490

make olddefconfig
make
sudo make modules_install && sudo make install

.. test ..; rebooting my machine, watch the display work or not upon booting

and bisect good or bad depending upon the outcome

Reverting commit 0f8839f5f323da04a800e6ced1136e4b1e1689a9 works find with `git revert 0f8839f5f323da04` against the v5.8 stable tree.
Against the linux master branch, currently at v5.9-rc7 it does not revert cleanly.
I have looked into the conflict, modified the revert and devised a patch which fixes the regression for v5.9-rc7 on my system.

I have also created an email to sent to the maintainers of the i915 driver. Standing by in my drafts box. Probably going to send it off somewhere tomorrow.

Created attachment 1717874 [details]
patch for v5.9-rc7

Thank you Stefan for your instruction! I had same problem with my T590 with Fedora32 and pro docking station 40ah. I have downgraded kernel to 5.6.6 and now all is good! If you get response from maintainers of i915, please text it here (or send link).

I'm happy to hear you have a usable workaround now.
And you can built your own kernel with the mentioned commit reverted, that works fine as well (I run 5.9.0-rc7 with my patch at the moment).

I have been kind of swamped these past two weeks.
In the mean time I did indeed receive a reply from a i915 maintainer: https://lists.freedesktop.org/archives/intel-gfx/2020-October/249495.html
My "patch" (revert) is in there as well.

I have also opened an issue per his request on the freedesktop.org GitLab: https://gitlab.freedesktop.org/drm/intel/-/issues/2579

We'll see what happens.

This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

I have upgrade my machine to Fedora 33. The issue remains with kernel 5.9.13-200.fc33.x86_64.

The issue has been resolved in the DRM upstream, discussed at https://gitlab.freedesktop.org/drm/intel/-/issues/2579
Fixed code is now present in the DRM git repositories, available for testing at:
- Freedesktop drm-intel: https://cgit.freedesktop.org/drm-intel , branch drm-intel-next-queued
- Freedesktop drm-tip  : https://cgit.freedesktop.org/drm-tip   , branch drm-tip

Currently I run kernel 5.10.0-rc6-g62a57fc69781 built from drm-tip sources. That works well for me.

I guess it's a matter of time before this trickles down to Linux master.
Linux 5.10 has been released yesterday. I haven't had a look at it yet. Will go and check that out to test it and report back here.
I'll also keep an eye on Bodhi (https://bodhi.fedoraproject.org/updates/?packages=kernel) for a 5.10.0-0.fc34 to test that when it becomes available.

I am seeing similar issues with kernel-5.11.8-200.fc33.x86_64. The external monitor connected to the dock doesn't work and I get a flood of "i915 0000:00:02.0: [drm] *ERROR* Link Training Unsuccessful" messages on dmesg.

With kernel-5.8.14-300.fc33.x86_64, the external monitor works properly.

I'm also facing this issue with latest update: moving from kernel-5.8.15-301.fc33.x86_64 to newer kernel 5.11.8-200.fc33.x86_64 made the display though dock stop working properly. Same messages in dbus.
I could also try kernel 5.10.16.200.fc33.x86_64 which was still available in grub, and 5.10.16 also shows the issue. Comment #13 pinpoints the possible cause and solution. Any chance the patches from the reference branches can reach upstream or Fedora kernel soon? As this Lenovo laptop + Lenovo USB dock + external monitor is a pretty standard setting in the industry, this kernel update will very quickly affect a lot of people who will be disappointed, or nostalgic, to see a good old Linux display issue like in the mid 2000's ;)
Workaround is to reinstall and use kernel-5.8.15-301.fc33.x86_64 if you're willing to use the dock; or if you forgot to start the older kernel to plug the HDMI cable directly to the laptop.

Possible duplicate: bug 1943261

From the failure case where a flood of link training errors are reported, could some provide a dmesg log booting with drm.debug=0x10e?

I'm experiencing the same Problem with a Lenovo T480s Laptop and a Lenovo ThinkPad Thunderbolt 3 USB-C Dock.

With this kernel version/package "kernel-5.10.22-200.fc33.x86_64" there's no problem, but with those both kernels:
kernel-5.11.7-200.fc33.x86_64
kernel-5.11.10-200.fc33.x86_64

I'm getting this error messages when executing dmesg:
---
[  132.689201] input: Lenovo ThinkPad Thunderbolt 3 Dock USB Audio as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4.1/1-4.1.1/1-4.1.1.2/1-4.1.1.2:1.3/0003:17EF:3083.0013/input/input47
[  132.741523] hid-generic 0003:17EF:3083.0013: input,hidraw3: USB HID v1.11 Device [Lenovo ThinkPad Thunderbolt 3 Dock USB Audio] on usb-0000:00:14.0-4.1.1.2/input3
[  133.125003] [drm:drm_dp_mst_dpcd_read [drm_kms_helper]] *ERROR* mstb 000000004dc6b692 port 1: DPCD read on addr 0x4b0 for 1 bytes NAKed
[  133.169621] [drm:drm_dp_mst_dpcd_read [drm_kms_helper]] *ERROR* mstb 000000004dc6b692 port 2: DPCD read on addr 0x4b0 for 1 bytes NAKed
[  133.214549] [drm:drm_dp_mst_dpcd_read [drm_kms_helper]] *ERROR* mstb 000000004dc6b692 port 3: DPCD read on addr 0x4b0 for 1 bytes NAKed
[ 4724.962704] perf: interrupt took too long (2532 > 2500), lowering kernel.perf_event_max_sample_rate to 78000
---
My workaround was to remove the failing kernel(s) with:
$ sudo dnf remove kernel*5.11.[7,9]*
or pressing the CTRL key to select the previous kernel in the Grub menu list.

I'm hoping that there's a fix on the way in one of the next 5.11.x kernel versions.


Thanks a lot, best regards
Kurt

Created attachment 1767337 [details]
dmesg w/ drm.debug=0x10e on 5.11.10-200.fc33.x86_64

@imre.deak I attached the requested dmesg output. This was generated w/ an external monitor attached to the dock.
I don't see any errors when attached to the HDMI on the laptop.

@jstrunk , 

drm/i915/ilk-glk: Fix link training on links with LTTPRs

in drm-tip kernel should fix this issue. Could you let me know what is your platform?

I'm using a Lenovo T490s w/ Thunderbolt 3 Dock Gen 2

@jstrunk , thanks. The commit I pointed to should fix that issue.

@ehabkost the problem you reported sounds different, link training succeeds first, but then the link is lost later. Could you add a dmesg log capturing the failure and could you try if the problem can be also reproduces with drm-tip?

Imre, thank you for looking into this.

I've started a Fedora test-kernel build with the following cherry-picked on top of 5.11.10:

984982f3ef7b240cd24c2feb2762d81d9d8da3c2 ("drm/i915/ilk-glk: Fix link training on links with LTTPRs") (from drm-intel-next)
3b7bbb3619d2cc92f04ba10ad27d3b616aabf175 ("drm/i915/dp: Prevent setting the LTTPR LT mode if no LTTPRs are detected") (from 5.12)
264613b406eb0d74cd9ca582c717c5e2c5a975ea ("drm/i915: Disable LTTPR support when the DPCD rev < 1.4") (from drm-intel-next)

I've also added the last 2 because the last one is marked as:

Fixes: 7b2a4ab8b0ef ("drm/i915: Switch to LTTPR transparent mode link training")
Cc: <stable.org> # v5.11

And the middle / 5.12 patch is a dep to get the last one to apply cleanly.

Imre, please let me know if adding these last 2 patches somehow is a bad idea.


For everyone who is having issues with DP-MST / thunderbolt / Type-C docks with kernel 5.11, a test-kernel with these patches added is currently building here:

https://koji.fedoraproject.org/koji/taskinfo?taskID=64839432

This should be done building in about 2 hours, please give this kernel a try.

Here are some generic instructions for installing a kernel directly from koji (the Fedora buildsystem) :
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Note since this is a test kernel-build it is not signed, so you need to disable secure-boot to test this.

Imre, while building 5.12 + the 2 drm-intel-next fixes for testing locally I just noticed the following compiler messages:

  CC [M]  drivers/gpu/drm/i915/display/intel_dp_link_training.o
In function ‘intel_dp_check_mst_status’,
    inlined from ‘intel_dp_hpd_pulse’ at drivers/gpu/drm/i915/display/intel_dp.c:5902:8:
drivers/gpu/drm/i915/display/intel_dp.c:4555:22: warning: ‘drm_dp_channel_eq_ok’ reading 6 bytes from a region of size 4 [-Wstringop-overread]
 4555 |                     !drm_dp_channel_eq_ok(&esi[10], intel_dp->lane_count)) {
      |                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/i915/display/intel_dp.c: In function ‘intel_dp_hpd_pulse’:
drivers/gpu/drm/i915/display/intel_dp.c:4555:22: note: referencing argument 1 of type ‘const u8 *’ {aka ‘const unsigned char *’}
In file included from drivers/gpu/drm/i915/display/intel_dp.c:38:
./include/drm/drm_dp_helper.h:1459:6: note: in a call to function ‘drm_dp_channel_eq_ok’
 1459 | bool drm_dp_channel_eq_ok(const u8 link_status[DP_LINK_STATUS_SIZE],
      |      ^~~~~~~~~~~~~~~~~~~~

This might be worthwhile looking into...

*** Bug 1943261 has been marked as a duplicate of this bug. ***

@hdegoede , yes those are the fixes related to LTTPR functionality and pending for the 5.11 stable tree, thanks for building the test kernel with those.

The compiler warning is unrelated and doesn't cause an actual problem, but it's discussed at
https://lore.kernel.org/lkml/20210322160253.4032422-12-arnd@kernel.org/

The test kernel has finished building, so everyone who is seeing issues, please give it a go (see comment 24).


Imre, thank you for confirming that I've cherry-picked the right fixes. I'll reply to the compiler warning thing on the list.

The problem seems to be fixed for me with 5.11.10-200.bz1879442.fc33.x86_64

I also confirm that it fixed the problem.

```
$ uname -a
Linux thinkpad-t590 5.11.10-200.bz1879442.fc33.x86_64 #1 SMP Tue Mar 30 09:46:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ journalctl -k | grep '*ERROR* Link Training Unsuccessful'

$
```

Waiting for official kernel release.

+1 fixed on T490s too

Thank you all for testing the test-build. I've submitted a merge-req for getting the 3 backported patches added to the official Fedora-5.11 kernel builds: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/988

So this should be fixed in the next Fedora kernel build done after that merge-request has been merged.

FEDORA-2021-6b0f287b8b has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2021-6b0f287b8b

FEDORA-2021-2306e89112 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-2306e89112

FEDORA-2021-41fb54ae9f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-41fb54ae9f

FEDORA-2021-2306e89112 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-2306e89112`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-2306e89112

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2021-41fb54ae9f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-41fb54ae9f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-41fb54ae9f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2021-6b0f287b8b has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-6b0f287b8b`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-6b0f287b8b

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

I have the 5.11.11 kernel in Fedora 33 where this seems to be included and the Lenovo dock now became usable. However, when it is hotplugged in, sometimes the USB devices attached to it don't work - as in they start working, then stop. Error messages in dmesg include:

[34384.440009] xhci_hcd 0000:09:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 1 comp_code 1
[34384.440022] xhci_hcd 0000:09:00.0: Looking for event-dma 000000036855ea30 trb-start 000000036855ea40 trb-end 000000036855ea40 seg-start 000000036855e000 seg-end 000000036855eff0
[34384.497734] xhci_hcd 0000:09:00.0: WARN Event TRB for slot 9 ep 2 with no TDs queued?
[34386.678424] xhci_hcd 0000:09:00.0: WARN Event TRB for slot 9 ep 2 with no TDs queued?
[34386.993754] xhci_hcd 0000:09:00.0: WARN Event TRB for slot 9 ep 2 with no TDs queued?
[34387.092384] xhci_hcd 0000:09:00.0: WARN Event TRB for slot 9 ep 2 with no TDs queued?
[34387.173247] xhci_hcd 0000:09:00.0: WARN Event TRB for slot 9 ep 1 with no TDs queued?
[34387.173988] xhci_hcd 0000:09:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 1 comp_code 1
[34387.174001] xhci_hcd 0000:09:00.0: Looking for event-dma 00000003e7d10530 trb-start 00000003e7d10600 trb-end 00000003e7d10600 seg-start 00000003e7d10000 seg-end 00000003e7d10ff0
[34387.176010] xhci_hcd 0000:09:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 1 comp_code 1
[34387.176025] xhci_hcd 0000:09:00.0: Looking for event-dma 00000003e7d10540 trb-start 00000003e7d10600 trb-end 00000003e7d10600 seg-start 00000003e7d10000 seg-end 00000003e7d10ff0
[34387.176040] xhci_hcd 0000:09:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 1 comp_code 1
[34387.176046] xhci_hcd 0000:09:00.0: Looking for event-dma 00000003e7d10550 trb-start 00000003e7d10600 trb-end 00000003e7d10600 seg-start 00000003e7d10000 seg-end 00000003e7d10ff0
[34387.177357] xhci_hcd 0000:09:00.0: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 1 comp_code 1
(the last two lines repeat many times)

[34387.364015] xhci_hcd 0000:09:00.0: WARN Event TRB for slot 9 ep 2 with no TDs queued?
(also repeats many times)

[34393.921049] usb 3-2.1.1.2: 2:0: usb_set_interface failed (-110)
[34399.041153] usb 3-2.1.1.2: 1:0: usb_set_interface failed (-110)
[34408.001617] pcieport 0000:05:01.0: pciehp: Slot(1): Link Down
[34408.001642] pcieport 0000:08:04.0: can't change power state from D3cold to D0 (config space inaccessible)
[34408.002426] xhci_hcd 0000:09:00.0: remove, state 1
[34408.002448] usb usb4: USB disconnect, device number 1
[34408.002458] usb 4-2: USB disconnect, device number 2


Plugging it out then back in seems to have gotten USB devices to work, though the "transfer event" error messages still appear.

The "USB devices die after hotplugging" thing is happening again, but normally the "death" actually happens some minutes after hotplugging. Should I create a separate bugzilla entry?

I'm experiencing this issue on Fedora 35 kernel-5.14.11-300.fc35 when DP MST is not disabled (default).

Hardware:
* Framework laptop
* i7-1165G7
* Lenovo USB-C Dock (40as0090us)

Connecting the dock after booting with the default kernel + params freezes the system for a time, then prints the following in dmesg:

[  177.354779] i915 0000:00:02.0: [drm] *ERROR* mstb 0000000045b3ca1b port 1: DPCD read on addr 0x4b0 for 1 bytes NAKed
[  177.408984] i915 0000:00:02.0: [drm] *ERROR* mstb 0000000045b3ca1b port 3: DPCD read on addr 0x4b0 for 1 bytes NAKed
[  182.404681] i915 0000:00:02.0: [drm] *ERROR* failed to enable link training
[  183.072048] i915 0000:00:02.0: [drm] *ERROR* Link Training Unsuccessful

Sometimes the "DPCD read" lines are not present, and it is just the last two error lines.

Booting with i915.enable_dp_mst=0 flag does get one display working without errors, but I cannot use both DP outputs on the dock (to be expected as they use MST).

I downgraded to kernel-5.6.19-200.fc31.x86_64 like the original reporter, but there the display was not recognized at all, thus no error or display. Wanted to try the above 5.11.10-200.bz1879442.fc33.x86_64 test kernel, but the build expired and I couldn't seem to figure out how to build that patchset myself.

This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 33 changed to end-of-life (EOL) status on 2021-11-30. Fedora 33 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.