Bug 2155639

Summary: mesa-22.3.1 breaks totem video on AMD gpus
Product: [Fedora] Fedora Reporter: louisgtwo
Component: mesaAssignee: Adam Jackson <ajax>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 37CC: airlied, ajax, andreas27x, bskeggs, CorruptComputer, csoriano, dominik, drusek, grumpey0, igor.raits, jglisse, j, lyude, mail, mail, mcatanza, ndegraef, rafaellos83, rhughes, rlledo, rstrode, tpopela, tstellar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mesa-22.3.3-3.fc37 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-27 08:56:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description louisgtwo 2022-12-21 17:13:02 UTC
After upgrading f37 to mesa-22.3.1 I can no longer view videos from totem, I can still hear audio. Downgrading to mesa-22.2.3 restored video playback. I am using rpmfusion h264 codes but even the free codecs I get no video.

Comment 1 Andreas Strahm 2022-12-21 17:43:08 UTC
mesa 22.3.1-1.fc37 also breaks the Chrome hardware video decoding that was working with mesa 22.2.3-1. 

$ uname -r
6.0.13-300.fc37.x86_64

# dmesg | grep i915
[    1.643071] i915 0000:00:02.0: vgaarb: deactivate vga console
[    1.643136] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    1.644022] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    1.645043] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/tgl_dmc_ver2_12.bin (v2.12)
[    1.658626] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    1.675902] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 1
[    1.683950] fbcon: i915drmfb (fb0) is primary device
[    1.683954] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[    3.529906] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[    3.530840] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[    3.658739] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[  937.568445] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  938.377412] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  939.308391] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  943.833577] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  974.988176] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  975.105175] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  975.479180] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  975.989100] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  976.143102] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  976.320137] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play
[  981.135311] i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play

$ lspci
0000:00:00.0 Host bridge: Intel Corporation 11th Gen Core Processor Host Bridge/DRAM Registers (rev 01)
0000:00:02.0 VGA compatible controller: Intel Corporation TigerLake-LP GT2 [Iris Xe Graphics] (rev 01)
0000:00:04.0 Signal processing controller: Intel Corporation TigerLake-LP Dynamic Tuning Processor Participant (rev 01)
0000:00:08.0 System peripheral: Intel Corporation GNA Scoring Accelerator module (rev 01)
0000:00:0d.0 USB controller: Intel Corporation Tiger Lake-LP Thunderbolt 4 USB Controller (rev 01)
0000:00:0e.0 RAID bus controller: Intel Corporation Volume Management Device NVMe RAID Controller
0000:00:14.0 USB controller: Intel Corporation Tiger Lake-LP USB 3.2 Gen 2x1 xHCI Host Controller (rev 20)
0000:00:14.2 RAM memory: Intel Corporation Tiger Lake-LP Shared SRAM (rev 20)
0000:00:14.3 Network controller: Intel Corporation Wi-Fi 6 AX201 (rev 20)
0000:00:15.0 Serial bus controller: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #0 (rev 20)
0000:00:15.1 Serial bus controller: Intel Corporation Tiger Lake-LP Serial IO I2C Controller #1 (rev 20)
0000:00:16.0 Communication controller: Intel Corporation Tiger Lake-LP Management Engine Interface (rev 20)
0000:00:17.0 System peripheral: Intel Corporation RST VMD Managed Controller
0000:00:1d.0 PCI bridge: Intel Corporation Device a0b1 (rev 20)
0000:00:1e.0 Communication controller: Intel Corporation Tiger Lake-LP Serial IO UART Controller #0 (rev 20)
0000:00:1e.3 Serial bus controller: Intel Corporation Tiger Lake-LP Serial IO SPI Controller #1 (rev 20)
0000:00:1f.0 ISA bridge: Intel Corporation Tiger Lake-LP LPC Controller (rev 20)
0000:00:1f.3 Multimedia audio controller: Intel Corporation Tiger Lake-LP Smart Sound Technology Audio Controller (rev 20)
0000:00:1f.4 SMBus: Intel Corporation Tiger Lake-LP SMBus Controller (rev 20)
0000:00:1f.5 Serial bus controller: Intel Corporation Tiger Lake-LP SPI Controller (rev 20)
0000:01:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS522A PCI Express Card Reader (rev 01)
10000:e0:17.0 SATA controller: Intel Corporation Tiger Lake-LP SATA Controller (rev 20)
10000:e0:1c.0 System peripheral: Intel Corporation RST VMD Managed Controller
10000:e0:1c.4 PCI bridge: Intel Corporation Device a0bc (rev 20)
10000:e1:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980

Comment 2 Andreas Strahm 2022-12-21 18:17:54 UTC
Update to my previous comment: 
It seems that the not working Chrome hardware video decoding is not related to mesa 22.3.1-1.fc37.
I have verified this on another system without this particular update. 
I still don't know which update was responsible for this.

Comment 3 Andreas Strahm 2022-12-22 14:33:11 UTC
As I've just learned, hardware decoded video playback is probably disabled to legal reasons and not expected to return at least in Fedora.
Luckily the software decoding seems to be improved so the CPU fan does no longer run as it used to do.
 
The annoying "i915 0000:00:02.0: [drm] *ERROR* arb session failed to go in play" message is still there.
See the following "bug" for reference: https://gitlab.freedesktop.org/drm/intel/-/issues/7706

But it can be suppressed by blacklisting the mei_pxp kernel module. Kernel parameter: module_blacklist=mei_pxp

Comment 4 Daniel Rusek 2022-12-23 13:28:35 UTC
Same issue on my Fedora 37 system with AMD RX 6600 XT.

Comment 5 louisgtwo 2022-12-23 15:05:59 UTC
How do you have totem use software software decoding? Shouldn't it be automatic?

Comment 6 louisgtwo 2022-12-24 01:12:23 UTC
This issue happens on my AMD vega system. My intel system works fine.

Comment 7 grumpey0 2022-12-24 17:00:50 UTC
Issues listed on ask.fedoraproject.org: 
https://ask.fedoraproject.org/t/f37-no-video-in-gnome-videos-totem/30137
https://ask.fedoraproject.org/t/totem-video-player-half-black-screen-on-dvd/30063

I observe the same issue with: 
mesa-dri-drivers: 22.3.1.1

On Wayland I observe no video and just sound. 
On X11 I observe the half black screen and half played video. 

sudo dnf downgrade mesa-dri-drivers (22.2.2.1), allowed me to play video in totem. 

Thanks

Comment 8 Rafal 2022-12-24 22:53:15 UTC
I have the same error. F37 GNOME AMD RX480 i5-11400.

Comment 9 grumpey0 2022-12-25 15:17:48 UTC
The upstream bug looks to be, https://gitlab.freedesktop.org/mesa/mesa/-/issues/7948

You can run from a command line with, env mesa_glthread=false totem to work around.

Comment 10 louisgtwo 2022-12-27 21:54:44 UTC
So should this bug be against gstreamer1?

Comment 11 Rubén 2023-01-01 17:09:22 UTC
Same issue (Radeon RX 580 8 GB). Downgrading to 22.2.2.1 solved it.

Comment 12 louisgtwo 2023-01-02 22:48:26 UTC
So is this a mesa, gstreamer1 or totem bug???

Comment 13 Michel Dänzer 2023-01-03 11:23:08 UTC
Linking the corresponding upstream Mesa issue.

Comment 14 Rubén 2023-01-03 16:22:33 UTC
mesa-dri-drivers-22.3.2 does NOT solve the issue.

Comment 15 louisgtwo 2023-01-03 17:16:39 UTC
I think the core problem is that gstreamer gtkglsink element is not thread safe. Since mesa 22.3 defaults to threaded opengl it uncovered this bug. for now I running totem with mesa_glthread=false. Either mesa needs to blacklist totem and any other gstreamer video app from running with treads or gstreamer needs to fix gtkglsink.

Comment 16 Michel Dänzer 2023-01-04 09:56:26 UTC
(In reply to louisgtwo from comment #15)
> I think the core problem is that gstreamer gtkglsink element is not thread
> safe. Since mesa 22.3 defaults to threaded opengl it uncovered this bug.

There is no such direct connection. glthread is a Mesa internal helper thread, it's supposed to be transparent to the thread(s) calling OpenGL APIs. If gtkglsink was "not thread safe" WRT OpenGL calls (the only thing that comes to mind offhand would be trying to make the same context current for multiple threads), that should fail the same way with or without glthread.

It is possible though that gtkglsink uses the OpenGL API incorrectly in some other way, which happens to cause an issue only with glthread. But the primary suspect is Mesa itself, per the linked upstream issue.

Comment 17 Michel Dänzer 2023-01-12 11:57:44 UTC
*** Bug 2157130 has been marked as a duplicate of this bug. ***

Comment 18 Daniel Rusek 2023-01-19 11:44:09 UTC
It seems to be a GTK issue:
https://gitlab.gnome.org/GNOME/gtk/-/issues/5517

Comment 19 Daniel Rusek 2023-01-21 22:58:45 UTC
(In reply to grumpey0 from comment #9)
> The upstream bug looks to be,
> https://gitlab.freedesktop.org/mesa/mesa/-/issues/7948
> 
> You can run from a command line with, env mesa_glthread=false totem to work
> around.

To use this workaround without the need to open Totem from a terminal, add this to your ~/.drirc file (or create one with this content if it does not exist):

<application name="totem" executable="totem">
   <option name="mesa_glthread" value="false" />
</device>

Comment 20 Daniel Rusek 2023-01-21 23:02:43 UTC
> <application name="totem" executable="totem">
>    <option name="mesa_glthread" value="false" />
> </device>

Sorry, small correction of the last line:

<application name="totem" executable="totem">
   <option name="mesa_glthread" value="false" />
</application>

Comment 21 Michael Catanzaro 2023-01-23 15:41:52 UTC
Since this is an emergency, we should have a downstream workaround in place ASAP. Can we simply revert the upstream changes for now?

Comment 22 Daniel Rusek 2023-01-24 12:04:35 UTC
(In reply to Michael Catanzaro from comment #21)
> Since this is an emergency, we should have a downstream workaround in place
> ASAP. Can we simply revert the upstream changes for now?

Probably the easiest hotfix would be to add a downstream patch that disables glthread for Totem in the official application workaround file that Mesa has: https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/util/00-mesa-defaults.conf

Just adding "<option name="mesa_glthread" value="false" />" after this line should be sufficient to workaround the issue: https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/util/00-mesa-defaults.conf#L765

Comment 23 Michael Catanzaro 2023-01-24 16:37:04 UTC
That is not sufficient because totem is not the only affected application here. At least sushi is also affected, and presumably more.

Comment 24 Fedora Update System 2023-01-25 15:27:05 UTC
FEDORA-2023-a9179f018f has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a9179f018f

Comment 25 Fedora Update System 2023-01-26 02:26:12 UTC
FEDORA-2023-a9179f018f has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-a9179f018f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-a9179f018f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 26 Daniel Rusek 2023-01-26 12:14:17 UTC
The update does not seem to help on my system (with AMD RX 6600 XT) - there is still only a black screen in Totem.

Comment 27 Daniel Rusek 2023-01-26 21:24:09 UTC
I apologize for my previous comment. It turns out that I also had a Flatpak build of Totem installed on my system and Nautilus for some reason decided to change mime types for all media files to prefer this version. I have retested it using a proper (Fedora) Totem version and can confirm that everything works fine now.

Comment 28 Fedora Update System 2023-01-27 08:56:04 UTC
FEDORA-2023-a9179f018f has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 29 Rubén 2023-01-28 13:27:51 UTC
mesa-dri-drivers-22.3.3 solves the issue

Thank you!

Comment 30 Dominik 'Rathann' Mierzejewski 2023-01-30 11:59:19 UTC
It looks like the fix in gtk3 will be included in 3.24.37:
https://gitlab.gnome.org/GNOME/gtk/-/commit/9811485990b400cd60f7db2c5466c2e813099cfe

One that's released, mesa_glthread can be re-enabled.

Comment 31 Michel Dänzer 2023-01-31 09:58:42 UTC
(In reply to Dominik 'Rathann' Mierzejewski from comment #30)
> It looks like the fix in gtk3 will be included in 3.24.37:
> https://gitlab.gnome.org/GNOME/gtk/-/commit/
> 9811485990b400cd60f7db2c5466c2e813099cfe
> 
> One that's released, mesa_glthread can be re-enabled.

Doesn't seem enough unfortunately: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7948#note_1746469

Comment 32 Dominik 'Rathann' Mierzejewski 2023-02-01 14:49:13 UTC
(In reply to Michel Dänzer from comment #31)
> (In reply to Dominik 'Rathann' Mierzejewski from comment #30)
> > It looks like the fix in gtk3 will be included in 3.24.37:
> > https://gitlab.gnome.org/GNOME/gtk/-/commit/
> > 9811485990b400cd60f7db2c5466c2e813099cfe
> > 
> > One that's released, mesa_glthread can be re-enabled.
> 
> Doesn't seem enough unfortunately:
> https://gitlab.freedesktop.org/mesa/mesa/-/issues/7948#note_1746469

Correct, but there's a mesa fix that works now:
https://gitlab.freedesktop.org/mesa/mesa/-/issues/7948#note_1749156
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21039

Comment 33 Michel Dänzer 2023-02-02 09:03:05 UTC
(In reply to Dominik 'Rathann' Mierzejewski from comment #32)
> Correct, but there's a mesa fix that works now:
> https://gitlab.freedesktop.org/mesa/mesa/-/issues/7948#note_1749156
> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21039

Right, the plan for upstream stable branches is to disable glthread by default again (in general, not just for specific apps) though.