Bug 2193335 - Mesa/LLVM/etc updates break Chrome/Chromium rendering
Summary: Mesa/LLVM/etc updates break Chrome/Chromium rendering
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: mesa
Version: 39
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
Assignee: Adam Jackson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://discussion.fedoraproject.org...
: 2208152 2238198 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-05-05 09:36 UTC by Yanko Kaneti
Modified: 2023-12-22 19:02 UTC (History)
56 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad.net ubuntu/+source/mesa/+bug/2020604 0 None None None 2023-06-01 07:49:11 UTC
chromium.org chromium 1442633 0 None None None 2023-05-29 07:06:33 UTC

Description Yanko Kaneti 2023-05-05 09:36:01 UTC
With mesa-23.0.3-4.fc39 Chrome starts but doesn't render most of the page other than color rectangles.
On the console there a multitude of errors like:
ERROR:shared_context_state.cc(77)] Skia shader compilation error
(followed by a dump of the shader itself)

Reverting back to mesa-23.0.3-3.fc39 gets Chrome back to working.

The GPU is  Arc A380 (ASRock card)

Comment 1 john getsoian 2023-05-05 17:23:09 UTC
This probably the same report even though the version numbers don't line up:

mesa-23.0.3-3.fc38 breaks Google chrome (in this case on KDE desktop). Verified on:
Lenovo X1-Yoga i7-1265U - GPU-Iris graphics
Lenovo X1-Carbon i7-8650U - GPU-Intel graphics
Asus Z690 I7-12600K, GPU-Radeon RX570

Remedy - back out Mesa driver to: 23.0.3-1.fc38

First screen paint looks OK, regions of Chrome window quickly begin to fall apart with each successive repaint.

Comment 2 tamas 2023-05-05 17:54:05 UTC
Can confirm problem on Fedora 38 w/ google-chrome-stable-113.0.5672.63-1.x86_64

Tuxedo InfinityBook Pro 14gen7 i7-12700H - GPU Intel XE (96EUs)
    Vendor: Intel (0x8086)
    Device: Mesa Intel(R) Graphics (ADL GT2) (0x46a6)
    Version: 23.0.3

Comment 3 Steven Stern 2023-05-05 20:02:13 UTC
Confirmed

[root@sds-desk log]# dnf downgrade mesa-libGL
Last metadata expiration check: 2:26:18 ago on Fri 05 May 2023 12:30:50 PM CDT.
Dependencies resolved.
========================================================================================================================================================================================
 Package                                           Architecture                            Version                                        Repository                               Size
========================================================================================================================================================================================
Downgrading:
 mesa-dri-drivers                                  x86_64                                  23.0.1-2.fc38                                  fedora                                   18 M
 mesa-filesystem                                   x86_64                                  23.0.1-2.fc38                                  fedora                                   18 k
 mesa-libEGL                                       x86_64                                  23.0.1-2.fc38                                  fedora                                  130 k
 mesa-libGL                                        x86_64                                  23.0.1-2.fc38                                  fedora                                  173 k
 mesa-libglapi                                     x86_64                                  23.0.1-2.fc38                                  fedora                                   54 k
 mesa-va-drivers                                   x86_64                                  23.0.1-2.fc38                                  fedora                                  3.4 M

Comment 4 Anthony J Starks 2023-05-05 20:19:58 UTC
Confirmed on Fedora 38, mesa 23.0.3-3.fc38, Chrome 113.0.5672.63

X1 Carbon 5th Gen  (Intel Skylake GT2 [HD Graphics 520] missing items on the page)
Minisforum UM690 (AMD Ryzen 9 6900HX) (blank pages, other programs leave screen artifacts in the location of the Chrome window)

Comment 5 Mário Lopes 2023-05-05 20:22:18 UTC
Can confirm problem on Fedora 38 w/ google-chrome-stable-113.0.5672.63-1.x86_64

OS: Fedora Linux 38 (KDE Plasma) x86_64 
Host: UM773 Lite Version 1.0 
Kernel: 6.2.14-300.fc38.x86_64 
CPU: AMD Ryzen 7 7735HS with Radeon Graphics (16) @ 3.200GHz 
GPU: AMD ATI Radeon 680M 
Memory: 3482MiB / 31317MiB 

Rollback to Mesa driver to: 23.0.3-1.fc38 solves the problem.

Comment 6 Anthony J Starks 2023-05-05 20:30:03 UTC
$ journalctl | grep 'May 05.*google-chrome.desktop'


May 05 16:12:55 slab google-chrome.desktop[14653]: // Vertex GLSL
May 05 16:12:55 slab google-chrome.desktop[14653]: #version 300 es
May 05 16:12:55 slab google-chrome.desktop[14653]: #extension GL_NV_shader_noperspective_interpolation : require
May 05 16:12:55 slab google-chrome.desktop[14653]: precision mediump float;
May 05 16:12:55 slab google-chrome.desktop[14653]: precision mediump sampler2D;
May 05 16:12:55 slab google-chrome.desktop[14653]: uniform highp vec4 sk_RTAdjust;
May 05 16:12:55 slab google-chrome.desktop[14653]: in highp vec2 inPosition;
May 05 16:12:55 slab google-chrome.desktop[14653]: in mediump vec4 inColor;
May 05 16:12:55 slab google-chrome.desktop[14653]: in highp vec4 inQuadEdge;
May 05 16:12:55 slab google-chrome.desktop[14653]: noperspective out highp vec4 vQuadEdge_S0;
May 05 16:12:55 slab google-chrome.desktop[14653]: noperspective out mediump vec4 vinColor_S0;
May 05 16:12:55 slab google-chrome.desktop[14653]: void main() {
May 05 16:12:55 slab google-chrome.desktop[14653]:     vQuadEdge_S0 = inQuadEdge;
May 05 16:12:55 slab google-chrome.desktop[14653]:     vinColor_S0 = inColor;
May 05 16:12:55 slab google-chrome.desktop[14653]:     highp vec2 _tmp_0_inPosition = inPosition;
May 05 16:12:55 slab google-chrome.desktop[14653]:     gl_Position = vec4(_tmp_0_inPosition, 0.0, 1.0);
May 05 16:12:55 slab google-chrome.desktop[14653]:     gl_Position = vec4(gl_Position.xy * sk_RTAdjust.xz + gl_Position.ww * sk_RTAdjust.yw, 0.0, gl_Position.w);
May 05 16:12:55 slab google-chrome.desktop[14653]: }
May 05 16:12:55 slab google-chrome.desktop[14653]: // Fragment GLSL
May 05 16:12:55 slab google-chrome.desktop[14653]: #version 300 es
May 05 16:12:55 slab google-chrome.desktop[14653]: #extension GL_NV_shader_noperspective_interpolation : require
May 05 16:12:55 slab google-chrome.desktop[14653]: uniform highp vec2 u_skRTFlip;
May 05 16:12:55 slab google-chrome.desktop[14653]: precision mediump float;
May 05 16:12:55 slab google-chrome.desktop[14653]: precision mediump sampler2D;
May 05 16:12:55 slab google-chrome.desktop[14653]: out mediump vec4 sk_FragColor;
May 05 16:12:55 slab google-chrome.desktop[14653]: noperspective in highp vec4 vQuadEdge_S0;
May 05 16:12:55 slab google-chrome.desktop[14653]: noperspective in mediump vec4 vinColor_S0;
May 05 16:12:55 slab google-chrome.desktop[14653]: void main() {
May 05 16:12:55 slab google-chrome.desktop[14653]:     mediump vec4 outputColor_S0;
May 05 16:12:55 slab google-chrome.desktop[14653]:     outputColor_S0 = vinColor_S0;
May 05 16:12:55 slab google-chrome.desktop[14653]:     mediump float edgeAlpha;
May 05 16:12:55 slab google-chrome.desktop[14653]:     mediump vec2 duvdx = dFdx(vQuadEdge_S0.xy);
May 05 16:12:55 slab google-chrome.desktop[14653]:     mediump vec2 duvdy = (u_skRTFlip.y * dFdy(vQuadEdge_S0.xy));
May 05 16:12:55 slab google-chrome.desktop[14653]:     if (vQuadEdge_S0.z > 0.0 && vQuadEdge_S0.w > 0.0) {
May 05 16:12:55 slab google-chrome.desktop[14653]:         edgeAlpha = min(min(vQuadEdge_S0.z, vQuadEdge_S0.w) + 0.5, 1.0);
May 05 16:12:55 slab google-chrome.desktop[14653]:     } else {
May 05 16:12:55 slab google-chrome.desktop[14653]:         mediump vec2 gF = vec2((2.0 * vQuadEdge_S0.x) * duvdx.x - duvdx.y, (2.0 * vQuadEdge_S0.x) * duvdy.x - duvdy.y);
May 05 16:12:55 slab google-chrome.desktop[14653]:         edgeAlpha = vQuadEdge_S0.x * vQuadEdge_S0.x - vQuadEdge_S0.y;
May 05 16:12:55 slab google-chrome.desktop[14653]:         edgeAlpha = clamp(0.5 - edgeAlpha / length(gF), 0.0, 1.0);
May 05 16:12:55 slab google-chrome.desktop[14653]:     }
May 05 16:12:55 slab google-chrome.desktop[14653]:     mediump vec4 outputCoverage_S0 = vec4(edgeAlpha);
May 05 16:12:55 slab google-chrome.desktop[14653]:     {
May 05 16:12:55 slab google-chrome.desktop[14653]:         sk_FragColor = outputColor_S0 * outputCoverage_S0;
May 05 16:12:55 slab google-chrome.desktop[14653]:     }
May 05 16:12:55 slab google-chrome.desktop[14653]: }
May 05 16:12:55 slab google-chrome.desktop[14653]: Errors:
May 05 16:12:55 slab google-chrome.desktop[14653]: link failed but did not provide an info log

Comment 7 Anatolii Vorona 2023-05-05 22:20:36 UTC
the same issue on fc38 with google-chrome-stable-113.0.5672.63-1.x86_64

OS: Fedora Linux 38 (Workstation Edition) x86_64 
Host: ThinkPad X1 Carbon Gen 8 
Kernel: 6.2.12-300.fc38.x86_64
DE: GNOME 44.1 
CPU: Intel i5-10210U (4) @ 4.200GHz 
GPU: Intel CometLake-U GT2 [UHD Graphics] 
Memory: 3014MiB / 15638MiB 


`dnf downgrade mesa-libGL mesa-libgbm` helps (from 23.0.3-3 to 23.0.1-2)

Comment 8 Nicholas La Roux 2023-05-05 23:58:38 UTC
Confirmed problem on Fedora 38 with Mesa 23.03 and Google Chrome 113.0.5672.63 (Official Build) (64-bit).

OS: Fedora Linux 38 (Workstation Edition) x86_64
Host: N7 B550
Kernel: 6.2.14-300.fc38.x86_64
DE: GNOME 44.1
CPU: AMD Ryzen 9 5900X (24) @ 3.700GHz
GPU: AMD ATI Radeon RX 6800/6800 XT / 6900 XT
Memory: 3506MiB / 32004MiB

Downgrading from 23.0.3-3 to 23.0.1-2 via `dnf downgrade mesa-libGL mesa-libgbm` resolved the issue.

Comment 9 halaitsingh 2023-05-06 02:47:36 UTC
Had the same problem. Deleting `~/.config/google-chrome/ShaderCache` directory worked for me. I did not downgrade anything.

Comment 10 john getsoian 2023-05-06 05:00:42 UTC
halaitsingh said:
>Had the same problem. Deleting `~/.config/google-chrome/ShaderCache` directory worked for me. I did not downgrade anything.

Interesting. I let mesa update again and tried deleting /ShaderCache but it did not work. However, deleting the entire /.config/google-chrome and /.cache/google-chrome dirs did. The later might or might not have have been needed, I didn't check the outcomes individually.

Seems to be a good call.

Comment 11 Vasilis Keramidas 2023-05-06 08:05:20 UTC
On Fedora 38 deleting ~/.config/google-chrome/ worked for me

Comment 12 Mário Lopes 2023-05-06 08:16:26 UTC
On Fedora 38

I let mesa update again and then deleted  ~/.config/google-chrome/GrShaderCache and ~/.config/google-chrome/ShaderCache did not work

However, deleting the entire ~/.config/google-chrome/ worked.

Comment 13 Michael J Gruber 2023-05-06 11:28:00 UTC
Just so that you don't get stuck or remove all of your chrome config by following well-meant advice:

google-chrome-stable --disable-gpu --disable-accelerated-video-encode

Comment 14 DavidC 2023-05-06 23:07:29 UTC
Same as Mario's solution I am on Fedora 38 and deleted: the entire ~/.config/google-chrome/

In the next Chrome launch Google was behaving fine and using hardware acceleration

Comment 15 DavidC 2023-05-06 23:20:07 UTC
Same as Mario's solution I am on Fedora 38 and deleted: the entire ~/.config/google-chrome/

In the next Chrome launch Google was behaving fine and using hardware acceleration.

Comment 16 Michael J Gruber 2023-05-07 10:12:31 UTC
(In reply to DavidC from comment #15)
> Same as Mario's solution I am on Fedora 38 and deleted: the entire
> ~/.config/google-chrome/
> 
> In the next Chrome launch Google was behaving fine and using hardware
> acceleration.

... and did not know its profiles nor config any more, did it?

Comment 17 Michael J Gruber 2023-05-07 10:30:26 UTC
Further observation:

The package now disables clc on ELN because libclc is not available there; it does not pull in libclc nor mesa-libOpenCL on Fedora, though. Mentioning this because it is GPU-related and intel-specific.

Comment 18 Anthony J Starks 2023-05-07 14:03:17 UTC
FYI: changelog for Mesa 23.0.3: https://lists.freedesktop.org/archives/mesa-dev/2023-April/225982.html
Also some mention of this bug in Chromium Issue 1442633: https://bugs.chromium.org/p/chromium/issues/detail?id=1442633

Comment 19 Yanko Kaneti 2023-05-07 14:28:46 UTC
Removing .config/google-chrome/Default/GPUCache is what helped here with 23.0.3-4.fc39

Perhaps I should close the bug given its probably not mesa-s fault..

Comment 20 john getsoian 2023-05-07 20:36:38 UTC
yaneti wrote:
>Perhaps I should close the bug given its probably not mesa-s fault

Have had no further issue here since re-initializing Chrome.

Comment 21 Dennis Jacobfeuerborn 2023-05-07 20:51:24 UTC
Seeing the same thing with a Radeon RX-7900-XTX. Using "dnf downgrade mesa-libGL" to downgrade to 23.0.1-2.fc38 fixed things for me.

Comment 22 Dane 2023-05-08 07:52:09 UTC
Can confirm I'm seeing the same thing on Fedora 38 with an AMD Radeon RX590. 

Removing .config/google-chrome/*Shader* directories did nothing. Downgrading mesa-libGL fixed the Chrome issue but now I don't have any animations within Gnome, so I suspect the downgrade has broken something else instead.

Comment 23 Anatolii Vorona 2023-05-08 08:07:08 UTC
```
mesa-dri-drivers.x86_64                          23.0.3-3.fc38                       @updates                                               
mesa-filesystem.x86_64                           23.0.3-3.fc38                       @updates                                               
mesa-libEGL.x86_64                               23.0.3-3.fc38                       @updates                                               
mesa-libgbm.x86_64                               23.0.3-3.fc38                       @updates                                               
mesa-libglapi.x86_64                             23.0.3-3.fc38                       @updates                                               
mesa-libGLU.x86_64                               9.0.1-8.fc38                        @fedora                                                
mesa-libGL.x86_64                                23.0.3-3.fc38                       @updates                                               
mesa-va-drivers.x86_64                           23.0.3-3.fc38                       @updates                                               
mesa-vulkan-drivers.x86_64                       23.0.3-3.fc38                       @updates
```
Removing .config/google-chrome/Default/GPUCache resolved the issue.

Comment 24 Nicholas La Roux 2023-05-08 08:45:07 UTC
> Removing .config/google-chrome/Default/GPUCache resolved the issue.

Can confirm as someome who had to downgrade to Mesa 23.0.1-2 to fix Chrome issues with Mesa 23.0.3-3 before, upgrading then removing .config/google-chrome/Default/GPUCache resolved the issue. 

OS: Fedora Linux 38 (Workstation Edition) x86_64
Kernel: 6.2.14-300.fc38.x86_64
DE: GNOME 44.1

Comment 25 Mike Harvey 2023-05-16 12:05:27 UTC
This is an ugly bug.  Downgrade did fix the problem:  sudo dnf downgrade mesa-libGL

Comment 26 Laurențiu Nicola 2023-05-18 07:36:37 UTC
*** Bug 2208152 has been marked as a duplicate of this bug. ***

Comment 27 Laurențiu Nicola 2023-05-18 07:40:27 UTC
Funny, I fixed my chromium by deleting GPUCache, but GNOME animations work fine. I didn't have to downgrade anything. AMD GPU.

Comment 28 Nicholas La Roux 2023-05-19 12:40:17 UTC
Strangely, 11 days after upgrading to latest Mesa and removing `.config/google-chrome/Default/GPUCache` and not having any issues, I'm now experiencing graphical issues in Chrome again. Falling back to downgrading to previous Mesa version resolves the issue once again.

Comment 29 Dane 2023-05-19 13:48:46 UTC
I also had similar issues again this morning.

Deleting .config/google-chrome/Default/GPUCache and restarting Chrome (chrome://restart) resolved the issue for me, again.

Comment 30 john getsoian 2023-05-19 19:34:41 UTC
yes - when Chrome upgraded to 113.0.5672.126, I also started to see my Chrome window drop-out and rapidly repaint, sometimes incompletely. Disabling acceleration stopped it. Then, as noted, deleting the GPUcache allowed acceleration to be reapplied an no further problem so far.

Comment 31 Sammy 2023-05-26 14:04:44 UTC
I had fixed this by removing .config/google-chrome but with today's update to 23.0.3-5 the problem reappeared. It may be related to the issue #2193135, which seems to be related.

Comment 32 Hamid 2023-05-28 06:07:15 UTC
I also confirm the problem is back with 23.0.3-5.fc38 on Fedora 38 with Chromium and chrome Version 113.0.5672.126.

Comment 33 DexterG 2023-05-28 13:31:05 UTC
Google Chrome would not run properly this morning after installing the Fedora 38 automatic updates last night. I searched bugzilla to see if this problem had already been reported, which lead me to this bug report. I don't know what Mesa is, but the symptoms symptoms described match mine: Chrome opens but the window does not redraw properly and the menus flicker and nothing works.

Deleting .config/google-chrome/Default/GPUCache and restarting Chrome, as suggested by others, resolved the issue for me.

Comment 34 john getsoian 2023-05-28 15:58:09 UTC
It does seem like each Chrome update is falling over on its own cached data. I have zero knowledge around video subsystems but would love a plain English explanation of how that would occur. In any case from an uninformed view at least, it doesn't seem like is the OS's responsibility to insure that an application doesn't foul its own nest. ;)

Comment 35 Manvendra Bhangui 2023-05-29 05:56:00 UTC
I did the following

1) ~/.config/google-chrome/ShaderCache
2) rm -rf ~/.config/google-chrome/Default/GPUCache

The above two did not help. On further investigating I found many directories named GPUCache. Deleted all of them and now chrome displays images correctly.

3) find ~/.config/google-chrome -name GPUCache -exec /bin/rm -rf {} \;

Comment 36 Sandro Bonazzola 2023-05-29 06:24:40 UTC
The change seems to be included also in mesa-23.0.3-5.fc38 which is now in stable, breaking google chrome.
Reverting to 23.0.1-2 make it working again.
Flagging as regression and raising severity to high.

Comment 37 Sandro Bonazzola 2023-05-29 06:29:51 UTC
Workaround in comment #35 fixed for me, dropping regression as it seems to be a chrome issue rather than a mesa issue and reducing severity to medium as simple workaround exists.

Comment 38 Sammy 2023-05-29 12:39:38 UTC
Why is it a chrome issue? First, chrome is not updated. This happens after mesa updates with the same chrome version. Second, it is not happening on systems using nvidia propriety drivers but only on those using mesa.

Comment 39 Adam Williamson 2023-05-29 15:57:05 UTC
CCing Than and spot - can you see if chromium is doing something wrong here? It looks rather like it's caching stuff that it's not safe to cache across mesa versions.

Comment 40 Anthony J Starks 2023-05-29 20:31:29 UTC
(In reply to Sammy from comment #38)
> Why is it a chrome issue? First, chrome is not updated. This happens after
> mesa updates with the same chrome version. Second, it is not happening on
> systems using nvidia propriety drivers but only on those using mesa.

See: https://bugs.chromium.org/p/chromium/issues/detail?id=1442633 for the Chrome side of the story

Comment 41 Tom "spot" Callaway 2023-05-30 14:21:39 UTC
The upstream ticket (despite the noise) is enlightening, specifically: https://bugs.chromium.org/p/chromium/issues/detail?id=1442633#c14

A couple of things stand out:
1. This is not just a Fedora bug, lots of other distributions who are updating Mesa are having this issue.
2. This is not just a _chromium_ bug, it happens in Google Chrome.
3. It doesn't happen with the NVIDIA proprietary driver in play, which makes sense because it uses its own libGL (thus, distro Mesa changes are irrelevant)
4. I don't think it happens without hardware acceleration turned on, because without this, it uses the egl-angle copy inside of Chrome/Chromium. This makes it slightly more difficult to reproduce in a VM, but you can see the change if you go to chrome://flags and Enable "Override software rendering list".

On a stock Fedora 38 system with hardware acceleration enabled, this is what chrome://gpu shows for GL_RENDERER:

ANGLE (Mesa, llvmpipe (LLVM 16.0.0 256 bits), OpenGL 4.5 (Core Profile) Mesa 23.0.1)

This matches with the Fedora mesa-libGL: mesa-libGL-23.0.1-2.fc38.x86_64

Chrome/Chromium says it uses the GL_RENDERER string to detect when something changes in Mesa that would cause the GL Shader cache to be incompatible. Updating to the latest mesa-libGL in Fedora (mesa-libGL-23.0.3-5.fc38) shows this GL_RENDERER string in Chromium:

ANGLE (Mesa, llvmpipe (LLVM 16.0.4 256 bits), OpenGL 4.5 (Core Profile) Mesa 23.0.3)

Now, I _think_ what is happening here is that we're doing a significant number of Mesa updates in the Fedora 38 life-cycle:

* mesa-23.0.3-5.fc38
* mesa-23.0.3-3.fc38
* mesa-23.0.3-1.fc38
* mesa-23.0.2-2.fc38
* mesa-23.0.2-1.fc38
* mesa-23.0.1-2.fc38 (I think this one was GA)
* mesa-23.0.1-1.fc38

Here is where I defer to smarter Mesa maintainers, but it seems like at least in some of these cases, when the $MAJOR.$MINOR.$RELEASE Mesa version isn't changing, but the code is, the result is that Mesa has changed something making the GL Shader cache from Chrome/Chromium incompatible BUT the string that Chrome/Chromium checks isn't changed. Chrome/Chromium generates a ShaderPrefixKey to determine if the cache needs to be invalidated, the code for generating the ShaderPrefixKey is here:

https://source.chromium.org/chromium/chromium/src/+/refs/heads/main:components/viz/host/gpu_host_impl.cc;l=410;drc=4a0c2a8e5a4de735817463162218bbf84aa2bd74

I can see a few ways to "fix" this:

1. Just disable hardware acceleration (lol, okay, sorry).
2. Never update Mesa in Fedora. Obviously, this is sarcasm too, but it is why software rendering doesn't break, because Google bundles libGL. (They can't do this for hw rendering because NVIDIA.)
3. No Fedora Mesa update that doesn't change version should ever change ... uh... whatever it changes that causes the GL Shader cache to be incompatible in Chrome/Chromium without a corresponding $MAJOR.$MINOR.$RELEASE increment, which also probably isn't plausible.
4. Chrome/Chromium should handle invalid cache more gracefully and invalidate it when it causes errors rather than being broken. This may be more difficult than it sounds, especially if Chrome/Chromium still "works" and the only sign of error is visual corruption from bad cache. Not sure if that's always true though, so maybe this is possible.
5. Mesa's GL_RENDERER string incorporates the _build_ int ("%{release}" from the Fedora package) into the GL_RENDERER string. There are a few downsides to this: I think it would need to just be the int, not the int + disttag, because I'm sure something out there is parsing GL_RENDERER and expecting that part to be ints. I'm also certain something is not expecting a fourth .$VAR in that string and will break whether or not the disttag is there. Ignoring parsing issues, this also means that every time Fedora updates Mesa without changing the $MAJOR.$MINOR.$RELEASE, and the change in Mesa is _not_ something that causes GPUCache incompatibility in Chrome/Chromium, Chrome/Chromium would expire the cache without needing to. Arguably, this is a better outcome than corrupt cache, but if the rate of Mesa updates is aggressive, it could slow down Chrome/Chromium some amount on each reload. 

I don't see an obvious or easy fix that Fedora can make here. Fedora users who hit this problem do have a workaround, to manually get rid of all the GPUCache files as described here: https://bugzilla.redhat.com/show_bug.cgi?id=2193335#c35 ... but there's no obvious way for them to know that, and I'm not sure if we could meaningfully hack in a warning to output to the console (and a large % of users probably aren't launching from a terminal/console anyway)... bleh.

Comment 42 Tom "spot" Callaway 2023-05-30 14:24:40 UTC
Oh, I forgot. It seems like there might also be cases where the $MAJOR.$MINOR.$VERSION _is_ changed and Chrome/Chromium is still trying to use the old cache. It's hard to say for sure if that's actually happening and if it is, I'm not sure why. Chrome/Chromium's ShaderPrefixKey should be different. I'm assuming for now that this isn't really happening and focusing on the bug that does seem to happen more clearly.

Comment 43 Kamil Páral 2023-05-31 08:44:37 UTC
Tom, a great summary. But it is very interesting that people have hit this bug even with mesa-23.0.3-2.fc37 update [1], which haven't caused ANY change to the source code, just adjusted dependencies (see the list of commits [2], it only contained this one [3] since mesa-23.0.3-1.fc37). Of course, even if the source tarball hasn't changed, the build dependencies have ([4] vs [5]). One notable change is llvm-devel-15.0.7-1.fc37 vs llvm-devel-15.0.7-2.fc37, which changed some build flags [6] (I'm not able to judge their effect). And this is just one library, other libraries likely changed as well. So even a simple package dependency change in the spec file can trigger this problem, probably because of changed build dependencies.

I think the important question here is *why now and not before*? What changed in Chromium/elsewhere that we suddenly see this problem, and why haven't we seen it in all those months and years before? This is surely not the first time we do release-only bumps in Mesa.


[1] https://bodhi.fedoraproject.org/updates/FEDORA-2023-84965ba750
[2] https://src.fedoraproject.org/rpms/mesa/commits/f37
[3] https://src.fedoraproject.org/rpms/mesa/c/fda7bc52efe5534ebabe978f02265354b66a9419?branch=f37
[4] https://kojipkgs.fedoraproject.org//packages/mesa/23.0.3/1.fc37/data/logs/x86_64/root.log
[5] https://kojipkgs.fedoraproject.org//packages/mesa/23.0.3/2.fc37/data/logs/x86_64/root.log
[6] https://src.fedoraproject.org/rpms/llvm/c/8f088ecd033b841b65857bfd897e706ede848cae?branch=f37

Comment 44 Mario Limonciello 2023-05-31 14:58:00 UTC
This issue happened to me as well when just LLVM updated recently but *not* mesa.

> 5. Mesa's GL_RENDERER string incorporates the _build_ int ("%{release}" from the Fedora package) into the GL_RENDERER string. There are a few downsides to this: I think it would need to just be the int, not the int + disttag, because I'm sure something out there is parsing GL_RENDERER and expecting that part to be ints. I'm also certain something is not expecting a fourth .$VAR in that string and will break whether or not the disttag is there. Ignoring parsing issues, this also means that every time Fedora updates Mesa without changing the $MAJOR.$MINOR.$RELEASE, and the change in Mesa is _not_ something that causes GPUCache incompatibility in Chrome/Chromium, Chrome/Chromium would expire the cache without needing to. Arguably, this is a better outcome than corrupt cache, but if the rate of Mesa updates is aggressive, it could slow down Chrome/Chromium some amount on each reload. 


This is somewhat similar to how this bug is aiming to be handled in Ubuntu.  They're looking to mangle the upstream version so that it's incorporating the package version.

https://bugs.launchpad.net/ubuntu/+source/mesa/+bug/2020604

Comment 45 Tom "spot" Callaway 2023-05-31 15:01:21 UTC
(In reply to Kamil Páral from comment #43)

> I think the important question here is *why now and not before*? What
> changed in Chromium/elsewhere that we suddenly see this problem, and why
> haven't we seen it in all those months and years before? This is surely not
> the first time we do release-only bumps in Mesa.

Honestly? No clue. It could be a number of things. Maybe the previous release-only bumps in Mesa didn't change whatever got changed this time. Maybe it's tied into the _compiler_ used on Mesa. Maybe Chrome/Chromium changed the way it makes GL Shader cache.

Comment 46 Michel Dänzer 2023-06-01 09:42:25 UTC
(In reply to Mario Limonciello from comment #44)
> This issue happened to me as well when just LLVM updated recently but *not*
> mesa.
> 
> [...]
> 
> This is somewhat similar to how this bug is aiming to be handled in Ubuntu. 
> They're looking to mangle the upstream version so that it's incorporating
> the package version.

That won't help when only LLVM changes though. (It's also not clear that chromium is paying full attention to changes in the renderer string)

I wonder if Mesa couldn't check whether the application's cached shader data is from the current builds of itself and LLVM, similar to what is done for Mesa's own shader cache.

Comment 47 Warren Togami 2023-09-11 16:39:45 UTC
*** Bug 2238198 has been marked as a duplicate of this bug. ***

Comment 48 Mario Limonciello 2023-12-22 15:59:41 UTC
> That won't help when only LLVM changes though. (It's also not clear that chromium is paying full attention to changes in the renderer string)

Of course it happened again today after the mesa upgrade from 23.3.1-3.fc39 to 23.3.1-4.fc39.
Manually clearing the caches helps.

# rm GrShaderCache/ ShaderCache/ Default/GPUCache/ -rf

Mesa upgrades are more common than LLVM upgrades are, how about doing what Ubuntu did to encode that VERSION string to at least help this along like Ubuntu did?  There shouldn't be any harm in this.

Comment 49 Sammy 2023-12-22 16:05:37 UTC
I can confirm this today!

Comment 50 Gurenko Alex 2023-12-22 17:00:49 UTC
I've suggested in various forums somewhat universal workaround:

$ for i in $(find ~/.config ~/.var -type d -name "GPUCache" 2>/dev/null); do rm -ri ${i}; done 

This also clears cache for flatpak applications as they are also affected. Feel free to replace -ri with -rf for "silent clean"


Note You need to log in before you can comment on or make changes to this bug.