Bug 2193135

Summary: A partial Mesa update is possible even after 23.0.3-1
Product: [Fedora] Fedora Reporter: Kamil Páral <kparal>
Component: mesaAssignee: Adam Jackson <ajax>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rawhideCC: ajax, bskeggs, felix, igor.raits, j, lyude, mail, rhughes, rstrode, tstellar, walter.pete
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: mesa-23.0.3-5.fc38 mesa-23.1.4-1.fc39 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-13 19:21:43 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
full dnf output when installing gnome-calculator none

Description Kamil Páral 2023-05-04 12:31:33 UTC
We tried to fix the possibility for a partial Mesa update in bug 2187726. The fix is in mesa-23.0.3-1. However, today I found out that you can still get a partial update (breaking GUI apps), even when you have 23.0.3-1 as a baseline.

I found the problem in a toolbox container:

$ toolbox create --release 38 f38tb
$ toolbox enter f38tb

$ rpm -qa | grep mesa | sort
mesa-dri-drivers-23.0.1-1.fc38.x86_64
mesa-filesystem-23.0.1-1.fc38.x86_64
mesa-libglapi-23.0.1-1.fc38.x86_64
mesa-va-drivers-23.0.1-1.fc38.x86_64
mesa-vulkan-drivers-23.0.1-1.fc38.x86_64


Let's update everything to 23.0.3-1, which contains dependency fixes:

$ sudo dnf update mesa\*
...

$ rpm -qa | grep mesa
mesa-filesystem-23.0.3-1.fc38.x86_64
mesa-va-drivers-23.0.3-1.fc38.x86_64
mesa-libglapi-23.0.3-1.fc38.x86_64
mesa-dri-drivers-23.0.3-1.fc38.x86_64
mesa-vulkan-drivers-23.0.3-1.fc38.x86_64


Now you would think that a dependency problem can't happen, we fixed it. And on the first glance, it seems fixed. Mesa 23.0.3-3 is currently in updates-testing, let's test a partial update:

$ sudo dnf update mesa-va-drivers --enablerepo=updates-testing
Last metadata expiration check: 0:08:09 ago on Thu 04 May 2023 02:09:27 PM CEST.
Dependencies resolved.
==============================================================================
 Package              Arch       Version            Repository           Size
==============================================================================
Upgrading:
 mesa-dri-drivers     x86_64     23.0.3-3.fc38      updates-testing      18 M
 mesa-filesystem      x86_64     23.0.3-3.fc38      updates-testing      18 k
 mesa-libglapi        x86_64     23.0.3-3.fc38      updates-testing      54 k
 mesa-va-drivers      x86_64     23.0.3-3.fc38      updates-testing     3.4 M


Looks fine. Mesa libGL and libEGL are not installed by default in the container, let's try to install them:

$ sudo dnf install mesa-libGL --enablerepo=updates-testing
Last metadata expiration check: 0:09:02 ago on Thu 04 May 2023 02:09:27 PM CEST.
Dependencies resolved.
==============================================================================
 Package              Arch       Version            Repository           Size
==============================================================================
Installing:
 mesa-libGL           x86_64     23.0.3-3.fc38      updates-testing     173 k
Upgrading:
 mesa-dri-drivers     x86_64     23.0.3-3.fc38      updates-testing      18 M
 mesa-filesystem      x86_64     23.0.3-3.fc38      updates-testing      18 k
 mesa-libglapi        x86_64     23.0.3-3.fc38      updates-testing      54 k
 mesa-va-drivers      x86_64     23.0.3-3.fc38      updates-testing     3.4 M
Installing dependencies:
 libXfixes            x86_64     6.0.0-5.fc38       fedora               19 k
 libXxf86vm           x86_64     1.1.5-2.fc38       fedora               18 k
 libglvnd             x86_64     1:1.6.0-2.fc38     fedora              134 k
 libglvnd-glx         x86_64     1:1.6.0-2.fc38     fedora              142 k

$ sudo dnf install mesa-libEGL --enablerepo=updates-testing
Last metadata expiration check: 0:09:20 ago on Thu 04 May 2023 02:09:27 PM CEST.
Dependencies resolved.
==============================================================================
 Package              Arch      Version              Repository          Size
==============================================================================
Installing:
 mesa-libEGL          x86_64    23.0.3-3.fc38        updates-testing    130 k
Upgrading:
 mesa-dri-drivers     x86_64    23.0.3-3.fc38        updates-testing     18 M
 mesa-filesystem      x86_64    23.0.3-3.fc38        updates-testing     18 k
 mesa-libglapi        x86_64    23.0.3-3.fc38        updates-testing     54 k
 mesa-va-drivers      x86_64    23.0.3-3.fc38        updates-testing    3.4 M


Looks fine.

But now, look what happens, when we try to install gnome-calculator:

$ sudo dnf install gnome-calculator --assumeno --enablerepo updates-testing | grep mesa
 mesa-libEGL  x86_64  23.0.1-2.fc38  fedora           130 k
 mesa-libGL   x86_64  23.0.3-1.fc38  updates          173 k
 mesa-libgbm  x86_64  23.0.3-3.fc38  updates-testing   45 k


What on earth is happening here?! One package is pulled from fedora repo, one from updates, and only the third one gets the latest version from updates-testing. I don't understand why this happens.

But if you confirm this, GUI apps are broken:

$ gnome-calculator 
libEGL fatal: DRI driver not from this Mesa build ('23.0.3' vs '23.0.1')


Mesa versions are completely mixed up (3 different mesa versions installed):

$ rpm -qa | grep mesa | sort
mesa-dri-drivers-23.0.3-1.fc38.x86_64
mesa-filesystem-23.0.3-1.fc38.x86_64
mesa-libEGL-23.0.1-2.fc38.x86_64
mesa-libgbm-23.0.3-3.fc38.x86_64
mesa-libGL-23.0.3-1.fc38.x86_64
mesa-libglapi-23.0.3-1.fc38.x86_64
mesa-va-drivers-23.0.3-1.fc38.x86_64
mesa-vulkan-drivers-23.0.3-1.fc38.x86_64


I wonder if we're screwed up because the base repo contains a mesa version which doesn't have the strict dependencies we added in bug 2187726, and therefore the dependency solver can always fall back to it. But there might be even more issues combined. This is getting *very* complicated and going above my head.

Reproducible: Always

Comment 1 Kamil Páral 2023-05-04 12:33:23 UTC
Created attachment 1962207 [details]
full dnf output when installing gnome-calculator

Comment 2 Felix Kaechele 2023-05-04 13:13:18 UTC
I think your assessment is correct. The depsolver will fall back to whichever available mesa-libEGL it can find that satisfies all constraints of the transaction. And mesa-libEGL-23.0.1-2.fc38.x86_64 in the base repo will always be happy to oblige ;-)

Same goes for downgrades (bug 2191745). A downgrade of mesa-dri-drivers will always lead to a broken system (now and in the past). But downgrades, I believe, aren't a use case that come with any QA criteria attached, so technically NOTABUG. There's likely plenty of packages that, if downgraded, would either revert a huge portion of your system back to the base repository versions due to interdependencies or break your system because of undocumented dependencies (as is the case here).

Downgrade logs:

[felix@x1 ~]$ rpm -qa | grep mesa
mesa-libGLU-9.0.1-8.fc38.x86_64
mesa-libGLU-devel-9.0.1-8.fc38.x86_64
mesa-libglapi-23.0.3-1.fc38.x86_64
mesa-libgbm-23.0.3-1.fc38.x86_64
mesa-libGL-23.0.3-1.fc38.x86_64
mesa-libGL-devel-23.0.3-1.fc38.x86_64
mesa-libEGL-23.0.3-1.fc38.x86_64
mesa-libOSMesa-23.0.3-1.fc38.x86_64
mesa-vulkan-drivers-23.0.3-1.fc38.x86_64
mesa-libxatracker-23.0.3-1.fc38.x86_64
mesa-libglapi-23.0.3-1.fc38.i686
mesa-libgbm-23.0.3-1.fc38.i686
mesa-libEGL-23.0.3-1.fc38.i686
mesa-libGL-23.0.3-1.fc38.i686
mesa-libOSMesa-23.0.3-1.fc38.i686
mesa-vulkan-drivers-23.0.3-1.fc38.i686
mesa-filesystem-23.0.3-1.fc38.x86_64
mesa-filesystem-23.0.3-1.fc38.i686
mesa-dri-drivers-23.0.3-1.fc38.x86_64
mesa-dri-drivers-23.0.3-1.fc38.i686

[felix@x1 ~]$ sudo dnf downgrade mesa-dri-drivers
Last metadata expiration check: 0:04:04 ago on Thu 04 May 2023 08:44:19 AM.
Dependencies resolved.
================================================================================
 Package                 Architecture  Version              Repository     Size
================================================================================
Downgrading:
 mesa-dri-drivers        i686          23.0.1-2.fc38        fedora         20 M
 mesa-dri-drivers        x86_64        23.0.1-2.fc38        fedora         18 M
 mesa-filesystem         i686          23.0.1-2.fc38        fedora         18 k
 mesa-filesystem         x86_64        23.0.1-2.fc38        fedora         18 k

Transaction Summary
================================================================================
Downgrade  4 Packages

Comment 3 Kamil Páral 2023-05-04 16:21:47 UTC
(In reply to Felix Kaechele from comment #2)
> The depsolver will fall back to
> whichever available mesa-libEGL it can find that satisfies all constraints
> of the transaction. And mesa-libEGL-23.0.1-2.fc38.x86_64 in the base repo
> will always be happy to oblige ;-)

The question is *why* is this happening, in which cases it happens, so that we can try to do something about it.

Comment 4 Kamil Páral 2023-05-05 08:12:15 UTC
Mesa 23.0.3-3 has been unfortunately pushed stable overnight, so the reproducer no longer works as described. But a similar issue can be seen when we use the default package versions and just try to install gnome-calculator. This will install mesa-libEGL 23.0.3-3 but keep mesa-dri-drivers at 23.0.1-1. gnome-calculator will then fail to start due to version mismatch.

$ rpm -qa | grep mesa | sort
mesa-dri-drivers-23.0.1-1.fc38.x86_64
mesa-filesystem-23.0.1-1.fc38.x86_64
mesa-libglapi-23.0.1-1.fc38.x86_64
mesa-va-drivers-23.0.1-1.fc38.x86_64
mesa-vulkan-drivers-23.0.1-1.fc38.x86_64

$ sudo dnf install gnome-calculator --assumeno | grep mesa
 mesa-libglapi    x86_64  23.0.3-3.fc38    updates   54 k
 mesa-libEGL      x86_64  23.0.3-3.fc38    updates  130 k
 mesa-libGL       x86_64  23.0.3-3.fc38    updates  173 k
 mesa-libgbm      x86_64  23.0.3-3.fc38    updates   45 k
Operation aborted.

Comment 5 Kamil Páral 2023-05-05 10:54:34 UTC
(In reply to Kamil Páral from comment #3)
> The question is *why* is this happening, in which cases it happens, so that
> we can try to do something about it.

The "why" is probably "because it can". Even when I try a similar situation with `dnf --best`, it still picks older versions. And in `man dnf`, this is explicitly stated under --best documentation:

-b, --best
       ...
       Note that the use of the newest available version is only guaranteed for  the  packages
       directly  requested  (e.g.  as  a command line arguments), and the solver may use older
       versions of dependencies to meet their requirements.

I can only speculate that it's prioritizing the smallest transaction possible. So if installing older mesa-libEGL causes fewer packages to be installed then installing newer mesa-libGL (because that would cause an update for all mesa-* subpackages), it picks the former.

This is further backup up by the fact, that when I replace `sudo dnf install gnome-calculator` with `sudo dnf install gnome-calculator mesa-libEGL`, it picks up the latest versions instead. That matches the documentation.

So, this is unfortunate. But, I think we can still fix this. I'm currently working on a potential solution and testing it, I'll post it once ready.

Comment 6 Kamil Páral 2023-05-05 12:46:44 UTC
This is my attempt to solve this situation:
https://src.fedoraproject.org/rpms/mesa/pull-request/24

Please note that I only now noticed that the f38 toolbox image contains mesa 23.0.1-1, while the base 'fedora' repo contains mesa 23.0.1-2 (built against llvm 16, on top of that). This sadly further changes the dnf output when testing things. But I decided to ignore this issue, it's a bug in toolbox OCI (it needs to be regenerated using final rpms), and it's not the cause of these dependency issues, it just helped to uncover it.

Comment 7 Kamil Páral 2023-05-23 09:49:06 UTC
Adam, or some other Mesa maintainer, can you please look at the PR in comment 6? Thanks!

Comment 8 Pete Walter 2023-05-25 13:28:22 UTC
I went ahead and merged this. Thanks!

Comment 9 Fedora Update System 2023-05-25 19:49:27 UTC
FEDORA-2023-a20a26a598 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a20a26a598

Comment 10 Fedora Update System 2023-05-25 19:49:30 UTC
FEDORA-2023-84965ba750 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-84965ba750

Comment 11 Fedora Update System 2023-05-26 02:10:42 UTC
FEDORA-2023-84965ba750 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-84965ba750`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-84965ba750

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Fedora Update System 2023-05-26 02:23:05 UTC
FEDORA-2023-a20a26a598 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-a20a26a598`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-a20a26a598

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Kamil Páral 2023-05-26 13:38:19 UTC
(In reply to Pete Walter from comment #8)
> I went ahead and merged this. Thanks!

Thanks Pete! Sorry, the PR description wasn't completely clear, but a part of this also needs to go to Rawhide. I created another PR in https://src.fedoraproject.org/rpms/mesa/pull-request/25 .

Comment 14 Fedora Update System 2023-05-27 01:17:59 UTC
FEDORA-2023-a20a26a598 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Kamil Páral 2023-05-29 06:55:11 UTC
Reopening for Rawhide (comment 13).

Comment 16 Kamil Páral 2023-07-07 13:50:14 UTC
Pete, could you please also merge the PR from comment 13?

Comment 17 Kamil Páral 2023-07-13 19:21:43 UTC
(In reply to Kamil Páral from comment #16)
> Pete, could you please also merge the PR from comment 13?

This is now merged, closing. Thanks Igor.

Comment 18 Pete Walter 2023-07-17 08:59:52 UTC
Thanks!

Comment 19 Fedora Update System 2023-07-22 23:24:47 UTC
FEDORA-2023-5a0786b714 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-5a0786b714

Comment 20 Fedora Update System 2023-07-23 00:56:58 UTC
FEDORA-2023-5a0786b714 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.