We tried to fix the possibility for a partial Mesa update in bug 2187726. The fix is in mesa-23.0.3-1. However, today I found out that you can still get a partial update (breaking GUI apps), even when you have 23.0.3-1 as a baseline. I found the problem in a toolbox container: $ toolbox create --release 38 f38tb $ toolbox enter f38tb $ rpm -qa | grep mesa | sort mesa-dri-drivers-23.0.1-1.fc38.x86_64 mesa-filesystem-23.0.1-1.fc38.x86_64 mesa-libglapi-23.0.1-1.fc38.x86_64 mesa-va-drivers-23.0.1-1.fc38.x86_64 mesa-vulkan-drivers-23.0.1-1.fc38.x86_64 Let's update everything to 23.0.3-1, which contains dependency fixes: $ sudo dnf update mesa\* ... $ rpm -qa | grep mesa mesa-filesystem-23.0.3-1.fc38.x86_64 mesa-va-drivers-23.0.3-1.fc38.x86_64 mesa-libglapi-23.0.3-1.fc38.x86_64 mesa-dri-drivers-23.0.3-1.fc38.x86_64 mesa-vulkan-drivers-23.0.3-1.fc38.x86_64 Now you would think that a dependency problem can't happen, we fixed it. And on the first glance, it seems fixed. Mesa 23.0.3-3 is currently in updates-testing, let's test a partial update: $ sudo dnf update mesa-va-drivers --enablerepo=updates-testing Last metadata expiration check: 0:08:09 ago on Thu 04 May 2023 02:09:27 PM CEST. Dependencies resolved. ============================================================================== Package Arch Version Repository Size ============================================================================== Upgrading: mesa-dri-drivers x86_64 23.0.3-3.fc38 updates-testing 18 M mesa-filesystem x86_64 23.0.3-3.fc38 updates-testing 18 k mesa-libglapi x86_64 23.0.3-3.fc38 updates-testing 54 k mesa-va-drivers x86_64 23.0.3-3.fc38 updates-testing 3.4 M Looks fine. Mesa libGL and libEGL are not installed by default in the container, let's try to install them: $ sudo dnf install mesa-libGL --enablerepo=updates-testing Last metadata expiration check: 0:09:02 ago on Thu 04 May 2023 02:09:27 PM CEST. Dependencies resolved. ============================================================================== Package Arch Version Repository Size ============================================================================== Installing: mesa-libGL x86_64 23.0.3-3.fc38 updates-testing 173 k Upgrading: mesa-dri-drivers x86_64 23.0.3-3.fc38 updates-testing 18 M mesa-filesystem x86_64 23.0.3-3.fc38 updates-testing 18 k mesa-libglapi x86_64 23.0.3-3.fc38 updates-testing 54 k mesa-va-drivers x86_64 23.0.3-3.fc38 updates-testing 3.4 M Installing dependencies: libXfixes x86_64 6.0.0-5.fc38 fedora 19 k libXxf86vm x86_64 1.1.5-2.fc38 fedora 18 k libglvnd x86_64 1:1.6.0-2.fc38 fedora 134 k libglvnd-glx x86_64 1:1.6.0-2.fc38 fedora 142 k $ sudo dnf install mesa-libEGL --enablerepo=updates-testing Last metadata expiration check: 0:09:20 ago on Thu 04 May 2023 02:09:27 PM CEST. Dependencies resolved. ============================================================================== Package Arch Version Repository Size ============================================================================== Installing: mesa-libEGL x86_64 23.0.3-3.fc38 updates-testing 130 k Upgrading: mesa-dri-drivers x86_64 23.0.3-3.fc38 updates-testing 18 M mesa-filesystem x86_64 23.0.3-3.fc38 updates-testing 18 k mesa-libglapi x86_64 23.0.3-3.fc38 updates-testing 54 k mesa-va-drivers x86_64 23.0.3-3.fc38 updates-testing 3.4 M Looks fine. But now, look what happens, when we try to install gnome-calculator: $ sudo dnf install gnome-calculator --assumeno --enablerepo updates-testing | grep mesa mesa-libEGL x86_64 23.0.1-2.fc38 fedora 130 k mesa-libGL x86_64 23.0.3-1.fc38 updates 173 k mesa-libgbm x86_64 23.0.3-3.fc38 updates-testing 45 k What on earth is happening here?! One package is pulled from fedora repo, one from updates, and only the third one gets the latest version from updates-testing. I don't understand why this happens. But if you confirm this, GUI apps are broken: $ gnome-calculator libEGL fatal: DRI driver not from this Mesa build ('23.0.3' vs '23.0.1') Mesa versions are completely mixed up (3 different mesa versions installed): $ rpm -qa | grep mesa | sort mesa-dri-drivers-23.0.3-1.fc38.x86_64 mesa-filesystem-23.0.3-1.fc38.x86_64 mesa-libEGL-23.0.1-2.fc38.x86_64 mesa-libgbm-23.0.3-3.fc38.x86_64 mesa-libGL-23.0.3-1.fc38.x86_64 mesa-libglapi-23.0.3-1.fc38.x86_64 mesa-va-drivers-23.0.3-1.fc38.x86_64 mesa-vulkan-drivers-23.0.3-1.fc38.x86_64 I wonder if we're screwed up because the base repo contains a mesa version which doesn't have the strict dependencies we added in bug 2187726, and therefore the dependency solver can always fall back to it. But there might be even more issues combined. This is getting *very* complicated and going above my head. Reproducible: Always
Created attachment 1962207 [details] full dnf output when installing gnome-calculator
I think your assessment is correct. The depsolver will fall back to whichever available mesa-libEGL it can find that satisfies all constraints of the transaction. And mesa-libEGL-23.0.1-2.fc38.x86_64 in the base repo will always be happy to oblige ;-) Same goes for downgrades (bug 2191745). A downgrade of mesa-dri-drivers will always lead to a broken system (now and in the past). But downgrades, I believe, aren't a use case that come with any QA criteria attached, so technically NOTABUG. There's likely plenty of packages that, if downgraded, would either revert a huge portion of your system back to the base repository versions due to interdependencies or break your system because of undocumented dependencies (as is the case here). Downgrade logs: [felix@x1 ~]$ rpm -qa | grep mesa mesa-libGLU-9.0.1-8.fc38.x86_64 mesa-libGLU-devel-9.0.1-8.fc38.x86_64 mesa-libglapi-23.0.3-1.fc38.x86_64 mesa-libgbm-23.0.3-1.fc38.x86_64 mesa-libGL-23.0.3-1.fc38.x86_64 mesa-libGL-devel-23.0.3-1.fc38.x86_64 mesa-libEGL-23.0.3-1.fc38.x86_64 mesa-libOSMesa-23.0.3-1.fc38.x86_64 mesa-vulkan-drivers-23.0.3-1.fc38.x86_64 mesa-libxatracker-23.0.3-1.fc38.x86_64 mesa-libglapi-23.0.3-1.fc38.i686 mesa-libgbm-23.0.3-1.fc38.i686 mesa-libEGL-23.0.3-1.fc38.i686 mesa-libGL-23.0.3-1.fc38.i686 mesa-libOSMesa-23.0.3-1.fc38.i686 mesa-vulkan-drivers-23.0.3-1.fc38.i686 mesa-filesystem-23.0.3-1.fc38.x86_64 mesa-filesystem-23.0.3-1.fc38.i686 mesa-dri-drivers-23.0.3-1.fc38.x86_64 mesa-dri-drivers-23.0.3-1.fc38.i686 [felix@x1 ~]$ sudo dnf downgrade mesa-dri-drivers Last metadata expiration check: 0:04:04 ago on Thu 04 May 2023 08:44:19 AM. Dependencies resolved. ================================================================================ Package Architecture Version Repository Size ================================================================================ Downgrading: mesa-dri-drivers i686 23.0.1-2.fc38 fedora 20 M mesa-dri-drivers x86_64 23.0.1-2.fc38 fedora 18 M mesa-filesystem i686 23.0.1-2.fc38 fedora 18 k mesa-filesystem x86_64 23.0.1-2.fc38 fedora 18 k Transaction Summary ================================================================================ Downgrade 4 Packages
(In reply to Felix Kaechele from comment #2) > The depsolver will fall back to > whichever available mesa-libEGL it can find that satisfies all constraints > of the transaction. And mesa-libEGL-23.0.1-2.fc38.x86_64 in the base repo > will always be happy to oblige ;-) The question is *why* is this happening, in which cases it happens, so that we can try to do something about it.
Mesa 23.0.3-3 has been unfortunately pushed stable overnight, so the reproducer no longer works as described. But a similar issue can be seen when we use the default package versions and just try to install gnome-calculator. This will install mesa-libEGL 23.0.3-3 but keep mesa-dri-drivers at 23.0.1-1. gnome-calculator will then fail to start due to version mismatch. $ rpm -qa | grep mesa | sort mesa-dri-drivers-23.0.1-1.fc38.x86_64 mesa-filesystem-23.0.1-1.fc38.x86_64 mesa-libglapi-23.0.1-1.fc38.x86_64 mesa-va-drivers-23.0.1-1.fc38.x86_64 mesa-vulkan-drivers-23.0.1-1.fc38.x86_64 $ sudo dnf install gnome-calculator --assumeno | grep mesa mesa-libglapi x86_64 23.0.3-3.fc38 updates 54 k mesa-libEGL x86_64 23.0.3-3.fc38 updates 130 k mesa-libGL x86_64 23.0.3-3.fc38 updates 173 k mesa-libgbm x86_64 23.0.3-3.fc38 updates 45 k Operation aborted.
(In reply to Kamil Páral from comment #3) > The question is *why* is this happening, in which cases it happens, so that > we can try to do something about it. The "why" is probably "because it can". Even when I try a similar situation with `dnf --best`, it still picks older versions. And in `man dnf`, this is explicitly stated under --best documentation: -b, --best ... Note that the use of the newest available version is only guaranteed for the packages directly requested (e.g. as a command line arguments), and the solver may use older versions of dependencies to meet their requirements. I can only speculate that it's prioritizing the smallest transaction possible. So if installing older mesa-libEGL causes fewer packages to be installed then installing newer mesa-libGL (because that would cause an update for all mesa-* subpackages), it picks the former. This is further backup up by the fact, that when I replace `sudo dnf install gnome-calculator` with `sudo dnf install gnome-calculator mesa-libEGL`, it picks up the latest versions instead. That matches the documentation. So, this is unfortunate. But, I think we can still fix this. I'm currently working on a potential solution and testing it, I'll post it once ready.
This is my attempt to solve this situation: https://src.fedoraproject.org/rpms/mesa/pull-request/24 Please note that I only now noticed that the f38 toolbox image contains mesa 23.0.1-1, while the base 'fedora' repo contains mesa 23.0.1-2 (built against llvm 16, on top of that). This sadly further changes the dnf output when testing things. But I decided to ignore this issue, it's a bug in toolbox OCI (it needs to be regenerated using final rpms), and it's not the cause of these dependency issues, it just helped to uncover it.
Adam, or some other Mesa maintainer, can you please look at the PR in comment 6? Thanks!
I went ahead and merged this. Thanks!
FEDORA-2023-a20a26a598 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a20a26a598
FEDORA-2023-84965ba750 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-84965ba750
FEDORA-2023-84965ba750 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-84965ba750` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-84965ba750 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-a20a26a598 has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-a20a26a598` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-a20a26a598 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
(In reply to Pete Walter from comment #8) > I went ahead and merged this. Thanks! Thanks Pete! Sorry, the PR description wasn't completely clear, but a part of this also needs to go to Rawhide. I created another PR in https://src.fedoraproject.org/rpms/mesa/pull-request/25 .
FEDORA-2023-a20a26a598 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.
Reopening for Rawhide (comment 13).
Pete, could you please also merge the PR from comment 13?
(In reply to Kamil Páral from comment #16) > Pete, could you please also merge the PR from comment 13? This is now merged, closing. Thanks Igor.
Thanks!
FEDORA-2023-5a0786b714 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-5a0786b714
FEDORA-2023-5a0786b714 has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.