|Summary:||Provide broken/missing group package behaviour identical to yum's (accept missing packages, fail on broken packages)|
|Product:||[Fedora] Fedora||Reporter:||Adam Williamson <awilliam>|
|Component:||dnf||Assignee:||Daniel Mach <dmach>|
|Status:||CLOSED EOL||QA Contact:||Fedora Extras Quality Assurance <extras-qa>|
|Version:||26||CC:||dennis, dmach, jkonecny, kevin, mcepl, packaging-team-maint, rpm-software-management, samuel-rhbugs, vmukhame|
|Fixed In Version:||Doc Type:||If docs needed, set a value|
|Doc Text:||Story Points:||---|
|Last Closed:||2018-05-29 12:00:51 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
Description Adam Williamson 2017-06-14 17:50:01 UTC
See also: https://bugzilla.redhat.com/show_bug.cgi?id=1427365 https://bugzilla.redhat.com/show_bug.cgi?id=1303978 https://bugzilla.redhat.com/show_bug.cgi?id=1303991 We've been around the houses a few times on expected behaviour when trying to install a package group which includes: i) packages that don't exist ii) packages that exist, but cannot be installed for some reason but in #1427365 , we think we pinned down what we really want (at least in the short term). This is particularly regarding the compose process, which relies on package groups. We believe that 'classic' yum / anaconda behaviour when installing package groups during the compose process was this: * If a package name listed in the group can't be found at all, accept this, carry on (not sure if any kind of warning/informational message was shown in various circumstances) * If a package name listed in the group can be found but can't be installed due to dependency issues etc., do not accept this, fail It seems that we've never quite replicated this behaviour in DNF. We've ping ponged between "accept both missing packages and broken packages" and "fail on both missing packages and broken packages", but never split them in the way yum did. Of course, there's no obvious from-first-principles justification for yum's behaviour here being "correct". But we do have a *practical* reason for wanting that specific behaviour: the compose process rather expects it. Specifically, there are several occasions where a package is included in a comps group that only exists on certain arches (e.g. bootloader packages that are only relevant on certain arches - say, grub2-efi is listed in the important 'anaconda-tools' group, but it does not exist at all on several arches, including armhfp). When using that comps group on other arches we do *not* want dnf to bail out and break the compose. This is why we don't want 'package does not exist at all' to be a fatal error in this workflow. But we *do* want the compose to fail if a package exists but cannot be installed due to dependency issues, otherwise we can wind up with situations where the compose apparently 'succeeded' but in fact important packages are missing from the media (this actually happened a couple of times already in the F26 cycle). We *have* made a change to comps which allows us to specify packages by arch, so we could in theory adjust comps such that the 'package named in group isn't available at all' situation never happens, but it'd probably involve quite a lot of trial and error to get to that point, eating up quite a lot of someone's time. So at least until we can spare a person to do that and verify it's correct, it'd be nice if we could have some way to make dnf behave the way yum did, during composes at least.
Comment 1 Matěj Cepl 2017-09-04 10:17:35 UTC
This is also quite useful when porting package to new platform. E.g., I was porting OpenQA to EPEL7 (resulting in 88 packages in https://copr.fedorainfracloud.org/coprs/mcepl/openQA-EPEL7/ …). Then it is absolutely essential that I am able to run yum install long-list-of-required-packages with some of them failing to be installed (yum then reports "Package foo cannot be found" or something). yum-builddep behaved same as dnf now and that made it absolutely useless for me. This doesn't have to be a default behaviour, just something which could be achieved optionally, but without that my life would be a way more hard.
Comment 2 Adam Williamson 2018-03-10 08:32:58 UTC
So I've spent the last two goddamned hours just trying to even *remotely* understand how this stuff actually works in dnf, with somewhat limited success. Does it really have to be this...baroque? Anyway, where I have finally pitched up - by reverse engineering the CLI, since trying to figure this out from the other end just pitched me up in a gigantic twisted pile of indirection, C, and sadness - is here: https://github.com/rpm-software-management/dnf/blob/master/dnf/cli/output.py#L1030 which appears to be the bit which causes the CLI to log when a package is skipped from a transaction because of broken dependencies. I got *there* by actually doing it, in current F28: sudo dnf --disablerepo=* --enablerepo=fedora groupinstall 3d-printing which tells us: Skipping packages with broken dependencies: blender x86_64 1:2.79-6.fc28 fedora 35 M 'blender' is a default (not mandatory) package in the 3d-printing group. that's *basically* the kind of behaviour we want to make available to anaconda: be aware of when a package that otherwise would have been included in a transaction was left out because of broken dependencies. But looking at that code in dnf...how is anaconda supposed to do that? The dnf code uses a private attribute of the base, `_goal` - presumably dnf shouldn't do that. So what can it do instead?
Comment 3 Adam Williamson 2018-03-10 08:35:06 UTC
Note that oddly enough, doing sudo dnf --disablerepo=* --enablerepo=fedora --best groupinstall 3d-printing causes dnf to *silently* omit blender from the transaction. At this point I'm not entirely sure what I expected adding '--best' to do, but it probably wasn't that.
Comment 4 Igor Gnatenko 2018-03-10 08:37:42 UTC
So IMO all this behavior is stupid. Checking if package exists and then installing it is _S L O W_. We should get right behavior by _installing_ mandatory packages no matter what.
Comment 5 Adam Williamson 2018-03-10 09:11:03 UTC
1. We still can't do something quite that simple because we know there are cases where packages are listed as mandatory that do not in fact exist at all. yum's historic behaviour was that 'package is listed but does not exist at all' was *never* fatal - no matter whether it was mandatory or default. So these cases exist. I'd like to clean them up, but we need to do that *before* making mandatory packages unavoidably 100% mandatory. 2. This isn't just about mandatory packages. yum used to fail on dependency error *for default packages too*. Not just mandatory. yum's behaviour was this: * 'package listed in group but doesn't exist' was always a skip * 'package listed in group and would be included in transaction, but has dependency errors' was always an error, *regardless of type* The 'type' *didn't matter* to yum in deciding whether there was an error or not. It was only significant in deciding which packages to try and install at all. The historic difference between 'default' and 'mandatory' was not "fail on mandatory, don't fail on default", but to do with *package selection tools*, specifically the old anaconda UI. When you selected a package group in the old anaconda UI, all 'mandatory' and 'default' packages in it were initially selected for install...but you could then *de*-select 'default' packages. You could *not* de-select mandatory packages. *That* was the difference and the initial reason why these 'types' exist at all. 'optional' packages were not selected by default, but were shown under the group and you could then *select* them if you chose. Once you'd made your choices, though, if *any* of the packages that ultimately wound up being selected had dependency errors, the install would fail. This is the behaviour dnf has never quite had, whatever permutations we've been through over the years.
Comment 6 Adam Williamson 2018-03-10 09:13:44 UTC
To be super clear: the fact that dnf happily skips default packages with dependency issues *is a problem*. Just recently, it's led to gnome-software being left off GNOME images, and cinnamon being left off Cinnamon images. I'm currently sending PRs to make some packages 'mandatory' in comps that probably should be mandatory, but there are some cases where it's not really correct to make the package mandatory in comps, but image composes should still probably fail if the package has dependency errors.
Comment 7 Adam Williamson 2018-03-13 22:58:08 UTC
I have a three-line patch (and a 100-line commit message...) that I think should actually do what we want here. Testing it now.
Comment 8 Adam Williamson 2018-03-14 01:54:39 UTC
Comment 9 Fedora End Of Life 2018-05-03 08:20:02 UTC
This message is a reminder that Fedora 26 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '26'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 26 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Comment 10 Fedora End Of Life 2018-05-29 12:00:51 UTC
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 11 Adam Williamson 2022-09-02 20:22:13 UTC
"We *have* made a change to comps which allows us to specify packages by arch, so we could in theory adjust comps such that the 'package named in group isn't available at all' situation never happens, but it'd probably involve quite a lot of trial and error to get to that point, eating up quite a lot of someone's time. So at least until we can spare a person to do that and verify it's correct, it'd be nice if we could have some way to make dnf behave the way yum did, during composes at least." Well, it took a few years, but I got there ;) https://pagure.io/fedora-comps/pull-request/767 https://github.com/rpm-software-management/dnf/pull/1848