Bug 1461539 - Provide broken/missing group package behaviour identical to yum's (accept missing packages, fail on broken packages)
Provide broken/missing group package behaviour identical to yum's (accept mis...
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: dnf (Show other bugs)
26
All Linux
unspecified Severity medium
: ---
: ---
Assigned To: Daniel Mach
Fedora Extras Quality Assurance
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-14 13:50 EDT by Adam Williamson
Modified: 2018-05-29 08:00 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-05-29 08:00:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Adam Williamson 2017-06-14 13:50:01 EDT
See also:

https://bugzilla.redhat.com/show_bug.cgi?id=1427365
https://bugzilla.redhat.com/show_bug.cgi?id=1303978
https://bugzilla.redhat.com/show_bug.cgi?id=1303991

We've been around the houses a few times on expected behaviour when trying to install a package group which includes:

i) packages that don't exist
ii) packages that exist, but cannot be installed for some reason

but in #1427365 , we think we pinned down what we really want (at least in the short term). This is particularly regarding the compose process, which relies on package groups.

We believe that 'classic' yum / anaconda behaviour when installing package groups during the compose process was this:

* If a package name listed in the group can't be found at all, accept this, carry on (not sure if any kind of warning/informational message was shown in various circumstances)

* If a package name listed in the group can be found but can't be installed due to dependency issues etc., do not accept this, fail

It seems that we've never quite replicated this behaviour in DNF. We've ping ponged between "accept both missing packages and broken packages" and "fail on both missing packages and broken packages", but never split them in the way yum did.

Of course, there's no obvious from-first-principles justification for yum's behaviour here being "correct". But we do have a *practical* reason for wanting that specific behaviour: the compose process rather expects it. Specifically, there are several occasions where a package is included in a comps group that only exists on certain arches (e.g. bootloader packages that are only relevant on certain arches - say, grub2-efi is listed in the important 'anaconda-tools' group, but it does not exist at all on several arches, including armhfp).

When using that comps group on other arches we do *not* want dnf to bail out and break the compose. This is why we don't want 'package does not exist at all' to be a fatal error in this workflow. But we *do* want the compose to fail if a package exists but cannot be installed due to dependency issues, otherwise we can wind up with situations where the compose apparently 'succeeded' but in fact important packages are missing from the media (this actually happened a couple of times already in the F26 cycle).

We *have* made a change to comps which allows us to specify packages by arch, so we could in theory adjust comps such that the 'package named in group isn't available at all' situation never happens, but it'd probably involve quite a lot of trial and error to get to that point, eating up quite a lot of someone's time. So at least until we can spare a person to do that and verify it's correct, it'd be nice if we could have some way to make dnf behave the way yum did, during composes at least.
Comment 1 Matěj Cepl 2017-09-04 06:17:35 EDT
This is also quite useful when porting package to new platform. E.g., I was porting OpenQA to EPEL7 (resulting in 88 packages in https://copr.fedorainfracloud.org/coprs/mcepl/openQA-EPEL7/ …). Then it is absolutely essential that I am able to run yum install long-list-of-required-packages with some of them failing to be installed (yum then reports "Package foo cannot be found" or something). yum-builddep behaved same as dnf now and that made it absolutely useless for me.

This doesn't have to be a default behaviour, just something which could be achieved optionally, but without that my life would be a way more hard.
Comment 2 Adam Williamson 2018-03-10 03:32:58 EST
So I've spent the last two goddamned hours just trying to even *remotely* understand how this stuff actually works in dnf, with somewhat limited success. Does it really have to be this...baroque?

Anyway, where I have finally pitched up - by reverse engineering the CLI, since trying to figure this out from the other end just pitched me up in a gigantic twisted pile of indirection, C, and sadness - is here:

https://github.com/rpm-software-management/dnf/blob/master/dnf/cli/output.py#L1030

which appears to be the bit which causes the CLI to log when a package is skipped from a transaction because of broken dependencies. I got *there* by actually doing it, in current F28:

sudo dnf --disablerepo=* --enablerepo=fedora groupinstall 3d-printing

which tells us:

Skipping packages with broken dependencies:
 blender                          x86_64          1:2.79-6.fc28                                 fedora           35 M

'blender' is a default (not mandatory) package in the 3d-printing group.

that's *basically* the kind of behaviour we want to make available to anaconda: be aware of when a package that otherwise would have been included in a transaction was left out because of broken dependencies. But looking at that code in dnf...how is anaconda supposed to do that? The dnf code uses a private attribute of the base, `_goal` - presumably dnf shouldn't do that. So what can it do instead?
Comment 3 Adam Williamson 2018-03-10 03:35:06 EST
Note that oddly enough, doing sudo dnf --disablerepo=* --enablerepo=fedora --best groupinstall 3d-printing causes dnf to *silently* omit blender from the transaction. At this point I'm not entirely sure what I expected adding '--best' to do, but it probably wasn't that.
Comment 4 Igor Gnatenko 2018-03-10 03:37:42 EST
So IMO all this behavior is stupid.

Checking if package exists and then installing it is _S L O W_.

We should get right behavior by _installing_ mandatory packages no matter what.
Comment 5 Adam Williamson 2018-03-10 04:11:03 EST
1. We still can't do something quite that simple because we know there are cases where packages are listed as mandatory that do not in fact exist at all. yum's historic behaviour was that 'package is listed but does not exist at all' was *never* fatal - no matter whether it was mandatory or default. So these cases exist. I'd like to clean them up, but we need to do that *before* making mandatory packages unavoidably 100% mandatory.

2. This isn't just about mandatory packages. yum used to fail on dependency error *for default packages too*. Not just mandatory. yum's behaviour was this:

* 'package listed in group but doesn't exist' was always a skip
* 'package listed in group and would be included in transaction, but has dependency errors' was always an error, *regardless of type*

The 'type' *didn't matter* to yum in deciding whether there was an error or not. It was only significant in deciding which packages to try and install at all. The historic difference between 'default' and 'mandatory' was not "fail on mandatory, don't fail on default", but to do with *package selection tools*, specifically the old anaconda UI. When you selected a package group in the old anaconda UI, all 'mandatory' and 'default' packages in it were initially selected for install...but you could then *de*-select 'default' packages. You could *not* de-select mandatory packages. *That* was the difference and the initial reason why these 'types' exist at all. 'optional' packages were not selected by default, but were shown under the group and you could then *select* them if you chose. Once you'd made your choices, though, if *any* of the packages that ultimately wound up being selected had dependency errors, the install would fail.

This is the behaviour dnf has never quite had, whatever permutations we've been through over the years.
Comment 6 Adam Williamson 2018-03-10 04:13:44 EST
To be super clear: the fact that dnf happily skips default packages with dependency issues *is a problem*. Just recently, it's led to gnome-software being left off GNOME images, and cinnamon being left off Cinnamon images.

I'm currently sending PRs to make some packages 'mandatory' in comps that probably should be mandatory, but there are some cases where it's not really correct to make the package mandatory in comps, but image composes should still probably fail if the package has dependency errors.
Comment 7 Adam Williamson 2018-03-13 18:58:08 EDT
I have a three-line patch (and a 100-line commit message...) that I think should actually do what we want here. Testing it now.
Comment 9 Fedora End Of Life 2018-05-03 04:20:02 EDT
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Comment 10 Fedora End Of Life 2018-05-29 08:00:51 EDT
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26
is no longer maintained, which means that it will not receive any
further security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.