Hi, Please see this[1] post from the `F36 Change: Enable exclude_from_weak_autodetect by default in LIBDNF (System-Wide Change proposal)` mailing list thread. Kamil Paral instructed me to create a bug blocking the change's bug [2]. [1]: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ERPOO3YGBHUJAMNXCEIUXGGH6LWEA7LN/ [2]: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/HAQ62Q56DCVRNSL5GIE74C5SLIX3ARHS/ Here is the text of my post: Hi everyone, I am resending this message, because I think it got lost. I sent it at the begining of a weekend, so people must not have seen it. I am CC'ing the change owner, as I feel that more clarification is required. I maintain that this change should only apply to updates; `dnf install`, `dnf reinstall` should behave as they have been. At least, this change and all of its effects should be fully explained to packagers. We should probably discuss the shell completion stuff seperately. Thanks, Maxwell On Fri, 2021-11-12 at 15:48 -0600, Maxwell G wrote: > Hi everyone, > > On Thu, 2021-09-16 at 15:17 -0400, Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/ExcludeFromWeakAutodetect > > > > > > == Summary == > > exclude_from_weak_autodetect enables autodetection of unmet weak > > dependencies (Recommends or Supplements) of installed packages and > > blocks installation of packages satisfying already unmet > > dependencies. > > In other words: When you don't have the recommended package > > installed, > > it won't be automatically installed with future upgrades of the > > recommending package. > > I am not sure if this was intended, but this change has broken rich > weak dependencies when both packages are not installed as part of the > same transaction. > > In my yt-dlp package's specfile[1], I have three subpackages for > shell > completions: `yt-dlp-bash-completion`, `yt-dlp-zsh-completion`, and > `yt-dlp-fish-completion. Here is the `bash-completion` block: > > ``` spec > %package bash-completion > Summary: Bash completion for %{name} > Requires: %{name} = %{version} > Requires: bash-completion > Supplements: (%{name} and bash-completion) > BuildArch: noarch > ``` > > The intended effect is for the shell completions to be installed at > the > time `yt-dlp` itself is installed if the respective shell package > (`bash-completion`, `zsh` or `fish`) is already present while still > allowing users to opt out. However, now this does not work; dnf will > only install the completions if both `yt-dlp` and the shell package > are > installed as part of the same transaction. I can confirm that this is > caused by this change, because adding `-- > setopt=exclude_from_weak_autodetect=false` fixes the problem. > Replacing > `Supplements` with forward facing boolean `Requires` did not work > either. > > ``` spec > Recommends: (%{name}-bash-completion if bash-completion) > Recommends: (%{name}-zsh-completion if zsh) > Recommends: (%{name}-fish-completion if fish) > ``` > > While I agree that {rich,} weak dependencies should not be > reinstalled > as part of updates, I do believe that they should be installed if one > of the packages is being installed for the first time. > > I also think we should consider implementing better guidelines for > shell completions in Fedora. I believe that shell completions should > be > split into subpackages and that these subpackages should depend on > the > shells themselves or a `-filesystem` package that actually own the > directories. Right now, directory ownership is kind of a mess. At > least > on my system, there are several packages that own /usr/share/bash- > completion, /usr/share/zsh/vendor-completions, /usr/share/zsh/site- > functions, and /usr/share/fish/vendor_completions.d/. We can also use > this oppurtunity to create macros for each of these directories. > > Management of shell completion packages was discussed further in my > package review ticket [2]. > > I am relatively new to Fedora, so please correct me if I got anything > wrong. > > Thanks, > Maxwell > > [1]: > https://src.fedoraproject.org/rpms/yt-dlp/blob/rawhide/f/yt-dlp.spec > [2]: https://bugzilla.redhat.com/show_bug.cgi?id=2012522 >
Thank you very much for the report. I know the implementation is not perfect due to technical difficulties. The mechanism cannot be applied differently for upgrade, downgrade, reinstall and install operation because DNF does all of them together and the rule can be only applyed to all operations. I also know that what is a beneficial for some one is an issue for another user. In case that autodetectios does not work, it can be switched off (from commandline --setopt=exclude_from_weak_autodetect=false on permanently in /etc/dnf/dnf.con) and maintained manually by user using `exclude_from_weak` configuration option. I am really not sure how we can help. What do you think?
(In reply to Jaroslav Mracek from comment #1) > Thank you very much for the report. I know the implementation is not perfect > due to technical difficulties. > > The mechanism cannot be applied differently for upgrade, downgrade, > reinstall and install operation because DNF does all of them together and > the rule can be only applyed to all operations. I also know that what is a > beneficial for some one is an issue for another user. > > In case that autodetectios does not work, it can be switched off (from > commandline --setopt=exclude_from_weak_autodetect=false on permanently in > /etc/dnf/dnf.con) and maintained manually by user using `exclude_from_weak` > configuration option. > > I am really not sure how we can help. What do you think? At least from the user side, isn't it possible to put `exclude_from_weak_autodetect=false` and then run `dnf update --setopt=exclude_from_weak_autodetect=true`? If this really can't be changed, then I think the change proposal needs to be updated and reconsidered. The change description makes it sound like this only applies to updates which is apparently not the case. Here[1] is a list of at least some of the packages that will be broken by this change. Also, it looks like you assigned the bug to yourself but didn't change its status. [1]: https://sourcegraph.com/search?q=context:global+r:src.fedoraproject.org+file:.*%5C.spec%24+%5ERecommends:%5Cs%2B%5C%28.*+if+.*%5C%29&patternType=regexp&case=yes
(In reply to Maxwell G from comment #2) > (In reply to Jaroslav Mracek from comment #1) > > Thank you very much for the report. I know the implementation is not perfect > > due to technical difficulties. > > > > The mechanism cannot be applied differently for upgrade, downgrade, > > reinstall and install operation because DNF does all of them together and > > the rule can be only applyed to all operations. I also know that what is a > > beneficial for some one is an issue for another user. > > > > In case that autodetectios does not work, it can be switched off (from > > commandline --setopt=exclude_from_weak_autodetect=false on permanently in > > /etc/dnf/dnf.con) and maintained manually by user using `exclude_from_weak` > > configuration option. > > > > I am really not sure how we can help. What do you think? > > At least from the user side, isn't it possible to put > `exclude_from_weak_autodetect=false` and then run `dnf update > --setopt=exclude_from_weak_autodetect=true`? If this really can't be > changed, then I think the change proposal needs to be updated and > reconsidered. The change description makes it sound like this only applies > to updates which is apparently not the case. Here[1] is a list of at least > some of the packages that will be broken by this change. > > Also, it looks like you assigned the bug to yourself but didn't change its > status. > > [1]: > https://sourcegraph.com/search?q=context:global+r:src.fedoraproject.org+file: > .*%5C.spec%24+%5ERecommends:%5Cs%2B%5C%28.*+if+. > *%5C%29&patternType=regexp&case=yes To clarify, I meant that it was possible to put `exclude_from_weak_autodetect=false` *in dnf.conf* and then override that on the command-line when running `dnf update`.
A related issue was reported on the RHEL side of things[1]. After skimming through the mountain that is rhbz#1699672, I understand that this is a very complicated issue to solve (and it seems like this rich weak dep situation was not fully considered). My previous regex `^Recommends:\s+\(.* if .*\)` found 29 affected packages. However, boolean reverse weak dependencies `Supplements: (foo and bar)` are much more common. Searching for both '^Supplements:\s+\(.* and .*\)' and '^Recommends:\s+\(.* if .*\)' found 874 affected packages! Thanks, Maxwell [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2048394
This seems to be in all? ghc-*.spec files: %package prof Summary: Haskell %{pkg_name} profiling library Requires: %{name}-devel%{?_isa} = %{version}-%{release} Supplements: (%{name}-devel and ghc-prof) It is in https://docs.fedoraproject.org/en-US/packaging-guidelines/Haskell/#_spec_file_templates
It looks like that the feature made a user case with langpacks broken. See additional reports: Bug 2048394 - dnf should pull weak dependencies in install transaction Bug 2033130 - exclude_from_weak_autodetect=true effectively renders rich weak dependencies useless Bug 2042808 - weakdeps not working on rawhide system The feature was requested: Bug 2005305 - dnf should not pull (already unmet) weak dependencies on updates Bug 1699672 - RFE: dnf should not pull (already broken) weak dependencies on updates Firs of all I need to clarify that the feature cannot be implemented only on upgrades - because there are technical reasons for that - 1. DNF creates one transaction for all operations (install, upgrades are performed together). 2.a Install operation or commands (not only install) also triggers update. (example - I have already installed foo-1-1.noarch. Then I will install bar-2-2.noarch that requires foo-2. It means the install command will trigger upgrade that dnf cannot detect in advance. And if foo recommends something, it will be installed) 2.b Install operation with --best (default in RHEL) triggers always upgrade when package is already installed but in lower version. Be honest I do not know what to do. Basically I see only 3 option with one additional: 1. Keep it like it is 2. Disable autodetection 3. Start to ignore rich dependencies for autodetection of unmet weak dependencies. We have a problem to detect rich dependencies correctly in autodetections because we do not know whether their conditions were met or not in past. 4. In theory the auto-detection can be only triggered by upgrade command but it will create an inconsistency in DNF behavior when upgrade operation is triggered by the another command (install, buildeps, downgrade, ...) - not preferable, see above.
*** Bug 2042808 has been marked as a duplicate of this bug. ***
Please correct me if I'm wrong, but doing the autodetection on upgrade only would not be a 100% solution anyway. On the langpacks example, IIUC, if someone adds localization support to a new version of an existing package, you also want the localization for that package to be installed for all installed langpacks. Granted this is an order of magnitude less common than fresh installations, but let's aim for a robust solution if any is possible at all in this situation. I have a suggestion coming form a different angle: There are two levels of weak dependencies, "weak" and "very weak": https://rpm-software-management.github.io/rpm/manual/dependencies.html#weak-dependencies. The link actually describes "weak" as: "By default the dependency solver shall attempt to process the dependency as though it were strong. If this is results in an error then they should be ignored and not trigger an error or warning." And the "very weak": "By default the dependency solver shall ignore them. But they may be used to show the matching packages as option to the user." With the introduction of autodetection, these definitions are actually no longer accurate. How about changing the semantics a bit. Make the "weak" deps adhere to the aforementioned definition, meaning they will be treated as strong unless there's an error. And make the "very weak" work with the autodetection by default, with a config option along the lines of install_very_weak_deps={always,auto,never}. This would imply packaging updates for all packages that use weak deps, to update all "weak" deps that are supposed to be really optional to "very weak". The current semantics of "very weak" deps (which we would lose with this change) always seemed not very useful to me, but that might just be my perception. The above would allow the langpacks use case and any other non-trivial, rich dependency based resolution to work reliably and unimpeded. It would also (hopefully) allow a reasonable autodetection for really optional dependencies.
> How about changing the semantics a bit. Make the "weak" deps adhere to the aforementioned definition, meaning they will be treated as strong unless there's an error. And make the "very weak" work with the autodetection by default, with a config option along the lines of install_very_weak_deps={always,auto,never}. -1. This solution is in effect similar to the other previously proposed ideas to make install and upgrade behave differently. Essentially, it binds the "exclude on upgrades" with something that is orthogonal and should remain independent. (Before, what verb was used. In your proposal, the exact dependency type.) The same semantics on upgrade should apply to various weak dependency types, except where they differ by design.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #9) > This solution is in effect similar to the other previously proposed ideas to make install and upgrade behave differently. No, my suggestion doesn't differentiate install and upgrade in any way. Not sure whether you misunderstood. > The same semantics on upgrade should apply to various weak dependency types, except where they differ by design. It seems to me there are different requirements on the weak dependencies. There are at least the langpacks where the dependencies are expected to be strong (while rich and in a reverse direction (Supplements)). Then there are the run-of-the-mill weak deps which people expect to have the choice to not install (and, might I add, a lot of those people expect dnf to somehow know what is their particular intention with a particular dep in a particular situation...).
(In reply to Lukáš Hrázký from comment #10) > (In reply to Zbigniew Jędrzejewski-Szmek from comment #9) > > This solution is in effect similar to the other previously proposed ideas to make install and upgrade behave differently. > > No, my suggestion doesn't differentiate install and upgrade in any way. Not > sure whether you misunderstood. What I meant is that you are making a similar kind of conceptual mistake: connecting two things which should be independent. > > The same semantics on upgrade should apply to various weak dependency types, except where they differ by design. > > It seems to me there are different requirements on the weak dependencies. The requirements *are* of course a bit different, because the semantics of different weak dep types are different. But the property being discussed here (not trying to satisfy the dep in certain cases) cuts across those dependency types and applies to all dependency types. In particular, we cwant it to apply to Recommends, i.e. the strongest weak dependency type. And if it applies to Recommends, it should also apply to the weaker dependency types.
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle. Changing version to 36.
Proposed as a Blocker for 36-beta by Fedora user bcotton using the blocker tracking app because: Appears to violate https://fedoraproject.org/wiki/Basic_Release_Criteria#software-install-remove-update Specifically, it causes dnf to NOT "choose the software to be installed, updated or removed in ways that are broadly in line with the user's intent and typical expectations, and the project's intent as to which software should be provided"
Discussed during the 2022-02-14 blocker review meeting: [0] The decision to classify this bug as an "AcceptedBlocker (Beta)" was made as it violates the following criterion: "The installed system must be able appropriately to install, remove, and update software...Appropriately means...broadly in line with the user's intent and typical expectations, and the project's intent as to which software should be provided from which repositories etc." [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-02-14/f36-blocker-review.2022-02-14-17.01.txt
I have a patch that improves behavior of autodetection - https://github.com/rpm-software-management/libdnf/pull/1439 For testing purpose I created a repository - https://copr.fedorainfracloud.org/coprs/jmracek/unmet-weakdeps-hotfix/.
Hmm, can you build the full dnf stack in the copr? libdnf failed to build, and we also need dnf-plugins-core to test this locally.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #16) > Hmm, can you build the full dnf stack in the copr? libdnf failed to build, > and we also need dnf-plugins-core > to test this locally. I am sorry, I provided a build with additional change that requires additional updates (taken form upstream). The patch does not require any additional update therefore I created a new build of libdnf that should be sufficient.
So far we do not have any negative reaction to the patch therefore we have released it in in libdnf-0.65.0-3. See https://bodhi.fedoraproject.org/updates/FEDORA-2022-f1febfd9b6. Please feel free to reopen the bug if the patch does not resolve the reported issue.
Bah. This is a blocker and there was no update associated with it. Changing to ON_QA so QA can verify.
The update was pushed stable already, so if this addresses concerns, we're good. If anyone can check and confirm that things seem to work better now it'd be great.
I tested the example in the description of this bug, and with get the expected behaviour: yt-dlp-zsh-completion if I have zsh, and yt-dlp-fish-completion if I have fish. For langpacks, I also seem to get the correct behaviour: $ sudo dnf install langpacks-fr libreoffice-writer --setopt=exclude_from_weak_autodetect=true --setopt=install_weak_deps=true ... Installing weak dependencies: glibc-langpack-en x86_64 2.35-2.fc37 rawhide 693 k glibc-langpack-fr x86_64 2.35-2.fc37 rawhide 706 k hunspell-en noarch 0.20140811.1-22.fc36 rawhide 180 k hunspell-fr noarch 6.2-9.fc36 rawhide 302 k hyphen-fr noarch 3.0-10.fc36 rawhide 17 k langpacks-en noarch 3.0-21.fc36 rawhide 9.6 k libreoffice-help-en x86_64 1:7.3.0.3-3.fc36 rawhide 2.9 M libreoffice-help-fr x86_64 1:7.3.0.3-3.fc36 rawhide 3.6 M libreoffice-langpack-fr x86_64 1:7.3.0.3-3.fc36 rawhide 899 k man-pages-fr noarch 4.12.1-2.fc36 rawhide 4.1 M ... (The installation set is the same with exclude_from_weak_autodetect=true/false.) Tested with: dnf-4.10.0-2.fc36.noarch libdnf-0.65.0-3.fc37.x86_64
OK, let me also check how does that correspond to the original https://fedoraproject.org/wiki/Changes/ExcludeFromWeakAutodetect [root@9e550997c16b /]# rpm -q libdnf libdnf-0.65.0-3.fc36.x86_64 [root@9e550997c16b /]# dnf install tox ... Installing weak dependencies: libxcrypt-compat x86_64 4.4.28-1.fc36 fedora 90 k pypy-devel x86_64 7.3.6-2.fc36 fedora 67 k pypy3.8-devel x86_64 7.3.7-3.fc36 fedora 61 k python2.7 x86_64 2.7.18-19.fc36 fedora 13 M python3-devel x86_64 3.10.2-3.fc36 fedora 207 k python3-pip noarch 21.3.1-2.fc36 fedora 1.8 M python3.11 x86_64 3.11.0~a5-1.fc36 fedora 24 M python3.6 x86_64 3.6.15-6.fc36 fedora 16 M python3.7 x86_64 3.7.12-8.fc36 fedora 17 M python3.8 x86_64 3.8.12-6.fc36 fedora 16 M python3.9 x86_64 3.9.10-2.fc36 fedora 18 M Is this ok [y/N]: N [root@9e550997c16b /]# dnf install tox --setopt=install_weak_deps=False ... Installed: ... tox-3.24.5-2.fc36.noarch ... [root@9e550997c16b /]# dnf reinstall tox [root@9e550997c16b /]# dnf downgrade tox (build from Koji) [root@9e550997c16b /]# dnf upgrade tox [root@9e550997c16b /]# dnf upgrade None of the installs Pythons. On Fedora 35, they do. So this looks correct. Thanks.
Thanks a lot for the testing, folks. I'm sure someone will find something else broken, but I guess for now we can call this fixed.