While testing installs for the F22 Anaconda DNF Test Day today, we noticed something odd: KDE installs wound up with two kernels, both kernel-core and kernel-debug-core . After some investigation I was able to identify the issue. It boils down to another case of DNF not handling an ambiguous dependency situation as well as yum. The offending package is 'xl2tpd', which is pulled into the kde-desktop somehow (I didn't bother tracking down if it's explicitly included or just comes in as a dep of something else). xl2tpd requires "kmod(l2tp_ppp.ko)". That is provided by all kernel -modules-extra packages. When you run 'yum install xl2tpd' without a kernel installed (e.g. from a mock chroot), yum selects kernel-modules-extra to provide it. When you run 'dnf install xl2tpd', DNF selects kernel-debug-modules-extra . At install time, if your package set happens to have xl2tpd in it, when anaconda first resolves the deps of the packages selected for install, 'kernel-debug-core' is pulled in to satisfy xl2tpd. Later, anaconda runs a method to ensure the 'preferred' kernel is installed, dnfpayload.py's DNFPayload._select_kernel_package() . The logic of this method is (and has been for years, it's pretty much identical to the yumpayload one): 1. If 'kernel' is explicitly excluded figure the user knows what they're doing and don't install anything 2. If 'kernel-PAE' is available, install it 3. Otherwise if we're on ARM and 'kernel-(platform)' or 'kernel-lpae' is available, install it 4. Otherwise install 'kernel' So the upshot is, that if your package set has xl2tpd in it, and the kernel currently in the repos is a 'production' build (so kernel-debug exists), you get two kernels installed: kernel-core (as a dep of 'kernel') and kernel-debug-core (as a dep of 'kernel-debug-modules', a dep of 'kernel-debug-modules-extra'). If you install with yum instead of dnf (by passing inst.nodnf), you only get one kernel, because kernel-modules-extra is selected to satisfy xl2tpd, not kernel-debug-modules-extra. I'll note that DNF's resolution here is really pretty dumb. Check this: <mock-chroot>sh-4.3# dnf install kernel ... <mock-chroot>sh-4.3# rpm -q kernel kernel-core kernel-modules kernel-3.19.0-1.fc22.x86_64 kernel-core-3.19.0-1.fc22.x86_64 kernel-modules-3.19.0-1.fc22.x86_64 <mock-chroot>sh-4.3# dnf install xl2tpd ... Installing: ... kernel-debug-core x86_64 3.19.0-1.fc22 rawhide 20 M kernel-debug-modules x86_64 3.19.0-1.fc22 rawhide 18 M kernel-debug-modules-extra x86_64 3.19.0-1.fc22 rawhide 2.3 M ... That is, even if kernel-modules is already installed and it could satisfy the requirement with a single additional package - 'kernel-modules-extra' - it prefers to install a whole separate kernel to satisfy the dep. This is significant because anaconda resolves deps once the whole 'desired' package set is generated, so the package manager does have the cue of anaconda's 'preferred' kernel package to go on. If you do an i686 F21 install with xl2tpd selected, yum correctly figures out it should install kernel-PAE-modules-extra to satisfy xl2tpd's dep, not kernel-modules-extra or kernel-debug-modules-extra or anything else, because it knows kernel-PAE is also selected. This case involves anaconda's behaviour as well as DNF's, but it's clearly primarily down to a difference in behaviour between DNF and yum, and so far anaconda devs are solidly of the opinion it should be resolved in DNF, not anaconda.
Same story... We are trying to find a simple rules how fix the yum vs dnf resolved sets. Btw thanks for writing a comment at libsolv github selection by provide issue so it's developer gets a better insight to Fedora packaging problem.
Adam, out of curiosity, why Anaconda cannot add the preferred kernel to the same transaction as the other packages?
It *does*. What it doesn't include is kernel-modules-extra, because most installs don't need it (that's why we have a split between kernel-modules and kernel-modules-extra in the first place). yum knows that, when a package needs a -modules-extra and we already have 'kernel', 'kernel-core' and 'kernel-modules' selected, it makes sense to pick 'kernel-modules-extra'. DNF doesn't, and happily chooses 'kernel-debug-modules-extra' - hence pulling in 'kernel-debug', 'kernel-debug-core' and 'kernel-debug-modules'. Sorry if the description wasn't clear. This is quite easy to play with: mock -r fedora-rawhide-x86_64 --clean mock -r fedora-rawhide-x86_64 --install dnf mock -r fedora-rawhide-x86_64 --install yum mock -r fedora-rawhide-x86_64 --shell dnf install kernel-modules dnf install xl2tpd (observe the list of deps, and say 'N') yum install xl2tpd (observe the list of deps, and compare to dnf)
*** Bug 1220821 has been marked as a duplicate of this bug. ***
Actually it seems, yum did the same thing in bug 1025301 comment 1 once. This was probably fixed in yum sometime, but didn't make it into dnf (or sth like that).
The issue is that xl2tpd requires kmod(l2tp_ppp.ko) which is provided both by kernel-modules-extra and kernel-debug-modules-extra. So this is a kernel packaging problem. The kernel-debug* packages should provide something different than kmod(xyz), e.g. kmod-debug(xyz), for debug versions of modules. Because yum preferred packages with shorter name the issue was just silently hidden in yum.
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle. Changing version to '23'. (As we did not run this process for some time, it could affect also pre-Fedora 23 development cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.) More information and reason for this action is here: https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23
"Because yum preferred packages with shorter name the issue was just silently hidden in yum." That's one way to characterize it, sure. The other way to characterize it is, as I have been doing all along, that yum consistently produces results with ambiguous dependencies that humans find significantly better than dnf's, and dnf's hard-line refusal to dirty up their nice clean code to make things better for humans is a pain the butt. In any *single* case of ambiguous dependencies you can probably come up with some kind of tweak to the dependencies which helps out dnf, yes. But packagers being fallible, squishy humans, they're probably going to keep making *new* ambiguous dependencies, and it seems like dnf is going to keep sucking at resolving them. In any case I'm not sure the change you suggest is really a sensible one, but the kernel team would know better whether it makes sense to 'flavor' the kmod virtual provides.
Well, but why does Yum "consistently produces results with ambiguous dependencies that humans find significantly better than dnf's"? Because maintainers abuse the knowledge about the Yum's algorithm of choosing providers. E.g. they choose a shorter name to make their package to be preferred. Or they choose a higher virtual provide version even if it does not match the reality. I think that they did "come up with some kind of tweak to the dependencies which helps out" Yum as well. What would Fedora do if someone creates a package foo-modules-extra which provides "kmod(l2tp_ppp.ko)" as well? Then, I think, Yum would choose this new package because it has shorter name [1]. Will they ask him to rename the package? My point is that we simply miss a Fedora recommended way how to express Fedora preferences. Until someone comes up with something like that, there is no reason to argue which workaround is better. [1] I'm not sure here but you know what I mean. It is always possible to create a new package which will be more preferred by Yum.
FYI - this is hitting KDE kickstart installs because @kde-desktop includes plasma-nm-l2tp which ends up pulling in x2ltpd. I've alerted the KDE folks here: https://lists.fedoraproject.org/pipermail/kde/2015-October/016218.html
How does DNF currently choose what packages to install to resolve a dependency when there are many options? If it is just the *first one* it finds, I'll suggest a probably better solution in such cases: it should select an option which pulls the least number of dependencies to be installed/updated. It certainly fixes the current situation and is not *hacky* at all. On the plus side, it even pulls the correct kernel-modules-extras package if a system is using kernel-debug packages rather than kernel packages. And even when all conditions are the same, it is better if DNF also has a predictable policy in choosing the package to be installed (e.g. smaller package name!), rather than being unpredictable. (Maybe it is predictably already? Why it chooses kernel-debug-modules extra currently?). And maybe it is better if RPM packages can express their preferences themselves, and DNF would choose the preferred provider in such cases. I'm sure I've seen such feature in a package format, but I'm not sure if it was .rpm or .deb.
Bug still present when installing KDE with Fedora-Everything-netinst-x86_64-24-1.2.iso. I suggest changing F23 => F24.
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Where do we stand here? 1) Dependency chain: plasma-nm-l2tp > NetworkManager-l2tp > xl2tpd (requires l2tp_ppp.ko) > kernel-modules-extra 2) Bodhi now put weak deps into metadata: https://github.com/fedora-infra/bodhi/issues/875 So maybe a "Suggests: kernel-modules-extra" in xl2tpd could help the resolver to choose the non-debug kernel package. Assigning this to xl2tpd should be worth a try, I guess.
@Adam So what do you think about assigning this to xl2tpd to try to improve resolving with weak deps? BTW, I would like to see F25 shipping without this ugly bug.
*** Bug 1284228 has been marked as a duplicate of this bug. ***
Please, consider adding "Suggests" tag as described in comment 14. DNF cannot guess which package it should choose as these provides are equal. xl2tpd seems to be the only package requiring "kmod(l2tp_ppp.ko)" so we don't need any general solution for that. More universal solution that comes to my mind is to add non-debug kernel packages weak dependency into fedora-repos [1] but there would need to be involved FESCo in the process. [1] https://fedoraproject.org/wiki/PackagingDrafts/ProvidesPreferences#Distribution_preference
so this is still not an xl2tpd bug. This should be re-assigned to either yum/dnf or kernel package.
(In reply to Paul Wouters from comment #18) > so this is still not an xl2tpd bug. This should be re-assigned to either > yum/dnf or kernel package. I'm confused. Please explain your statement. Why exactly Suggests: kernel-modules-extra in spec won't work here to help the resolver?
Will the suggests: tag ensure that kernel-modules-extra is installed? I mean, Then I can also just change the kmod(l2tp_ppp.ko) to kernel-modules-extra. Sine using the kernel module saves every single data packet from being send from kernel to userland and back to kernel, to then make it down the regular network stack, it really should be used when using xl2tpd. If kernel-modules-extra is supposed to be for more rarely used modules, perhaps l2tp.ko needs to move back into the core module set?
(In reply to Paul Wouters from comment #20) > Will the suggests: tag ensure that kernel-modules-extra is installed? I > mean, Then I can also just change the kmod(l2tp_ppp.ko) to > kernel-modules-extra. Don't do that. The entire point of introducing the Provides for individual modules is so that modules can move around within the kernel subpackages and userspace packages do not need to be updated to keep things working. > Sine using the kernel module saves every single data packet from being send > from kernel to userland and back to kernel, to then make it down the regular > network stack, it really should be used when using xl2tpd. > > If kernel-modules-extra is supposed to be for more rarely used modules, > perhaps l2tp.ko needs to move back into the core module set? DNF needs to be fixed.
(In reply to Josh Boyer from comment #21) > DNF needs to be fixed. I disagree. The resolver chooses a working package, so I consider the decision as week, no as wrong. I think no bug. But we don't need to argue, as this was already discussed here: https://bugzilla.redhat.com/show_bug.cgi?id=1349743 Conclusion: 1. Resolving is a matter of libsolv (not DNF). 2. Libsolv Dev doesn't see proper options for improvements (#43). 3. (Currently) Needs to be fixed at package level (what worked). @Paul So why no test with week deps? Won't break anything (even if it fails to help the resolver).
How am I guaranteed that a year from now the weak deps won't get skipped and xl2tpd will not have the kernel module it wants for proper performance? The package requires the kernel module to be installed. Whoever is responsible for that should ensure the kernel module is installed. If xl2tpd needs to use a different syntax in the spec file to ensure the kernel module is installed, then I can update that. But a "weak dependency" is not what xl2tpd wants. It Requires: the kernel module. Should I leave the component to dnf or should the component be libsolv ?
(In reply to Paul Wouters from comment #23) > How am I guaranteed that a year from now the weak deps won't get skipped and > xl2tpd will not have the kernel module it wants for proper performance? Oh, there might be some misunderstanding here. It's *not* intended at all to remove "Requires: ppp >= 2.4.5-18, kmod(l2tp_ppp.ko)" The module is "required" and will be installed, no matter which package will provide it in the future. The "Suggests:"-tag is just an addition, no a substitution.
So, why is there still no xl2tpd package with "Suggests:"-tag submitted for testing? I was pretty sure, the concerns and options were already properly discussed here.
Ping once more for xl2tpd F24 to F26 testing updates...
xl2tpd-1.3.8-2.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-5e09aacd3f
xl2tpd-1.3.8-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d460c89c93
xl2tpd-1.3.8-2.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5e09aacd3f
xl2tpd-1.3.8-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d460c89c93
Tested with F24. I removed all packages of the dependency chain and installed plasma-nm-l2tp with --enablerepo=updates-testing. This perfectly pulled in kernel-modules-extra. Removing --enablerepo=updates-testing from the dnf command leads to kernel-debug-modules-extra. So this is clearly fixed with xl2tpd-1.3.8-2.fc24 for now. Fine.
xl2tpd-1.3.8-2.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
xl2tpd-1.3.8-2.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.