Bug 1183835 - Bad choice of provider for ambiguous "mvn(org.fusesource.hawtjni:hawtjni-runtime)" requirement: picks older package with far more deps
Summary: Bad choice of provider for ambiguous "mvn(org.fusesource.hawtjni:hawtjni-runt...
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 22
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedFreezeException
Keywords:
Depends On:
Blocks: 1181567 1192182
TreeView+ depends on / blocked
 
Reported: 2015-01-19 23:59 UTC by Adam Williamson
Modified: 2016-01-06 20:20 UTC (History)
16 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2016-01-06 20:20:49 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 1148538 None None None Never
Red Hat Bugzilla 1149160 None None None Never

Internal Trackers: 1148538 1149160

Description Adam Williamson 2015-01-19 23:59:03 UTC
To reproduce, on a clean updated Fedora 21 Workstation install, try:

# yum install rhino
# dnf install rhino

yum wants to install four packages: rhino itself, and to provide the necessary dependencies, hawtjni-runtime-1.10-3.fc21 , jansi-1.11-6.fc21 , jansi-native-1.5-8.fc21 , and jline-2.10-14.fc21 .

dnf wants to install 28 packages, including a bunch of development-only stuff like kernel-headers, autoconf and automake.

The key point seems to be that dnf pulls in hawtjni-1.10-2.fc21 but does not pull in hawtjni-runtime-1.10-3.fc21 ; all the other deps stem from that.

Both hawtjni-1.10-2.fc21 and hawtjni-runtime-1.10-3.fc21 provide "mvn(org.fusesource.hawtjni:hawtjni-runtime)" , which is required by jansi-native . But yum seems to make by far the more sensible choice. hawtjni-runtime-1.10-3.fc21 is newer, and has far fewer dependencies. Why does dnf choose an older package with more dependencies?

Initially reported at http://www.forums.fedoraforum.org/showthread.php?t=302575

Comment 1 Honza Silhan 2015-01-20 19:45:49 UTC
Hi Adam, thanks for further investigation of dependency problem. DNF doesn't compute all possible solutions and find the smallest set, neither does yum. Yum has hack of rules [1]. We want it to solve it in more sophisticated way by weak dependencies [2] on package level i.e. add `Recommends: hawtjni-runtime` to jansi-native package.

[1] http://yum.baseurl.org/wiki/CompareProviders
[2] http://www.rpm.org/wiki/DynamicDependencies

Comment 2 Adam Williamson 2015-01-20 19:53:32 UTC
yum has some heuristics, yeah, which you can argue is hacky, but it *works*. Weak dependencies are all very well, but they're not a part of Fedora 22, and even when they become a part of Fedora, not all packages are magically going to start using them overnight; if you're relying on every individual case of this biting someone who then files a bug report which then causes the packager to deal with it via weak dependencies, that's a ton of pain and inconvenience for users that will drag on for years. I don't think that's a very nice way to go about it.

I understand it's nice to try and keep the design 'clean', but I'm afraid so long as you want to push for dnf to be made the system package manager on an aggressive schedule, we might need to compromise about issues like this.

compare_providers is a thing that people worked hard on in response to a clear need, that has proven itself to provide a clear benefit; it's not a good idea to just throw it away because it's not clean enough for Perfect Spherical Package Manager Land.

Comment 3 Mikolaj Izdebski 2015-01-21 05:26:49 UTC
This is not a bug in hatwjni or jansi-native.

hawtjni-runtime has ben split as subpackage of hawtjni specifically to optimize dependency chain (see bug #1166607).  Currently only hawtjni-runtime provides mvn(org.fusesource.hawtjni:hawtjni-runtime), so DNF should pick it as provider.

Weak dependencies are currently not allowed in Fedora, but they shouldn't be needed either. Packagers can't be expected to add Recommends each time provide is moved between packages, DNF should handle this case out of the box.

Comment 4 Adam Williamson 2015-01-21 05:35:37 UTC
yep, I should've pointed that out more strongly too. Thanks for the addition.

Comment 5 Jan Zeleny 2015-01-21 08:30:46 UTC
I would like to disprove one unfortunate misconception. Weak dependencies actually are in Fedora as of F21. The fact they are not explicitly approved doesn't mean they are forbidden or not available. The only part that might be missing (I'd have to check but I think it's working as well) is support on the Fedora infrastructure (particularly in createrepo that is used to create official Fedora repos).

Now to the point of this bug. What is strange to me is that dnf uses hawtjni-1.10-2.fc21 even though its newer version (hawtjni-1.10-3.fc21) should be available. Considering the package was split in the -3 release, the bug might have something to do with this specific scenarion, as there were some problems with it in the past.

Comment 6 Adam Williamson 2015-01-21 17:28:22 UTC
"The only part that might be missing (I'd have to check but I think it's working as well) is support on the Fedora infrastructure (particularly in createrepo that is used to create official Fedora repos)."

According to Miroslav, "I acknowledge that Recommends and Suggests can be problematic now because DNF and PackageKit is not ready yet." - https://lists.fedoraproject.org/pipermail/packaging/2015-January/010417.html .

Their use in F22 is under discussion by FPC (I thought it was already '23 at the earliest stuff' but apparently not), but has not yet been agreed upon. https://fedorahosted.org/fesco/ticket/1353 .

Comment 7 Jan Zeleny 2015-01-22 12:41:37 UTC
DNF is not ready in terms of repoquery command not supporting weak deps and a few other things like that. The basic functionality to include weak deps in transactions is there. Therefore weak deps are not technically blocked in Fedora.

Comment 8 Honza Horak 2015-01-26 16:15:47 UTC
(In reply to Jan Zeleny from comment #5)
> Now to the point of this bug. What is strange to me is that dnf uses
> hawtjni-1.10-2.fc21 even though its newer version (hawtjni-1.10-3.fc21)
> should be available. Considering the package was split in the -3 release,
> the bug might have something to do with this specific scenarion, as there
> were some problems with it in the past.

This is because dnf compares possible provides only based on package name (actually NVR), which is nonsense. It should look at epoch, version, release and only if those are the same, then look at name. Or it should use any other (ideally simple) but documented and usable algorithm. The current one is not usable at all unless we start prefixing packages names with numbers.

Similar discussion is at bz#1149160.

Comment 9 Adam Williamson 2015-01-26 16:36:38 UTC
doesn't comparing E, V, R only make much sense for binary packages from the same source? Comparing E, V, R first would work for *this* case, but what about this one?

cat-1.0 from srpm 'cat' provides animal
dog-3.7 from srpm 'dog' provides animal

cat requires mouse
dog requires leash, dogpound, citypark, dogsitter, ball, poopbags...

why prefer dog to cat in that case, when their versions really aren't comparable at all?

Comment 10 Mikolaj Izdebski 2015-01-26 17:46:58 UTC
(In reply to Adam Williamson (Red Hat) from comment #9)
> doesn't comparing E, V, R only make much sense for binary packages from the
> same source? Comparing E, V, R first would work for *this* case, but what
> about this one?
> 
> cat-1.0 from srpm 'cat' provides animal
> dog-3.7 from srpm 'dog' provides animal
> 
> cat requires mouse
> dog requires leash, dogpound, citypark, dogsitter, ball, poopbags...
> 
> why prefer dog to cat in that case, when their versions really aren't
> comparable at all?

Provides themselves can be versioned and provided version can be different from package version (it's actually quite common).

So adjusting your example accordingly:

cat-1.0 from srpm 'cat' provides 'animal=1.5'
dog-3.7 from srpm 'dog' provides 'animal=2.0-0.1.alpha1'

dog should be preferred because it is "better animal" than cat - provided version is greater.

Comment 11 Adam Williamson 2015-01-26 18:04:12 UTC
They *can* be, but there is no grounds for assuming that they *are*. We don't have a guideline telling people to do things that way. In practice, most virtual provides I have seen are unversioned.

Comment 12 Mikolaj Izdebski 2015-01-26 18:07:23 UTC
At the moment in rawhide we have 279618 provides, 209834 are versioned. That's over 75 %.  In the exact case of hawtjni provides are versioned too.

Comment 13 Adam Williamson 2015-01-26 23:44:54 UTC
I don't think many are versioned in the way you suggest, though. They just use the version of the source package.

Comment 14 Adam Williamson 2015-02-12 18:40:53 UTC
Some more notes:

This precise chain is hit when installed 'icedtea-web', which is a pretty common thing to install.

We hit another interesting case of bad ambiguous resolution during Anaconda DNF Test Day. There is a package xl2tpd which requires "kmod(l2tp_ppp.ko)". That is provided by all kernel -modules-extra packages. When you run 'yum install xl2tpd' without a kernel installed (e.g. from a mock chroot), yum selects kernel-modules-extra to provide it. When you run 'dnf install xl2tpd', DNF selects kernel-debug-modules-extra .

This causes an interesting effect at install time. If your package set happens to have xl2tpd in it, when anaconda first resolves the deps of the packages selected for install, 'kernel-debug-core' is pulled in. Later, anaconda runs a method to ensure the 'preferred' kernel is installed, dnfpayload.py's DNFPayload._select_kernel_package() . The logic of this method is (and has been for years, it's pretty much identical to the yumpayload one):

1. If 'kernel' is explicitly excluded figure the user knows what they're doing and don't install anything
2. If 'kernel-PAE' is available, install it
3. Otherwise if we're on ARM and 'kernel-(platform)' or 'kernel-lpae' is available, install it
4. Otherwise install 'kernel'

So the upshot is, that if your package set has xl2tpd in it, and the kernel currently in the repos is a 'production' build (so kernel-debug exists), you get two kernels installed: kernel-core (as a dep of 'kernel') and kernel-debug-core (as a dep of 'kernel-debug-modules', a dep of 'kernel-debug-modules-extra').

That was fun to work out!

Comment 15 Radek Holy 2015-02-13 14:46:19 UTC
Adam, I can see why you consider this a problem. On the other hand, strictly speaking, this is not a bug. You requested to install rhino. rhino was installed including all its dependencies. There is nothing more that DNF promises to user. Except that the provides are selected deterministically.
So, would it be acceptable for you if we close this bug and you open an RFE that describes the fitness function that you would like to be applied to select packages in case there are multiple choices?

Comment 16 Adam Williamson 2015-02-13 17:53:13 UTC
Well, the thing I worry about is the commonly-encountered icedtea-web case. That's one that real people are going to hit quite a lot.

Comment 17 Radek Holy 2015-02-14 19:14:26 UTC
Sorry for my ignorance but I don't think that you've answered my question. Simple yes or not would be enough.
Still I think that you are interested in the general DNF's candidates selection algorithm. Not just this case of icedtea-web. Otherwise you wouldn't make so much effort in reporting the other cases and even the "tracker" bug. I believe that you don't want a rule in DNF that handles specifically the "mvn(org.fusesource.hawtjni:hawtjni-runtime)" provide.
Could you please answer my question whether it would be OK if we close this bug and you open the new general RFE?
I apologize if it's just a language barrier that I didn't understand your answer.

Comment 18 Adam Williamson 2015-02-14 20:38:12 UTC
Sorry if it wasn't clear - my answer is usually I wouldn't mind (except I don't think there's a need for a downstream RFE, the upstream one is probably fine?), but this bug actually is quite likely to be commonly encountered - the original report shows the 'rhino' case, but you run into the same chain trying to install 'icedtea-web', which is something people commonly do as it's the standard Firefox Java plugin.

There's a couple of problem with closing bugs that are commonly encountered as something like 'oh, see this RFE':

1) it can look like we don't care
2) closed bugs do not show up in search, so you're likely to just get a duplicate filed soon if you close this one

so, that's why I might say it'd be better to leave it open. but ultimately, it's your call.

Comment 19 Adam Williamson 2015-02-14 20:39:17 UTC
oh, and then there's the fact that this specific case might actually be resolvable with a pretty clear-cut rule: for multiple providers that come from the same .src.rpm, prefer the one with a higher EVR.

Comment 20 Radek Holy 2015-02-14 22:31:04 UTC
OK, you are probably right. Currently, this is not a bug but since we care about this issue, it may change in the future. OK, let's track it here.

Let's see how libsolv will be improved...

Comment 21 Adam Williamson 2015-02-24 18:39:56 UTC
So this just landed upstream:

https://github.com/openSUSE/libsolv/commit/aa583e801055e3a17bca7f84feeef6898eecbda4

which I think may help. Could we backport it to Rawhide and F22's libsolv package so we can try it out?

As today is Alpha freeze, nominating as a freeze exception, it seems like we keep getting bugs along these lines (see also https://bugzilla.redhat.com/show_bug.cgi?id=1192182 ) and if we're going to try and make it better it would make sense to get it in early; I wouldn't want to land this last-minute before Alpha release, but right now is probably still early enough.

Comment 22 Germano Massullo 2015-03-01 10:08:30 UTC
This reminds me
https://bugzilla.redhat.com/show_bug.cgi?id=1070109

Comment 23 Petr Schindler 2015-03-02 21:15:39 UTC
Discussed at today's blocker review meeting [1].

This bug was rejected as Freeze Exception - We didn't get to this bug in time for it to be approved for Alpha FE status. Please submit before Beta freeze.

http://meetbot.fedoraproject.org/fedora-blocker-review/2015-03-02/

Comment 24 Jaroslav Reznik 2015-03-03 16:44:31 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Comment 25 Honza Silhan 2016-01-06 20:20:49 UTC
The algorithm of libsolv has been improved in version 0.6.14 and the main use cases should have find smallest set of dependencies. Moreover there is a guideline [1] where you can prefer one package over another. I am closing this, feel free to reopen and report other issues you encounter.

[1] https://fedoraproject.org/w/index.php?title=PackagingDrafts/ProvidesPreferences


Note You need to log in before you can comment on or make changes to this bug.