Bug 1107973 - [rfe] support package splitting
[rfe] support package splitting
Status: CLOSED UPSTREAM
Product: Fedora
Classification: Fedora
Component: hawkey (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Ales Kozumplik
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks: 871892
  Show dependency treegraph
 
Reported: 2014-06-11 03:51 EDT by Igor Gnatenko
Modified: 2015-01-05 12:31 EST (History)
17 users (show)

See Also:
Fixed In Version: hawkey-0.4.19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-07-28 07:56:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
debugdata from debugsolver (7.66 MB, application/x-xz)
2014-06-11 03:51 EDT, Igor Gnatenko
no flags Details

  None (edit)
Description Igor Gnatenko 2014-06-11 03:51:49 EDT
Created attachment 907506 [details]
debugdata from debugsolver

1. bump version
2. add new subpackage
3. for this new subpkg add Obsoletes: %{name} <= %{version}-%{release}
4. build rpms
5. create repo
6. try to update

Yum:
================================================================================
 Package          Arch         Version                        Repository   Size
================================================================================
Installing:
 shiny-test       x86_64       0.3-8.git411ac43.fc20          local       4.1 k
     replacing  shiny.x86_64 0.3-1.gitdc53364.fc20

Transaction Summary
================================================================================
Install  1 Package

Total download size: 4.1 k
Is this ok [y/d/N]: 

Dnf:
================================================================================
 Package       Arch           Version                       Repository     Size
================================================================================
Upgrading:
 shiny         x86_64         0.3-8.git411ac43.fc20         local         288 k

Transaction Summary
================================================================================
Upgrade  1 Package

Total size: 288 k
Is this ok [y/N]:
Comment 1 Igor Gnatenko 2014-06-11 03:52:42 EDT
also we can add obsoletes for main package.

yum will update base and install subpkg

dnf will update base
Comment 2 Ales Kozumplik 2014-06-11 03:57:07 EDT
Thanks Igor, we'll take a look.
Comment 3 Ales Kozumplik 2014-06-12 04:38:36 EDT
Hello, I debugged this in detail today. The behavior seen with DNF (and libsolv) is the correct one, shiny-test obsoletes "shiny <= 0.3-8", i.e. all such "shiny" packages with release-version less than or equal "0.3-8". But in this case, shiny-0.3-8.git411ac43.fc20.x86_64 is present in the 'local' repository. In the EVR algebra, it is technically correct to consider 0.3-8.git411ac43.fc20.x86_64 greater than 0.3-8 and so the solver decided shiny-test does not in fact obsolete the latest available 'shiny' package and correctly proceeds with an upgrade instead of obsoleting.
Comment 4 Ales Kozumplik 2014-06-12 04:41:40 EDT
FWIW, the latest release of NetworkManager does the same packaging mistake:

[akozumpl@localhost ~/Downloads]$ rpm -qp --obsoletes NetworkManager-wifi-0.9.9.95-1.git20140609.fc21.i686.rpm 
NetworkManager < 1:0.9.9.95-1
Comment 5 Dan Williams 2014-06-12 13:01:02 EDT
(In reply to Ales Kozumplik from comment #3)
> Hello, I debugged this in detail today. The behavior seen with DNF (and
> libsolv) is the correct one, shiny-test obsoletes "shiny <= 0.3-8", i.e. all
> such "shiny" packages with release-version less than or equal "0.3-8". But
> in this case, shiny-0.3-8.git411ac43.fc20.x86_64 is present in the 'local'
> repository. In the EVR algebra, it is technically correct to consider
> 0.3-8.git411ac43.fc20.x86_64 greater than 0.3-8 and so the solver decided
> shiny-test does not in fact obsolete the latest available 'shiny' package
> and correctly proceeds with an upgrade instead of obsoleting.

I don't quite understand this explanation...

What should the NetworkManager package do to ensure that the new sub-package gets correctly installed on upgrade?
Comment 6 Dan Williams 2014-06-12 14:02:19 EDT
(just re-opening to make sure that you see my question, I'd like to figure out how to make this work so I can rebuild NM if necessary to fix the issue)

But a larger question too... if the obsoletes behavior of dnf is different than yum, should that be noted somewhere, and should the packaging guidelines be updated to account for it?  If so, then it's going to be messy to make packages that work for both yum and dnf (eg, same specfile for F20 as F21).
Comment 7 Kevin Kofler 2014-06-12 19:23:05 EDT
Yum's behavior for Obsoletes was intentionally special-cased to support package splits, this was explicitly requested by packagers.

The rules in yum were:
1. Obsoletes *takes precedence* over a regular new version! I.e., if you have foo-1, and if the repository contains:
* foo-2 (no Obsoletes)
* bar-0 which Obsoletes: foo < 2
then bar will be preferred over foo. This is a very important feature for package renames (like qt→qt3 and qt4→qt). If you do not want foo to get replaced by bar, you have to use self-Obsoletes and rely on rule 2 below, i.e.:
* foo-2 "redundantly" Obsoletes: foo < 2
* bar-0 still Obsoletes: foo < 2
2. If multiple packages Obsolete the same existing package, ALL possible replacements will get dragged in (e.g., in the second example above, BOTH foo-2 and bar-0 will be picked to replace foo-1). This is the rule for package splits.

Splitting packages is a real PITA without those rules/hacks. Almost all the possibly surprising behavior in yum was there for a reason!
Comment 8 Kalev Lember 2014-06-16 06:05:49 EDT
(In reply to Ales Kozumplik from comment #4)
> FWIW, the latest release of NetworkManager does the same packaging mistake:
> 
> [akozumpl@localhost ~/Downloads]$ rpm -qp --obsoletes
> NetworkManager-wifi-0.9.9.95-1.git20140609.fc21.i686.rpm 
> NetworkManager < 1:0.9.9.95-1

I believe NetworkManager's obsoletes are correct here and there is no packaging mistake. Previous NM build had EVR "1:0.9.9.1-6.git20140319.fc21" which is covered by the obsoletes above.
Comment 9 Zdenek Kabelac 2014-06-17 11:53:17 EDT
Yep - I've had a real big fun trying to discover what went wrong when I've lost Wifi connection.

Learning few 'nmcli' command - and looking how to make  'Unmange' connection to be managed....

Such changes should not be scheduled around 'mass-rebuilds'....
Comment 10 Adam Williamson 2014-06-19 21:21:51 EDT
My understanding here is the same as KK's: yum's behaviour in this case was explicitly designed to allow the introduction of subpackages in such a way that they will be transparently installed on update for existing users.

I *believe* the relevant guideline in this situation is https://fedoraproject.org/wiki/Packaging:Conflicts#Splitting_Packages .

To clarify for the dnf folks: what happened here is that part of NetworkManager was split into a new subpackage called NetworkManager-wifi . The intent is that systems which currently have a pre-split version of NetworkManager installed should, after an update to a post-split version, have both NetworkManager and NetworkManager-wifi installed. This is what the "Obsoletes" is intended to achieve.

What we're considering is a situation like this:

foo-1.0.rpm is installed
foo-2.0.rpm and foo-bar-2.0.rpm are both available
foo-bar-2.0.rpm and foo-2.0.rpm both have "Obsoletes: foo < 1.1"

yum's behaviour in this case is to process the obsoletion ahead of the upgrade availability, and so install both packages that obsolete foo. dnf's behaviour is to process the update availability ahead of the obsoletion: it notes that foo-2.0 is an update to foo-1.0, adds it to the package set, *then* processes the obsoletion. Now the obsoletion doesn't take effect, because so far as dnf is concerned, we're onto foo-2.0 already.

Neither behaviour seems absolutely and objectively incorrect; both are plausible interpretations of an ambiguous situation. *However*, yum's behaviour provides a mechanism for causing a new subpackage to be transparently installed on update (without a hard dependency on the main package, which would make the subpackage split pointless). dnf's does not. Fedora has a clear need for such a mechanism.

If dnf is not going to switch to yum's behavior in this case, the dnf devs need to provide (or explain, if there is already one of which we're not aware) a mechanism for doing this kind of subpackage split, and we need to make sure that all F20+ cases use it, otherwise we may cause problems in supported cases when we switch to dnf with F22.
Comment 11 Ales Kozumplik 2014-06-23 07:32:45 EDT
(In reply to Dan Williams from comment #5)
> (In reply to Ales Kozumplik from comment #3)
> > Hello, I debugged this in detail today. The behavior seen with DNF (and
> > libsolv) is the correct one, shiny-test obsoletes "shiny <= 0.3-8", i.e. all
> > such "shiny" packages with release-version less than or equal "0.3-8". But
> > in this case, shiny-0.3-8.git411ac43.fc20.x86_64 is present in the 'local'
> > repository. In the EVR algebra, it is technically correct to consider
> > 0.3-8.git411ac43.fc20.x86_64 greater than 0.3-8 and so the solver decided
> > shiny-test does not in fact obsolete the latest available 'shiny' package
> > and correctly proceeds with an upgrade instead of obsoleting.
> 
> I don't quite understand this explanation...
> 
> What should the NetworkManager package do to ensure that the new sub-package
> gets correctly installed on upgrade?

Sorry, it looks like what I described above is relevant for the comment 0 only. NM might be a different situation after all. Looking into how this should work.
Comment 12 Ales Kozumplik 2014-06-23 09:48:30 EDT
Michael, are there some capabilities in libsolv to facilitate package splitting, something similar to what Adam described in comment 10 perhaps?
Comment 13 Michael Schröder 2014-06-23 10:26:00 EDT
We talked about package splitting at 2013's DevConf, AFAIK yum looks at the obsoletes (and not conflicts) to detect splits.

It's not hard to teach libsolv about splits, we just need to define the exact semantics. For example I don't think libsolv should try to install *all* packages obsoleting an installed package, there needs to be a way to determine which packages belong to a split (e.g. they need to come from the same repository).
Comment 14 Adam Williamson 2014-06-23 22:23:08 EDT
misc: the page title is somewhat misleading, the guideline in question does talk about Obsoletes. It's not laser-focused on this particular issue, but it does sort of cover it. I believe KK's correct to suggest that yum's behaviour in this case was actually explicitly designed to be used for this situation of subpackage splits, but I couldn't find a really great reference for that in our guidelines / docs anywhere - it was probably one of those things which only ever got written down on a mailing list.
Comment 15 Ales Kozumplik 2014-06-24 02:00:10 EDT
(In reply to Michael Schröder from comment #13)
> It's not hard to teach libsolv about splits, we just need to define the
> exact semantics. For example I don't think libsolv should try to install
> *all* packages obsoleting an installed package, there needs to be a way to
> determine which packages belong to a split (e.g. they need to come from the
> same repository).

I saw SUSE using some special provides for that. It is mentioned as a legacy way in libsolv somewhere. Is that really wrong? What are the SUSE packagers typically doing now?
Comment 16 Michael Schröder 2014-06-24 05:33:30 EDT
SUSE uses a very old feature called "split-provides": the split-off package provides a special "package-name:file-name" string, this gets auto-translated into a supplements by libsolv.
It has some drawbacks, so please do not consider to use the same mechanism for Fedora. If you really want to switch to a new way package splits work, please switch to something easy and sane ;)
Comment 17 Ales Kozumplik 2014-07-28 07:08:38 EDT
After much open discussion about supporting package splitting the team has decided to sustain the Yum semantics and Michael has implemented the libsolv support:

https://github.com/openSUSE/libsolv/commit/99e2d6786a2f18126932170eccd5da139eb5f438
Comment 18 Ales Kozumplik 2014-07-28 07:56:36 EDT
hawkey commit 8ceb623 enables the splitting semantics.
Comment 19 Adam Williamson 2014-07-28 11:48:36 EDT
what's the approximate time frame for that getting downstream into f21 and rawhide?
Comment 20 Ales Kozumplik 2014-07-29 01:11:12 EDT
It happened yesterday:

http://koji.fedoraproject.org/koji/buildinfo?buildID=548316
http://koji.fedoraproject.org/koji/buildinfo?buildID=548349

According to [1] I am supposed to make f21 updates but bodhi won't let me ("libsolv-0.6.4-1.fc21 not tagged as an update candidate").

[1] https://fedoraproject.org/wiki/Updates_Policy
Comment 21 Adam Williamson 2014-07-29 01:23:36 EDT
Bodhi hasn't kicked in for F21 yet, it doesn't come in right after branching. we should probably mention that. For now it works like rawhide, your build will go right into f21 in the next compose.
Comment 22 Ales Kozumplik 2014-07-29 01:31:32 EDT
We definitely should mention that, otherwise people could question the relevance of the documentation and the processes. Should I open a ticket somewhere?
Comment 23 Adam Williamson 2014-07-29 12:36:27 EDT
ales: I updated the wiki page last night. thanks for the catch. dgilmore tells me we used to branch and freeze simultaneously, hence there was no gap, but we changed that a while ago and didn't update that wiki page.
Comment 24 Ales Kozumplik 2014-07-30 01:16:10 EDT
Thanks Adam.

Note You need to log in before you can comment on or make changes to this bug.