Bug 1575626 - libdnf does not handle module stream updates properly
Summary: libdnf does not handle module stream updates properly
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: libdnf
Version: 28
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Keywords: Reopened, Triaged
Depends On:
Blocks: 1599413 F29BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2018-05-07 13:07 UTC by Stephen Gallagher
Modified: 2018-08-14 14:21 UTC (History)
14 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-08-14 14:21:35 UTC


Attachments (Terms of Use)

Description Stephen Gallagher 2018-05-07 13:07:27 UTC
Description of problem:

The special handling of module streams in DNF is not implemented at the appropriate layer. As a result, applications that rely on libdnf can end up with broken updates if there exists any package in any module stream in an enabled repo with a higher NEVRA.

The practical effect of this is that the highest-version module stream will always supersede any lower-versioned one if it's updated by a tool such as PackageKit or microdnf. It also can cause true system breakage in limited circumstances (where the dependencies of two higher-version streams of enabled modules conflict with one another).

Version-Release number of selected component (if applicable):
libdnf-0.11.1-3.fc28.x86_64

How reproducible:
Every time

Steps to Reproduce:
1. Install F28 Server Edition
2. `dnf module install nodejs:6`
3. `pkcon update`


Actual results:

The list of available packages will include Node.js 10.x:

```
 nodejs-1:10.0.0-1.module_1716+9b6ad2c1.x86_64	JavaScript runtime
 npm-1:5.6.0-1.10.0.0.1.module_1716+9b6ad2c1.x86_64	Node.js Package Manager
```

Expected results:

Packages must remain locked to their module streams.


Additional info:

Comment 1 Fedora Blocker Bugs Application 2018-05-07 13:15:14 UTC
Proposed as a Blocker for 29-beta by Fedora user sgallagh using the blocker tracking app because:

 Nominated as a FESCo blocker bug for Beta in https://pagure.io/fesco/issue/1889

Comment 2 Stephen Gallagher 2018-05-14 12:54:12 UTC
For the record, the Beta criteria have been updated to explicitly require this:

https://fedoraproject.org/wiki/Basic_Release_Criteria#updates

"The installed system must be able to download and install appropriate updates with the default tool for the relevant update type in all release-blocking desktops (e.g. default graphical package manager)... This includes - but is not necessarily limited to - updates for non-module packages, updates for any official module streams that are enabled (including any enabled by default in a release-blocking deployment),"

Comment 3 František Zatloukal 2018-05-14 22:36:02 UTC
Discussed during the 2018-05-14 blocker review meeting: [1]

The decision to classify this bug as an AcceptedBlocker was made as it violates the following blocker criteria:

"The installed system must be able to download and install appropriate updates with the default tool for the relevant update type in all release-blocking desktops (e.g. default graphical package manager)."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2018-05-14/f29-blocker-review.2018-05-14-16.01.log.txt

Comment 4 Matthew Miller 2018-05-31 12:40:50 UTC
Is there any update on this, beyond the devel whiteboard note?

Comment 5 Matthew Miller 2018-06-21 16:36:08 UTC
Ping! Any further update?

Comment 6 Martin Hatina 2018-06-25 06:31:23 UTC
Patch is merged in upstream, you can expect release in near future.

Comment 7 Daniel Mach 2018-06-25 10:09:31 UTC
The expected release date is tomorrow.

Comment 8 Adam Williamson 2018-06-27 19:07:24 UTC
Matt Miller asked me to add some notes on fallout here, so...

It seems the 'release' here involved landing DNF 3 along with libdnf 0.15. This sort of broke Rawhide quite a bit.

We (QA, releng etc) are working through the issues. Here are the issues that have showed up so far:

1. anaconda needed adjusting to some interface changes: https://github.com/rhinstaller/anaconda/pull/1515
2. lorax needed adjusting too: https://github.com/weldr/lorax/pull/388
3. PackageKit had to be rebuilt for the new libdnf soname (note, there was not a soname bump announcement to devel@ AFAICS)
4. a subtle issue in dnf config handling broke appliance composes: https://bugzilla.redhat.com/show_bug.cgi?id=1595917 , I am about to do a livecd-tools build with a workaround for that now: https://github.com/livecd-tools/livecd-tools/pull/102

That's where we're up to ATM. Once I've done the livecd-tools build we'll fire another compose and see how far that one gets.

Comment 9 Jaroslav Mracek 2018-06-28 06:34:42 UTC
The issue is solved by libdnf-0.15.1-1 that was released into rawhide.

We are also working on fixing other issues.

Comment 10 Petr Šabata 2018-06-29 16:06:21 UTC
Tested with libdnf-0.15.1-1.fc29.x86_64 and it failed.

# dnf module enable nodejs:8
# dnf install nodejs
(offers nodejs-8)
# microdnf install nodejs
(install nodejs-10)

Comment 11 Matthew Miller 2018-06-29 16:09:32 UTC
This seems like something we could easily add as an automatic test. At least, as long as those node streams exist.

Comment 12 Adam Williamson 2018-06-29 17:32:14 UTC
Adding dnf module-y tests to openQA is on my todo list, yeah.

Comment 13 Adam Williamson 2018-06-29 17:32:35 UTC
oh, but of course it could also be in the upstream test suite.

Comment 14 Matthew Miller 2018-07-11 13:19:06 UTC
Is there any update on this?

Comment 15 Daniel Mach 2018-07-11 13:55:53 UTC
We need one more fix to make it work:
https://github.com/rpm-software-management/dnf/pull/1130

DNF was writing garbage to the ini files.
When the fix is applied and module enabled again (or ini file fixed), microdnf works as expected.

AFAIK PackageKit still doesn't work as it doesn't contain necessary fix.

Comment 16 Matthew Miller 2018-07-11 17:11:11 UTC
It was my understanding that implementation in libdnf would cover the needs of all consumers for "phase 1" compatibility. Is that not the case?

Is there a bug for the packagekit work needed? Does someone have ownership of getting that fixed?

Comment 17 Daniel Mach 2018-07-12 13:24:00 UTC
That was also my initial assumption, but PackageKit creates additional package sacks for caching purposes and they need modularity filter applied.

The patch is a simple one-liner and is pushed upstream.
https://github.com/hughsie/PackageKit/commit/b4fe0ab853e8d8b9bc410a3d070e2d05229e543d

We communicate the necessity of a new build with PackageKit team on June 22 (couple days before release to Rawhide).
We can create a bug if you think it helps.

We have just released dnf-3.0.3-2.fc29 that writes correct data on disk.
Microdnf should be working just fine with modular contents already.

Comment 18 Matthew Miller 2018-07-12 13:35:08 UTC
Thanks Daniel. A bug would help with tracking.

Comment 19 Adam Williamson 2018-07-12 15:17:15 UTC
If it's sufficiently important, any proven packager could do a build with the patch backported, I guess?

Comment 20 Matthew Miller 2018-07-12 15:57:05 UTC
It blocks the Modularity for Everyone change, so probably counts as sufficiently important — if we can't get a timely update through normal channels.

Comment 21 Ian McLeod 2018-07-13 02:46:38 UTC
I have tested this by starting with an F28 vagrant box, enabling the F28 modular repo and then updating dnf and libdnf from koji/rawhide.  (The very latest dnf build for rawhide does not seem to have hit all the mirrors yet.)

I then enabled nodejs:8 and installed the nodejs package.  This seems to work.

[root@localhost modules.d]# rpm -q dnf libdnf
dnf-3.0.3-2.fc29.noarch
libdnf-0.15.2-1.fc29.x86_64
[root@localhost modules.d]# dnf module enable nodejs:8
Last metadata expiration check: 0:34:15 ago on Fri 13 Jul 2018 02:08:33 AM UTC.
'nodejs:8' is enabled
[root@localhost modules.d]# ls
nodejs.module
[root@localhost modules.d]# cat nodejs.module 
[nodejs]
name = nodejs
stream = 8
version = -1
profiles = 
enabled = True
locked = False
[root@localhost modules.d]# microdnf install nodejs
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Package                                                                                                                       Repository                                       Size
Installing:                                                                                                                                                                        
 nodejs-1:8.11.2-2.module_1837+5ed5379b.x86_64                                                                                updates-modular                                7.8 MB
 npm-1:5.6.0-1.8.11.2.2.module_1837+5ed5379b.x86_64                                                                           updates-modular                                4.5 MB
Transaction Summary:
 Installing:        2 packages
 Reinstalling:      0 packages
 Upgrading:         0 packages
 Removing:          0 packages
 Downgrading:       0 packages
Downloading packages...
Running transaction test...
Installing: nodejs;1:8.11.2-2.module_1837+5ed5379b;x86_64;updates-modular
Installing: npm;1:5.6.0-1.8.11.2.2.module_1837+5ed5379b;x86_64;updates-modular
Complete.
[root@localhost modules.d]#

Comment 22 Daniel Mach 2018-07-13 07:48:34 UTC
I've filed a PackageKit bug #1600848

Comment 23 Petr Šabata 2018-07-13 09:31:16 UTC
Is the non-PackageKit use case supposed to be fixed in Rawhide now, btw?
I've tried it just now and it still behaves like in comment 10.

dnf-3.0.2-1.fc29.noarch
libdnf-0.15.2-1.fc29.x86_64
microdnf-3.0.1-1.fc29.x86_64

Comment 24 Petr Šabata 2018-07-16 10:47:46 UTC
This seems to work with dnf-3.0.3-2.fc29.noarch.

Comment 25 Adam Williamson 2018-08-13 18:20:44 UTC
Stephen, can you confirm this is resolved now, and close the bug if so? Thanks!

Comment 26 Stephen Gallagher 2018-08-14 14:21:35 UTC
Looks like everything is working now. Thanks!


Note You need to log in before you can comment on or make changes to this bug.