Description of problem:
The special handling of module streams in DNF is not implemented at the appropriate layer. As a result, applications that rely on libdnf can end up with broken updates if there exists any package in any module stream in an enabled repo with a higher NEVRA.
The practical effect of this is that the highest-version module stream will always supersede any lower-versioned one if it's updated by a tool such as PackageKit or microdnf. It also can cause true system breakage in limited circumstances (where the dependencies of two higher-version streams of enabled modules conflict with one another).
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Install F28 Server Edition
2. `dnf module install nodejs:6`
3. `pkcon update`
The list of available packages will include Node.js 10.x:
npm-1:5.6.0-220.127.116.11.1.module_1716+9b6ad2c1.x86_64 Node.js Package Manager
Packages must remain locked to their module streams.
Proposed as a Blocker for 29-beta by Fedora user sgallagh using the blocker tracking app because:
Nominated as a FESCo blocker bug for Beta in https://pagure.io/fesco/issue/1889
For the record, the Beta criteria have been updated to explicitly require this:
"The installed system must be able to download and install appropriate updates with the default tool for the relevant update type in all release-blocking desktops (e.g. default graphical package manager)... This includes - but is not necessarily limited to - updates for non-module packages, updates for any official module streams that are enabled (including any enabled by default in a release-blocking deployment),"
Discussed during the 2018-05-14 blocker review meeting: 
The decision to classify this bug as an AcceptedBlocker was made as it violates the following blocker criteria:
"The installed system must be able to download and install appropriate updates with the default tool for the relevant update type in all release-blocking desktops (e.g. default graphical package manager)."
Is there any update on this, beyond the devel whiteboard note?
Ping! Any further update?
Patch is merged in upstream, you can expect release in near future.
The expected release date is tomorrow.
Matt Miller asked me to add some notes on fallout here, so...
It seems the 'release' here involved landing DNF 3 along with libdnf 0.15. This sort of broke Rawhide quite a bit.
We (QA, releng etc) are working through the issues. Here are the issues that have showed up so far:
1. anaconda needed adjusting to some interface changes: https://github.com/rhinstaller/anaconda/pull/1515
2. lorax needed adjusting too: https://github.com/weldr/lorax/pull/388
3. PackageKit had to be rebuilt for the new libdnf soname (note, there was not a soname bump announcement to devel@ AFAICS)
4. a subtle issue in dnf config handling broke appliance composes: https://bugzilla.redhat.com/show_bug.cgi?id=1595917 , I am about to do a livecd-tools build with a workaround for that now: https://github.com/livecd-tools/livecd-tools/pull/102
That's where we're up to ATM. Once I've done the livecd-tools build we'll fire another compose and see how far that one gets.
The issue is solved by libdnf-0.15.1-1 that was released into rawhide.
We are also working on fixing other issues.
Tested with libdnf-0.15.1-1.fc29.x86_64 and it failed.
# dnf module enable nodejs:8
# dnf install nodejs
# microdnf install nodejs
This seems like something we could easily add as an automatic test. At least, as long as those node streams exist.
Adding dnf module-y tests to openQA is on my todo list, yeah.
oh, but of course it could also be in the upstream test suite.
Is there any update on this?
We need one more fix to make it work:
DNF was writing garbage to the ini files.
When the fix is applied and module enabled again (or ini file fixed), microdnf works as expected.
AFAIK PackageKit still doesn't work as it doesn't contain necessary fix.
It was my understanding that implementation in libdnf would cover the needs of all consumers for "phase 1" compatibility. Is that not the case?
Is there a bug for the packagekit work needed? Does someone have ownership of getting that fixed?
That was also my initial assumption, but PackageKit creates additional package sacks for caching purposes and they need modularity filter applied.
The patch is a simple one-liner and is pushed upstream.
We communicate the necessity of a new build with PackageKit team on June 22 (couple days before release to Rawhide).
We can create a bug if you think it helps.
We have just released dnf-3.0.3-2.fc29 that writes correct data on disk.
Microdnf should be working just fine with modular contents already.
Thanks Daniel. A bug would help with tracking.
If it's sufficiently important, any proven packager could do a build with the patch backported, I guess?
It blocks the Modularity for Everyone change, so probably counts as sufficiently important — if we can't get a timely update through normal channels.
I have tested this by starting with an F28 vagrant box, enabling the F28 modular repo and then updating dnf and libdnf from koji/rawhide. (The very latest dnf build for rawhide does not seem to have hit all the mirrors yet.)
I then enabled nodejs:8 and installed the nodejs package. This seems to work.
[root@localhost modules.d]# rpm -q dnf libdnf
[root@localhost modules.d]# dnf module enable nodejs:8
Last metadata expiration check: 0:34:15 ago on Fri 13 Jul 2018 02:08:33 AM UTC.
'nodejs:8' is enabled
[root@localhost modules.d]# ls
[root@localhost modules.d]# cat nodejs.module
name = nodejs
stream = 8
version = -1
enabled = True
locked = False
[root@localhost modules.d]# microdnf install nodejs
Package Repository Size
nodejs-1:8.11.2-2.module_1837+5ed5379b.x86_64 updates-modular 7.8 MB
npm-1:5.6.0-18.104.22.168.2.module_1837+5ed5379b.x86_64 updates-modular 4.5 MB
Installing: 2 packages
Reinstalling: 0 packages
Upgrading: 0 packages
Removing: 0 packages
Downgrading: 0 packages
Running transaction test...
I've filed a PackageKit bug #1600848
Is the non-PackageKit use case supposed to be fixed in Rawhide now, btw?
I've tried it just now and it still behaves like in comment 10.
This seems to work with dnf-3.0.3-2.fc29.noarch.
Stephen, can you confirm this is resolved now, and close the bug if so? Thanks!
Looks like everything is working now. Thanks!