Bug 1629397 - Situation not explained clearly when an update pulls in packages from, and auto-enables, a new default module stream
Summary: Situation not explained clearly when an update pulls in packages from, and au...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
Assignee: Jaroslav Mracek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-15 17:13 UTC by Kevin Fenzi
Modified: 2019-08-13 18:55 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-22 10:48:45 UTC


Attachments (Terms of Use)

Description Kevin Fenzi 2018-09-15 17:13:44 UTC
dnf-3.5.1-1.fc30.noarch

No modules enabled: 

~ sudo dnf module list --installed
Last metadata expiration check: 0:09:03 ago on Sat 15 Sep 2018 09:46:49 AM PDT.                                       
Error: No matching Modules to list

~ sudo dnf distro-sync -x gnutls\* -x libreoffice-core -x liborcus                                                  
Last metadata expiration check: 0:25:49 ago on Sat 15 Sep 2018 09:46:49 AM PDT.                                        
Dependencies resolved.                                                                                                 
======================================================================================================================$
 Package                                   Arch     Version                                    Repository         Size
======================================================================================================================$
Upgrading:
 stratisd                                  x86_64   0.9.0-1.module_2156+f9892fa0               rawhide-modular   1.9 M
Downgrading:
 ant-lib                                   noarch   1.10.4-1.module_1886+ece9a977              rawhide-modular   2.0 M
 aopalliance                               noarch   1.0-17.module_1885+a6f9b3e6                rawhide-modular    16 k
 apache-commons-cli                        noarch   1.4-4.module_1885+a6f9b3e6                 rawhide-modular    73 k
 apache-commons-codec                      noarch   1.11-3.module_1885+a6f9b3e6                rawhide-modular   287 k
 apache-commons-io                         noarch   1:2.6-3.module_1885+a6f9b3e6               rawhide-modular   222 k
 apache-commons-lang3                      noarch   3.7-3.module_1885+a6f9b3e6                 rawhide-modular   482 k
 apache-commons-logging                    noarch   1.2-13.module_1885+a6f9b3e6                rawhide-modular    84 k
 atinject                                  noarch   1-28.20100611svn86.module_1885+a6f9b3e6    rawhide-modular    19 k
 cdi-api                                   noarch   1.2-8.module_1885+a6f9b3e6                 rawhide-modular    69 k
 geronimo-annotation                       noarch   1.0-23.module_1885+a6f9b3e6                rawhide-modular    24 k
 glassfish-el-api                          noarch   3.0.1-0.7.b08.module_1885+a6f9b3e6         rawhide-modular   104 k
 google-guice                              noarch   4.1-11.module_1885+a6f9b3e6                rawhide-modular   469 k
 guava20                                   noarch   20.0-6.module_1885+a6f9b3e6                rawhide-modular   2.1 M
 hawtjni-runtime                           noarch   1.16-1.module_1889+fac7270b                rawhide-modular    42 k
 httpcomponents-client                     noarch   4.5.5-4.module_1885+a6f9b3e6               rawhide-modular   717 k
 httpcomponents-core                       noarch   4.4.9-4.module_1885+a6f9b3e6               rawhide-modular   634 k
 jansi                                     noarch   1.17.1-1.module_1889+fac7270b              rawhide-modular    78 k
 jansi-native                              x86_64   1.7-5.module_1889+fac7270b                 rawhide-modular    37 k
 jboss-interceptors-1.2-api                noarch   1.0.0-8.module_1885+a6f9b3e6               rawhide-modular    32 k
 jcl-over-slf4j                            noarch   1.7.25-4.module_1885+a6f9b3e6              rawhide-modular    31 k
 jline                                     noarch   2.14.6-2.module_1889+fac7270b              rawhide-modular   156 k
 maven-lib                                 noarch   1:3.5.4-1.module_1885+a6f9b3e6             rawhide-modular   1.4 M
 maven-resolver-api                        noarch   1:1.1.1-2.module_1885+a6f9b3e6             rawhide-modular   137 k
 maven-resolver-impl                       noarch   1:1.1.1-2.module_1885+a6f9b3e6             rawhide-modular   176 k
 maven-resolver-spi                        noarch   1:1.1.1-2.module_1885+a6f9b3e6             rawhide-modular    39 k
 maven-resolver-util                       noarch   1:1.1.1-2.module_1885+a6f9b3e6             rawhide-modular   147 k
 maven-shared-utils                        noarch   3.2.1-0.1.module_1885+a6f9b3e6             rawhide-modular   164 k
 maven-wagon-provider-api                  noarch   3.1.0-1.module_1885+a6f9b3e6               rawhide-modular    62 k
 plexus-cipher                             noarch   1.7-14.module_1885+a6f9b3e6                rawhide-modular    28 k
 plexus-classworlds                        noarch   2.5.2-9.module_1885+a6f9b3e6               rawhide-modular    64 k
 plexus-containers-component-annotations   noarch   1.7.1-6.module_1885+a6f9b3e6               rawhide-modular    18 k
 plexus-interpolation                      noarch   1.22-9.module_1885+a6f9b3e6                rawhide-modular    78 k
 plexus-sec-dispatcher                     noarch   1.4-24.module_1885+a6f9b3e6                rawhide-modular    31 k
 plexus-utils                              noarch   3.1.0-1.module_1885+a6f9b3e6               rawhide-modular   258 k
 scala                                     noarch   2.10.6-8.module_1889+fac7270b              rawhide-modular    23 M
 sisu-inject                               noarch   1:0.3.3-3.module_1885+a6f9b3e6             rawhide-modular   337 k
 sisu-plexus                               noarch   1:0.3.3-3.module_1885+a6f9b3e6             rawhide-modular   178 k
 slf4j                                     noarch   1.7.25-4.module_1885+a6f9b3e6              rawhide-modular    76 k
Enabling module streams:
 ant                                                1.10
 maven                                              3.5
 scala                                              2.10

Transaction Summary
=======================================================================================================================
Upgrade     1 Package
Downgrade  38 Packages

Total download size: 35 M
Is this ok [y/N]: n
Operation aborted.

I don't expect dnf distro-sync to enable modules I didn't enable...

Comment 1 Adam Williamson 2018-09-15 17:41:36 UTC
I can reproduce this from f29 with:

dnf --disablerepo=updates-modular --disablerepo=updates-testing-modular --disablerepo=fedora-modular --disablerepo=fedora --disablerepo=updates-testing --enablerepo=rawhide --releasever=rawhide --enablerepo=rawhide-modular distro-sync -x liborcus

but I can't make it happen only using f29 repos, ATM. Might be something specific to Rawhide repos. Have we actually started allowing modules to be auto-enabled, or something?

Comment 2 Adam Williamson 2018-09-15 17:45:03 UTC
What....what *is* this:

https://github.com/rpm-software-management/dnf/commit/1b13ee79df086f13ea330184b4ea386d9be51085

"[module] Auto-enable module streams based on installed RPMs."

wat? why?

Comment 3 Adam Williamson 2018-09-15 17:59:28 UTC
Hmm, if I comment that code out of base.py, then I get this:

...

Downgrading:
 ant                                                  noarch 1.10.4-1.module_1886+ece9a977      rawhide-modular 188 k
 ant-lib                                              noarch 1.10.4-1.module_1886+ece9a977      rawhide-modular 2.0 M
 apache-commons-cli                                   noarch 1.4-4.module_1885+a6f9b3e6         rawhide-modular  73 k
 apache-commons-codec                                 noarch 1.11-3.module_1885+a6f9b3e6        rawhide-modular 287 k
 apache-commons-logging                               noarch 1.2-13.module_1885+a6f9b3e6        rawhide-modular  84 k
 glassfish-el-api                                     noarch 3.0.1-0.7.b08.module_1885+a6f9b3e6 rawhide-modular 104 k
 hawtjni-runtime                                      noarch 1.16-1.module_1889+fac7270b        rawhide-modular  42 k
 jansi                                                noarch 1.17.1-1.module_1889+fac7270b      rawhide-modular  78 k
 jansi-native                                         x86_64 1.7-5.module_1889+fac7270b         rawhide-modular  37 k
 jline                                                noarch 2.14.6-2.module_1889+fac7270b      rawhide-modular 156 k
 slf4j                                                noarch 1.7.25-4.module_1885+a6f9b3e6      rawhide-modular  76 k
 LibRaw                                               x86_64 0.19.0-3.fc29                      rawhide         308 k
 clang6.0-libs                                        x86_64 6.0.1-5.fc29                       rawhide          12 M
 dnssec-tools-libs                                    x86_64 2.2.1-1.fc29                       rawhide         148 k
 gdb                                                  x86_64 8.1.90.20180727-45.fc30            rawhide         132 k
 gdb-headless                                         x86_64 8.1.90.20180727-45.fc30            rawhide         3.3 M
 gtk-update-icon-cache                                x86_64 3.24.0-4.fc30                      rawhide          32 k
 gtk3                                                 x86_64 3.24.0-4.fc30                      rawhide         4.4 M
 gtk3-devel                                           x86_64 3.24.0-4.fc30                      rawhide         4.4 M
 gtk3-immodule-xim                                    x86_64 3.24.0-4.fc30                      rawhide          20 k
 libknet1                                             x86_64 1.3-2.fc29                         rawhide          67 k

Transaction Summary

note, it still wants to install packages from the modular repo, but it doesn't enable the modules. So, it's not that the 'auto-enable module streams' code is wrongly kicking in and enabling those modules, *then* packages from those modules are getting selected - it's the other way around: packages from the modular repo are somehow getting pulled into the transaction, *then* the 'auto-enable module streams' code causes the modules they come from to be enabled. So that code isn't the core of the problem it seems.

Comment 4 Adam Williamson 2018-09-15 18:01:56 UTC
dnf 3.2.0 / libdnf 0.17.0 does the same (I just downgraded and checked). So this isn't something introduced recently in dnf, it seems. Most likely it's some sort of problem in the Rawhide repodata?

Comment 5 Kevin Fenzi 2018-09-15 18:24:00 UTC
Adding sgallagh and lsedlar for any comments...

Comment 6 Daniel Mach 2018-09-17 11:39:43 UTC
Module auto-enablement based on installed RPMs is a key feature.
This behavior is expected.
From DNF perspective, this is NOTABUG.

Comment 7 Stephen Gallagher 2018-09-17 11:53:44 UTC
(In reply to Adam Williamson from comment #2)
> What....what *is* this:
> 
> https://github.com/rpm-software-management/dnf/commit/
> 1b13ee79df086f13ea330184b4ea386d9be51085
> 
> "[module] Auto-enable module streams based on installed RPMs."
> 
> wat? why?

To clarify this: what it means is that if the current system has RPMs installed that are part of a module but the module itself is not marked as enabled, it will do so to make sure that future updates are handled properly. This is mostly supposed to be a protective measure around when people install RPMs directly from Koji or similar.

Essentially what is happening here is that there are module streams that are a default stream for Rawhide. Since module streams "win" over bare RPMs, the default streams are being marked enabled during the distro-sync and are installing those packages.

This doesn't happen on F29 right now because only Rawhide has default streams for those modules.

I agree with Daniel that this is NOTABUG, but I think we probably need to figure out how to communicate that to the user better.

Comment 8 Adam Williamson 2018-09-18 22:06:21 UTC
"Essentially what is happening here is that there are module streams that are a default stream for Rawhide."

ah, ok. I did not know that was happening and had not caught any announcements of it. So to be clear, that means that when you try to install a package that is in one of these 'default streams', you will get the package *from the module* in preference to the package from the non-modular distro, and that module and stream will be automatically enabled?

Comment 9 Stephen Gallagher 2018-09-18 22:12:30 UTC
Yes. That is correct. And you’re right, we failed to announce that.

Comment 10 Adam Williamson 2018-09-18 22:23:03 UTC
Another question occurs: I can't find that PackageKit calls set_modules_enabled_by_pkgset anywhere. So, what happens if we run an update like this with PackageKit (e.g. gnome-software)?

Comment 11 Jaroslav Mracek 2018-09-24 13:57:49 UTC
According to my investigation, packages installed from default module by microdnf trigger enablement of module. Unfortunately this is not true for PackageKit. Tested with command "pkcon install dwm". I think that people from PackageKit can investigate the issue more closely and find the issue.

Comment 12 Adam Williamson 2018-09-24 15:24:37 UTC
That certainly sounds like a bug, but it's not *this* bug.

After the explanation by sgallagh, Kevin and I understand and are fine with the actual dnf *behaviour* here, but we still think it doesn't explain what's going on very well. If you just run 'dnf upgrade' and see this, it's not at all clear that modules are getting enabled to update packages which are now in default module streams. Thus this bug is now a request for dnf to explain what's going on a little more clearly.

We should certainly file a separate bug if PK installs the package from the module but does not enable the module, in this case.

Comment 13 Adam Williamson 2018-09-25 01:36:45 UTC
I filed https://bugzilla.redhat.com/show_bug.cgi?id=1632518 for the bug whereby PackageKit does not enable the module stream when installing a package from a default module stream, as jmracek reported in #c11. I also suspect dnfdragora does not do this; I'll test that.

Comment 14 Jaroslav Mracek 2019-03-22 10:48:45 UTC
I believe that DNF works here like expected -> closing

Comment 15 Adam Williamson 2019-03-22 16:27:55 UTC
Um. Did you read comment #12? It specifically says "After the explanation by sgallagh, Kevin and I understand and are fine with the actual dnf *behaviour* here, but we still think it doesn't explain what's going on very well. If you just run 'dnf upgrade' and see this, it's not at all clear that modules are getting enabled to update packages which are now in default module streams. Thus this bug is now a request for dnf to explain what's going on a little more clearly."

Comment 16 Jaroslav Mracek 2019-04-01 19:16:31 UTC
Please could you provide an example of the output that you would expect to see? Thanks a lot.

Comment 17 Kevin Fenzi 2019-04-01 19:32:42 UTC
I'd like to see dnf say something like: 

package xyz is in default module abc, installing package ayx version xxx from module abc and marking module abc enabled.

Comment 18 Adam Williamson 2019-04-01 19:37:54 UTC
+1 kevin.

Comment 19 Stephen Gallagher 2019-04-02 14:28:09 UTC
If it needs to be shorter, perhaps "Enabling default module stream module:stream for packages ayx[, ayz, ...]"

In general, I agree that this would be helpful to the user.

Comment 20 Stephen Gallagher 2019-04-02 14:28:45 UTC
If it needs to be shorter, perhaps "Enabling default module stream module:stream for packages ayx[, ayz, ...]"

In general, I agree that this would be helpful to the user.

Comment 21 Jaroslav Mracek 2019-04-02 15:42:47 UTC
I would recommend to log it using debug level. The list could be ling and it is according other workflows in dnf. Do you agree?

Comment 22 Adam Williamson 2019-04-03 01:06:08 UTC
No. What we want is for regular users to be alerted to this happening so they understand what is going on. Debug logging does not achieve that.

I'm not sure what you mean by "it is according other workflows in dnf" - can you clarify?

Comment 23 Jaroslav Mracek 2019-04-04 11:22:30 UTC
DNF move many information about transaction details into debug logging level. We prefer to provide a single transaction table that includes essential information for most of users. Also according to my experience, information printed outside of transaction table is often ignored. I believe that we should keep that strategy to not overload users.

Comment 24 Ben Cotton 2019-08-13 17:04:12 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 25 Ben Cotton 2019-08-13 18:55:27 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.


Note You need to log in before you can comment on or make changes to this bug.