Bug 1616167

I can reproduce this on Fedora 29 as well with the following steps:

1. `sudo buildah from --name bz1616167 registry.fedoraproject.org/fedora:29`
2. `sudo buildah run -t bz1616167 /usr/bin/bash`
(All further steps are inside the container)
3. `dnf -y update` (Get all the latest packages, including the modular repos)
4. `dnf module install nodejs:8` (Get an older Node.js than included in the ursine repo)
5. `dnf update --disablerepo=\*modular`

Actual results:
Last metadata expiration check: 0:05:55 ago on Wed Aug 15 10:43:54 2018.
Dependencies resolved.
================================================================================
 Package                     Arch    Version                     Repository
                                                                           Size
================================================================================
Upgrading:
 nodejs                      x86_64  1:10.8.0-1.fc29             rawhide  6.1 M
 npm                         x86_64  1:6.2.0-1.10.8.0.1.fc29     rawhide  3.7 M
Installing dependencies:
 libicu                      x86_64  62.1-2.fc29                 rawhide  8.9 M
 python2                     x86_64  2.7.15-6.fc29               rawhide   48 k
 python2-libs                x86_64  2.7.15-6.fc29               rawhide  6.3 M
 python2-pip                 noarch  18.0-2.fc29                 rawhide  1.9 M
 python2-setuptools          noarch  39.2.0-6.fc29               rawhide  629 k
Installing weak dependencies:
 python-unversioned-command  noarch  2.7.15-6.fc29               rawhide   13 k

Transaction Summary
================================================================================
Install  6 Packages
Upgrade  2 Packages

Total download size: 28 M
Is this ok [y/N]: 


Expected result: the module contents must be left alone.

I would call this a blocker for F29, because a module installed from a repository with skip_if_unavailable=True could be unintentionally upgraded due to this if the repository became unreachable.

Proposed as a Blocker for 29-beta by Fedora user sgallagh using the blocker tracking app because:

 "The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager)."

If the repo becomes unavailable or corrupt (missing its module metadata), it's possible that users who intended to remain on a module stream of an older version than is available in the non-modular repos might have it upgraded unexpectedly.

We should probably cover this in the test cases?

I tried to reproduce the bug with using the procedure described in #1 by contyk, but unfortunately, I cannot reproduce it as described.

My procedure went like this:

1. I did not have any dwm package installed previously, so I used the `dnf module install dwm:6.0/user` command to install it, which went ok and dnf enabled the selected module and the installed it onto my system.

2. Then, when I try to update by using `dnf update` I get the following warning:
====
Problem 1: cannot install the best update candidate for package dwm-6.0-1.module_1997+c375c79c.x86_64
  - package dwm-6.1-8.module_1995+c3e93812.x86_64 is disabled
 Problem 2: cannot install the best update candidate for package dwm-user-6.0-1.module_1997+c375c79c.x86_64
  - package dwm-user-6.1-8.module_1995+c3e93812.x86_64 is disabled
====
This is something I do not understand, because I would expect that I will not be bothered by information about modules, I have disabled or have not enabled.

3. When I try to disable the modular repos for updates and updates-testing, and I try to update using `sudo dnf update --disablerepo updates-modular --disablerepo updates-testing-modular`, I am getting the same warning ...  ergo dnf does not let me update to a newer version because it is disabled.

4. When I try to disable every modular repo, like this `dnf update --disablerepo updates-modular --disablerepo updates-testing-modular --disablerepo fedora-modular`, I am not getting any warning any more, however I still cannot upgrade to newer version, because this time, my dnf says:
====
Last metadata expiration check: 0:36:52 ago on Mon 20 Aug 2018 02:55:21 PM CEST.
Dependencies resolved.
Nothing to do.
Complete!
====

I believe that there isn't any suitable dwm package in the non-modular repository any more and therefore, I cannot override it from there.

5. When I really want to update dwm to the newer version, I have to enable the 6.1 module and distro-sync the system. This the older version gets disabled and the newer gets enabled. 

I believe that in this particular case, the dnf behaviour is correct. 
I will retry with the nodejs packages to reproduce Stephen's case.

I could reproduce this bug with nodejs packages. I will write the test case based on those packages then.

What about to install a virtual RPM to protect modular content. Even rpm will be unable to install incorrect non modular content over modular. This approach is even not new to Linux world.

Discussed during the 2018-08-20 blocker review meeting: [1]

The decision to classify this bug as an "AcceptedBlocker" was made as it violates the following criteria:

"The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type"; currently it means an update may be inappropriately installed in a fairly common scenario.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-08-20/f29-blocker-review.2018-08-20-16.00.txt

Lukas, note the ursine dwm package was retired in F29.  The nodejs example is still valid.

It's my understanding that solving this requires significant work and DNF team can't commit to having it done in a reasonable timeframe for F29 Beta (and possibly not for Final). On that basis, I've escalated this to FESCo for consideration of the best path forward:

https://pagure.io/fesco/issue/1974

please add any relevant comments (including corrections for anything I got wrong) to that ticket, thanks!

Quoting my reply to the FESCo ticket:


After a good bit of investigation we think that it's not going to affect very many people during the Beta cycle. The most likely situation to encounter this issue would be in cases where a user did `dnf update --enablerepo=updates-testing-modular` and then later did `dnf update` before the module they had updated had ended up in the stable repo. Since the u-t repos are available by default in Beta, we probably won't have people actually hitting this.

We considered two workarounds:
* Ensure that the repodata includes all modulemd for any module that has ever been present in the stream. This would address cases where users who don't update frequently-enough might end up with packages on their system from a release that was post-GA but not the latest module update, and thus DNF could lose track of it if the new module no longer included that package *and* the non-modular repos did. The downsides here were complexity in maintaining that information and a potential to rapidly grow the repodata size.
* Include a special tag in all packages built as part of a module so that DNF could check and refuse to replace any package containing this tag with a package lacking it. This would require specialized handling in DNF, would break the planned "hotfix repo" functionality and would also require a mass-rebuild of all modules post-Freeze, so that was not an option either. (Note that we HAVE implemented that tag for F30 and it was part of the mass-rebuild I did there just after we branched, so this remains in our back pocket for *future* Fedora releases.)

The full solution and proper solution will be to have DNF keep a copy of the modulemd matching any installed RPMs in a local database (similar to how it retains traditional packages in the RPMDB) so it can always identify which RPMs on the system belong to a module stream and act accordingly.

So, my recommendation would be for us not to block Beta on this issue, but we should consider blocking GA on it.

Discussed at 2018-09-10 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2018-09-10/f29-blocker-review.2018-09-10-16.01.html . As per FESCo's decision:

https://pagure.io/fesco/issue/1974#comment-530640

this is now rejected as a Beta blocker and accepted as a Final blocker.

In February 2018, Petr Sabata, Stephen Tweedie and me met to decide on how dnf should handle modules, because we needed clear rules how dnf should behave.

There were 2 options:
1) soft behavior (the current one)
Module streams are virtual repos.
Module versions are additions to the virtual repos.
Module profiles are comps groups.
You can't install any particular module version; upgrades or downgrades are also impossible. You simply work with the RPM package set given by the module metadata. What matters is the RPM NEVRAs.
You can install any combination of RPMs from the stream (from any versions) and hotfix repos and it's all valid from modularity perspective.
Installing module profiles is also soft - missing RPMs are ok.
Profile definitions are taken from the latest modulemd in a stream.


2) hard behavior
Module versions would be close to meta-packages.
Module versions could be installed incl. all artifacts.
Module version consistency could be enforced. 
The disadvantage was that hotfixes would have to contain module metadata and the hotfix versioning would have to fit into modularity upgrade path.


The decision was to proceed with option 1.
This has several implications:

You cannot say which module version is installed on your system.
An edge case is when you have all modular RPMs overriden with hotfixes and you can't pair your system with any modulemd by RPM NEVRA.

If DNF starts storing modulemd on disk and consider them in module processing, it may create new problems.

Consider following scenario:
* install fedora 29 modular system
* enable updates-testing
* work with modules
* modulemd from updates-testing get silently stored on your disk
* disable updates-testing
* modulemd stays on disk (dnf doesn't know if you have disabled a repo permanently, temporarily or by an accident)
* modulemd can be completely broken and affect:
  * installing profiles (modulemd has most likely higher version that anything in 'fedora' or 'updates' repos)
  * module dependency resolution (modulemd can depend on something that is not in stable yet)


I'm afraid that storing modulemd on disk would create more problems than it solves.
Modularity was designed to have soft behavior and the implementation is according to that.


I see an option that might work:
DNF could refuse to perform any installs/upgrades etc. if there's a single enabled stream missing from module metadata in currently available repositories.
It's a clean solution from the perspective that it uses only data from repos and no random, potentially broken and unwanted data from disk.
The disadvantage is that it might error our frequently. I cannot estimate that at the moment.

(In reply to Daniel Mach from comment #12)
> 
> You cannot say which module version is installed on your system.
> An edge case is when you have all modular RPMs overriden with hotfixes and
> you can't pair your system with any modulemd by RPM NEVRA.
> 

Incorrect: we have /etc/dnf/modules.d/*.module for any enabled module. We can retain a copy of the modulemd for the highest version of the name-stream-context described in any enabled stream in that directory.

> If DNF starts storing modulemd on disk and consider them in module
> processing, it may create new problems.
> 
> Consider following scenario:
> * install fedora 29 modular system
> * enable updates-testing
> * work with modules
> * modulemd from updates-testing get silently stored on your disk
> * disable updates-testing
> * modulemd stays on disk (dnf doesn't know if you have disabled a repo
> permanently, temporarily or by an accident)
> * modulemd can be completely broken and affect:
>   * installing profiles (modulemd has most likely higher version that
> anything in 'fedora' or 'updates' repos)
>   * module dependency resolution (modulemd can depend on something that is
> not in stable yet)
> 

This is detectable if we have the installed modulemd locally. If nothing else, we can see that there exist installed packages that have a higher "V" than anything that can be seen in the enabled repositories and inform the user that the transaction cannot be completed safely and that they should check whether any of their previously-enabled repositories are no longer available.

> 
> I'm afraid that storing modulemd on disk would create more problems than it
> solves.

I don't think this conclusion arises from the discussion above. These problems are surmountable.


> Modularity was designed to have soft behavior and the implementation is
> according to that.
> 
> 
> I see an option that might work:
> DNF could refuse to perform any installs/upgrades etc. if there's a single
> enabled stream missing from module metadata in currently available
> repositories.
> It's a clean solution from the perspective that it uses only data from repos
> and no random, potentially broken and unwanted data from disk.
> The disadvantage is that it might error our frequently. I cannot estimate
> that at the moment.

I think this should be the fallback for those cases that we cannot handle in a deterministic way like I suggest above. We should always fail safe.

I generally agree with the behaviour that sgallagh describes. The "soft behaviour" makes no sense to me — it seems to defeat the whole raison d'être for modules.

> If nothing else, we can see that there exist installed packages that have a higher "V" than anything that can be seen in the enabled repositories

I don't understand why comparing versions is useful. I expect instead that if something comes from a module stream, it can only be upgraded by packages in that module stream.

We met with the DNF team twice this week to discuss this (and other somewhat related issues), our long-term plan and the MVP for F29 GA to get this resolved.

General:
We're filtering non-modular (ursine) RPMs from the transaction if the same RPM name is available in an enabled (implicitly or explicitly) modules so that modular content is always selected when applicable.  This allows us to override conflicting content with the set the user has selected.

Problem A:
Basically the issue described in this bug.  Without knowing which RPMs are modular, DNF cannot do proper filtering.  This information is available in modeuldms which are currently only read from the repodata.  By disabling a repository from which you installed your packages, you lose access to this information and the RPMs might be replaced by something else, such as ursine packages with higher NVRs that are still available.  That's this bug and we need to fix this.

Problem B:
We could potentially lose access to modulemds and still have the packages they include available.  If this package set was in a disabled module and originally ignored, it could now be considered in transactions.  This probably shouldn't normally happen in our repos but could be the case with custom or third party repos or copied repos that don't sync the modular repodata.  This is not covered by this bug, nor it should be.

Solutions:
Remembering modulemds for enabled modules and possibly all installed modular packages would solve problem A as DNF would never forget what RPMs come from what modules.  As an added benefit, it would allow us to perform basic module queries such as "module list" and "module info" even without having access to the original repodata.

We believe a basic variant of this feature could be implemented for F29 GA.  There are some complications, such as remembering defective modulemds from updates-testing, determining how to clean unneded modulemds, report error states in some reasonable manner, or provide recovery mechanisms.  I do not think these need to be resolved by GA and could be added later, so that's not part of our MVP.

To address problem B, we label all modular RPMs with a special header.  We do that even today but this information isn't readily available in the repodata.  We'll have to fix that at some point but as it could be considered a "corner case" and isn't affecting Fedora directly, that's more of a long-term effort.

The DNF team will work on the MVP and will document all the known problems users could face.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #14)
> > If nothing else, we can see that there exist installed packages that have a higher "V" than anything that can be seen in the enabled repositories
> 
> I don't understand why comparing versions is useful. I expect instead that
> if something comes from a module stream, it can only be upgraded by packages
> in that module stream.

Yes, you trimmed out the part where I said "This is detectable if we have the installed modulemd locally." In other words, as long as we have that modulemd that says "this RPM is part of this module and stream", if we see that it's got a higher NVR than any RPM *currently* visible in the repodata (e.g. u-t is not enabled) it will still know that it's modular and not muck with it. I may have overstated the result of that; it almost certainly doesn't have to abort the transaction unless some other strange dependency interaction is occurring.

From the FESCo meeting, we have agreed on a FESCo special final blocker on this, try and get more estimate input from dnf team. Code freeze is tomorrow, so this need priority.

Estimated delivery for this feature is end of January.

(In reply to Daniel Mach from comment #18)
> Estimated delivery for this feature is end of January.

Are you saying the MVP discussed last week will take four months to code?

Discussed during the 2018-10-15 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" was made as FESCo has voted to drop their requirements around this issue for Final, so it is no longer a blocker per their decision.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-10-15/f29-blocker-review.2018-10-15-16.00.txt

By reading the first comment, this instantly reminds me of a slightly similar bug that I reported here:

https://bugzilla.redhat.com/show_bug.cgi?id=1644241

Same applies if you install, update or re-install packages from command line that has the same versioning as the packages offered within the repo.

In our use-case we need to alter (by using an automated process) some Fedora packages to do "things" with packages that get delivered.

You cam issue dnf install somepackage.rpm or dnf update somepackage.rpm or dnf reinstall somepackage.rpm from the command line.

The packages get either installed, updated or reinstalled properly.

After issuing dnf update afterwards, dnf enforces reinstallation of the packages with same versioning from repository and therefore overwriting our customized software.

This is a big regression (didn't happen before) and we can't deliver anything right now because there is no way to bypass this or silence DNF from enforcing reinstallation.

We don't use any modularity or modules (as in terms used by Fedora). We simply pass a list of files to the dnf command line and have it process the stuff. Used to work before. Sounds quite similar to the first post.

It's not at all the same, though, as this bug is specific to modularity bits. Yes, really.

Your case sounds like a 'doctor, it hurts', to be honest. If you're changing the packages, they're *not the same packages*. Have the thing that changes them also change the NVR, just by sticking a custom string in the dist tag or something at least.

(In reply to Adam Williamson from comment #22)
> Your case sounds like a 'doctor, it hurts', to be honest. If you're changing
> the packages, they're *not the same packages*.

Indeed! Still doesn't explain, why DNF wants to reinstall the packages from the official Fedora repos. This wasn't the case when we first migrated to Fedora with Fedora 16 and this wasn't the case when the transformation from YUM to DNF happened over the years. This situation appeared quite recently and exists now on Fedora 29. There is no signs within the DNF Changelogs or Features that explains this behavior. At least I wasn't able to find one.

> Have the thing that changes them also change the NVR, just by sticking a custom
> string in the dist tag or something at least.

How about keeping some sort of compatibility ? At least - if this is a wanted feature - provide some sort of config to disable it.

Basicly this is quite the same as the modularity thing. There is no difference if you install a version from modularity repos or command line repo or simply as is by applying the filename to dnf, disable it (the repo), then use update, which then enforces reinstallation of what the other repositories think to be right. Overwriting the software that already is present on the system. I am not speaking about updating here.

This is a change of behaviour of how DNF acted on Fedora 28 and acts now on Fedora 29.

Yes, but you can't expect absolutely all behaviour of very complex applications to remain the same over time - even the behaviour in odd niche cases with no clear justification, like this one. I don't believe we ever made any kind of promise that the behaviour yum and old-dnf had in this situation was intended and could be relied upon, have we? It's an odd and unusual situation and the behaviour was probably never particularly 'designed' in the first place, it's just what yum happened to do when you put it in this weird position. Again, I doubt the dnf behaviour is 'designed' exactly, it's likely just a sort of byproduct of some intended behaviour that was designed for a *different*, more reasonable use case.

dmach, is this still planned to be implemented by January? I'd like to know, as there's a test in openQA that is intended to validate the expected behaviour here, but of course as the expected behaviour hasn't been implemented yet, it just fails every time. I'm not sure whether we should temporarily disable it, or just remove it.

We discussed this issue with Petr Sabata and other modularity team members and we've made a commitment to start this work in January as our top priority.
End of January is still a realistic delivery date.

Thanks. I guess we'll keep this test around but temporarily disable it until you say it should be working.

*** Bug 1672389 has been marked as a duplicate of this bug. ***

So, it's now February :) What's current status on this? Thanks!

clearing RejectedBlocker so we can start up this circus again for F30...

Discussed during the 2019-02-11 blocker review meeting: [1]

The decision to punt (delay decision) was made:

"punt to FESCo - as we did for Fedora 29, due to the issues with delivery time on this bug, we will pass this to FESCo to decide what the requirements should be for F30 Beta and Final"

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2019-02-11/f30-blocker-review.2019-02-11-17.13.log.txt

I am removing this from F30 Beta Proposed Blockers as per FESCO decision [0]:

>   * AGREED: As we're fairly sure this wouldn't be fixed in time for F30,
>     we're moving this to F31. Modularity & DNF folks with prepare a
>     specific implementation plan and will update the ticket (+6, 0, -0)
>     (contyk, 15:20:34)

I'll propose it as F31 Beta Blocker once we have tracking bugs for F31.

[0] https://pagure.io/fesco/issue/1974

"I'll propose it as F31 Beta Blocker once we have tracking bugs for F31."

We already do ;) I create them two releases in advance, the F31 trackers have existed since F29 release.

Petr Sabata, Jaroslav Mracek and me are going to meet tomorrow and discuss this bug among other things.
We are not probably going to store anything in the database, but we'll store sufficient module metadata on disk so the system keeps working when repositories disappear or get disabled.

For the record, FESCo decided to once again punt this, to F31:

https://pagure.io/fesco/issue/1974#comment-554725

So per FESCo decision this bug is not a Beta or Final blocker for Fedora 30.

This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Clearing RejectedBlocker for F31.

Rephrased summary to match what is going to be delivered.
The work is in progress, we have a proof of concept for 1 fail-safe mechanism already.

The work is already part of dnf-4.2.7

I think we can realistically call this -1 blocker for F31. It's being (correctly) treated as a high priority by the DNF team, but there's nothing about this that should prevent us from shipping Fedora. It's basically about protecting against unexpected behavior rendering the system unable to update. If any such occurrences actually happen, we should probably address them individually as blockers instead of this fail-safe approach.

Discussed during the 2019-07-15 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" was made as we shipped the last two releases without this, so it seems arbitrary to start blocking on it now just because it might actually be ready this cycle. Clearly we've established a precedent that it's OK to release Fedora without it. We'd still like to see it done, though.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-07-15/f31-blocker-review.2019-07-15-16.05.txt

FEDORA-2019-049b4836a6 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-049b4836a6

dnf-4.2.11-2.fc30, dnf-plugins-core-4.0.10-1.fc30, libdnf-0.35.5-2.fc30, librepo-1.10.6-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-049b4836a6

dnf-4.2.11-2.fc30, dnf-plugins-core-4.0.10-1.fc30, libdnf-0.35.5-2.fc30, librepo-1.10.6-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.