Created attachment 1622735 [details]
dnf logs from dnf system-upgrade -v reboot --debugsolver

The output (./debugdata) from

dnf system-upgrade -v download --refresh --allowerasing --releasever=31 --debugsolver

is in the file Bug_1758588_debugadata_download.tar.gz shared here:

https://drive.google.com/file/d/1D9Q0TD-xZAenCulTkD9SRLr6GFIBuLVb/view?usp=sharing

Running 

dnf system-upgrade -v reboot --debugsolver

did not create a debugdata dir anywhere. Therefore I am attaching the dnf logs from /var/log/ in file Bug_1758588_dnf_logs.tar.gz

Possible workaround is to dnf remove the pkgs causing it and rerun system-upgrade download and reboot

Yes, it really looks like the transaction was resolved differently for "download" and "upgrade" phases and the root cause of it is here:

download command was run with --allowerasing and this switch gives solver more options how to solve the transaction:
2019-10-04T20:53:08Z DDEBUG Command: dnf system-upgrade -v download --refresh --allowerasing --releasever=31 --debugsolver

after reboot, upgrade command was run without this switch:
2019-10-04T21:38:04Z DDEBUG Command: dnf system-upgrade upgrade 

Dnf needs to remember switches used in the download phase and reuse them with upgrade. This is bug on the dnf side and will be fixed.

Well, it may be a bit more complicated. The plugin definitely tries to store the allowerasing state and load it before running the upgrade:

https://github.com/rpm-software-management/dnf-plugins-extras/blob/master/plugins/system_upgrade.py#L578 (stores it after download)
https://github.com/rpm-software-management/dnf-plugins-extras/blob/master/plugins/system_upgrade.py#L416 (loads it in configure_upgrade, before the upgrade runs)

what I wonder, though, is whether this being a part of `self.cli.demands` rather than `self.base.conf` like most of the other options is a problem? I'm wondering if the design of dnf itself is that the CLI code passes that value to the base functions - like at https://github.com/rpm-software-management/dnf/blob/master/dnf/cli/commands/shell.py#L246 , for e.g. - rather than the base functions reading it directly. However in this system_upgrade plugin we aren't running through the CLI code, we just set `self.cli.demands.allow_erasing` and then call `self.base.install(pkgspec, reponame=repo_id)` for each package in the list directly.

I'm not sure how `base.resolve()` ultimately gets run on the system upgrade path - it must be plugin magic? - but my guess is that, when it does get run, the value of `self.cli.demands.allow_erasing` is not passed to it.

Proposed as a Blocker for 31-final by Fedora user chrismurphy using the blocker tracking app because:

 Propose as a "0Day/PreviousRelease" blocker; while it can be fixed at any time, if accepted it means a stable fix must be available in F29/F30 by F31 release day.
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Normal.2C_0-Day_and_Previous_Release_blockers

Per beta criterion:
"For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from a fully updated, clean default installation of each of the last two stable Fedora releases with that package set installed. "

Seeing as the user is not flagged at the download step that the upgrade will fail, and then the reboot/upgrade step does fail, I propose that violates the criterion.

This is a regression because --allowerasing has worked in previous releases.

So, I've been poking at this today. So far:

1. I can reproduce it. My minimal reproducer so far: do a minimal install of F30, do 'dnf -y install cantor', do 'dnf --releasever=31 --allowerasing system-upgrade download', do 'dnf system-upgrade reboot'

2. When doing the above, /var/lib/dnf/system-upgrade.json has "allow_erasing": true

As a side note, one of the actual deps issue here, and the one that installing cantor triggers for testing, is our old friend https://bugzilla.redhat.com/show_bug.cgi?id=1747408 , AKA the libgit2 module upgrade bug: kf5-ktexteditor requires libgit, which triggers that bug. So the 'real' way to fix the upgrade for that issue would be 'dnf module reset libgit2'. The OP has a few other issues, though, including one to do with dnf-yum and dnf package versions in F30 and F31 that I'll file separately.

I'll continue digging into this tomorrow, I gotta go for dinner. I'll start logging what actually happens in the upgrade transaction and try to see why it's including cantor.

Hum. AFAICS, allowerasing is always being enabled here. I added logging statements, and it's properly set during the upgrade attempt, and it is set True when resolve() is called.

My next target is: exactly how system-upgrade implements the upgrade process. It basically takes the list of packages that was found by the `download` step and calls dnf Base method `install()` on each of them. I'm guessing perhaps cantor gets pulled into the set during that process, and somehow allowerasing doesn't result in it being kicked out again at resolve().

So I threw some logging in there, and it appears that when the actual upgrade step is trying to happen, cantor appears in the transaction when libqalculate does:

Oct 09 11:57:59 localhost.localdomain dnf[640]: XXX adding pkg spec kf5-sonnet-ui-5.61.0-1.fc31.x86_64
Oct 09 11:57:59 localhost.localdomain dnf[640]: XXX resolve called with allowerasing True
Oct 09 11:58:00 localhost.localdomain dnf[640]: XXX adding pkg spec libpwquality-1.4.1-1.fc31.x86_64
Oct 09 11:58:00 localhost.localdomain dnf[640]: XXX resolve called with allowerasing True
Oct 09 11:58:00 localhost.localdomain dnf[640]: XXX adding pkg spec kf5-syntax-highlighting-5.61.0-1.fc31.x86_64
Oct 09 11:58:00 localhost.localdomain dnf[640]: XXX resolve called with allowerasing True
Oct 09 11:58:01 localhost.localdomain dnf[640]: XXX adding pkg spec libqalculate-3.1.0-2.fc31.x86_64
Oct 09 11:58:01 localhost.localdomain dnf[640]: XXX resolve called with allowerasing True
Oct 09 11:58:01 localhost.localdomain dnf[640]: XXX cantor now in install set!

to do that I made base.install() resolve (with allowerasing=True) every time it is called and then check whether 'cantor' appears in self.transaction.install_set.

Looks like cantor-libs requires libqalculate, and there's an soname bump in F31, so that makes sense as far it goes: the new libqalculate soname is pulled into the transaction, so DNF notices that the installed 'cantor-libs' relies on the old soname and pulls the newer one from F31 into the transaction. So that's how cantor makes it into the transaction even though it's not listed directly in /var/lib/dnf/system-upgrade.json . The next question is, why is it kicked out of the transaction (or never included in it) at the `download` stage, but not at the `upgrade` stage?

On that question, I noticed a strange thing about the initial `download` phase transaction. Here's what it decides:

=====

Upgrading:
 libgit2                                  x86_64     0.27.8-1.module_f31+3326+1efaac5b       fedora-modular     412 k
[... hundreds of other things ...]
Removing dependent packages:
 cantor                                   x86_64     18.12.3-1.fc30                          @updates           3.1 M
 cantor-libs                              x86_64     18.12.3-1.fc30                          @updates           3.3 M
 kf5-ktexteditor                          x86_64     5.59.0-1.fc30                           @updates            14 M
Downgrading:
 python3-unbound                          x86_64     1.9.3-1.fc31                            fedora             100 k
 unbound-libs                             x86_64     1.9.3-1.fc31                            fedora             522 k

Transaction Summary
======================================================================================================================
Install     51 Packages
Upgrade    537 Packages
Remove       3 Packages
Downgrade    2 Packages

=====

so, when you look at it carefully, that's *odd*, isn't it? Why is it *only* kicking cantor, cantor-libs and kf5-ktexteditor out of the transaction? cantor and cantor-libs go because they depend on kf5-ktexteditor, okay. But why is kf5-ktexteditor being kicked out?

I'm...wondering if somehow dnf is getting confused about whether libgit2 is excluded or not? At some point it decides libgit2 is excluded and therefore kicks its dependency - kf5-ktexteditor - out of the transaction, but then somehow decides to include libgit2 after all but *not* pull kf5-ktexteditor back in?

unfortunately can't find anything useful in the solver debug info to explain exactly what goes on in the download transaction wrt kf5-ktexteditor , and haven't drunk enough yet today to start understanding libdnf code, so...I'll see if the devs can take it from here.

Oh, one more nugget - the issue here seems to be specific to distro-sync mode (which system-upgrade uses by default). If I do `dnf --releasever=31 --allowerasing upgrade`, I get a different result:

 Problem: cannot install the best update candidate for package kf5-ktexteditor-5.59.0-1.fc30.x86_64
  - problem with installed package kf5-ktexteditor-5.59.0-1.fc30.x86_64
  - package kf5-ktexteditor-5.61.0-1.fc31.x86_64 requires libgit2.so.28()(64bit), but none of the providers can be installed
  - package libgit2-0.28.2-2.module_f31+5411+fa1856a4.x86_64 is excluded
  - package libgit2-0.28.3-1.fc31.x86_64 is excluded
...
Skipping packages with broken dependencies:
 kf5-ktexteditor                                        x86_64                   5.61.0-1.fc31                                         fedora                           2.6 M

Transaction Summary
==============================================================================================================================================================================
Install   51 Packages
Upgrade  537 Packages
Skip       1 Package

which I believe means it plans to just leave kf5-ktexteditor at its F30 package state. I'm not sure what it plans to do about the dependency on libqalculate, but anyway.

By comparison if I do `dnf --releasever=31 --allowerasing distro-sync`, I get the same result as with `dnf --releasever=31 --allowerasing system-upgrade download` - it wants to erase cantor and kf5-ktexteditor.

OOH! Okay, I think I have figured this one out. At least I have a plausible theory.

So, okay, let's go over it from square 1. It is important to understand how system-upgrade operates.

The system-upgrade `download` phase sets various options, then calls (by default) `self.base.distro_sync()` to populate a transaction. i.e. it does a distro-sync *operation*. To dnf/libdnf, distro-sync is not a configuration option or anything, it's a particular *operation*. Then that transaction gets resolved and run, which results in all the packages being download, and system-upgrade reads out the list of packages that was included in the transaction and stashes it in a state file, along with all the configuration options like gpgcheck, allowerasing, best and so on.

The system-upgrade `upgrade` phase does *not* do a distro-sync operation. What it does is read in all the options and the list of packages from the state file, set all the options, and then call `self.base.install()` on each of the packages in the list.

So, here's what I think happens. During the *download* phase, we are doing a `distro-sync` operation, which essentially adds a constraint to the rules for solving the transaction - a constraint that libdnf/libsolv call `RULE_DISTUPGRADE`, and is defined as follows:

*SOLVER_DISTUPGRADE*::
Update the matching installed packages to the best version included in one of the repositories. After this operation, all come from one of the available repositories except orphaned packages. Orphaned packages are packages that have no relation to the packages in the repositories, i.e. no package in the repositories have the same name or obsolete the orphaned package. This action brings the installed packages in sync with the ones in the repository. By default it also turns of arch/vendor/version locking for the affected packages to simulate a fresh installation. This means that distupgrade can actually downgrade packages if only lower versions of a package are available in the repositories. You can tweak this behavior with the SOLVER_FLAG_DUP_ solver flags.

Note especially "After this operation, all come from one of the available repositories except orphaned packages." So, I believe kf5-ktexteditor is excluded from the `download` phase transaction because of this constraint. As we can see in #c12, its F31 build cannot be included in the transaction (basically Because Modularity Stream Issues - we don't have a libgit2 0.28 stream enabled). Because we're operating under SOLVER_DISTUPGRADE rules we're not allowed to just keep the fc30 kf5-ktexteditor, because SOLVER_DISTUPGRADE requires that if a newer version is available in the distupgrade repo we *must* update to it. However, because we have --allowerasing, libsolv/libdnf decide not to just fail on this, but to 'resolve' it by kicking kf5-ktexteditor and its dependencies out of the transaction entirely. This results in a transaction that meets all the requirements. So this transaction goes ahead and runs, and the system-upgrade plugin writes out its list of packages to the state file - a list that does not include kf5-ktexteditor or cantor. However, there's no mechanism that records the fact that those packages were actively *excluded* from the transaction, or anything like that.

We now head to the `upgrade` phase. As we noted earlier, this phase does *not* run a `distro-sync` operation, and there is no such thing as a distro-sync flag or config setting at the dnf level. What this phase does instead is create the transaction simply by calling `install()` on each package listed in the state file, then resolve that transaction. Because we're not doing a `distro-sync` operation, this time the transaction is resolved **WITHOUT** the `SOLVER_DISTUPGRADE` constraint.

kf5-ktexteditor and cantor and cantor-libs are never explicitly added to the transaction. But, as we saw in #c9, they get pulled in via a bumped library dependency - libqalculate *is* listed in the state file, so it gets pulled into the transaction, and libdnf notices that this breaks the dependencies of cantor-libs, so it pulls the updated F31 cantor-libs and cantor into the transaction. Now unlike at the `download` stage, there is no `SOLVER_DISTUPGRADE` constraint in operation, so it's valid for libdnf to keep the *old F30* kf5-ktexteditor package to satisfy cantor-libs' dependency on kf5-ktexteditor. Thus at the `upgrade` phase we wind up with a valid transaction that *includes* cantor and cantor-libs.

And voila, that's the explosion. The `download` transaction's additional constraint wound up in cantor, cantor-libs and kf5-ktexteditor being excluded, so they weren't downloaded. The `upgrade` transaction doesn't have that constraint, pulls them in, and so it blows up because the packages aren't there and it can't download them.

assigning back to the plugin as if my theory is correct, the problem here is ultimately in the way the plugin runs these two different transactions, libdnf is only doing what it's being asked to do.

I believe that the proper fix of the system-upgrade should be:

1. prepare transaction in the `download` phase and store it as a whole into the history database. (At the moment system-upgrade stores data into system-upgrade.json file and stores only "forward" operations - list of the packages that should be installed on the upgraded system. It doesn't trace any removals.)

2. after reboot do something like redo on this saved transaction - load it from the database and run.

This approach will also fix the problem with incorrect reasons of newly installed dependencies - bz#1574930 and will enable "offline" updates (or basically any transactions could be run offline after reboot).
OTOH the proper solution would require fixing it on all levels - system-upgrade, dnf itself and probably also libdnf enhancements will be required. Attempting to do this after the beta is too risky.

There might be a temporary workaround to this bug in storing also package removals in system-upgrade.json. This should be relatively easy to implement and can be done in the system-upgrade plugin itself.

Yeah, I was gonna suggest the same thing (track removals as well as installs). I even got started trying it yesterday, but shadowing the install code *exactly* doesn't work as attempting to use `pkg.repo.id` in the dict fails with "KeyError: '@System'". I didn't have time to debug that yesterday, not sure if the fix is just to use a simple list instead of a dict for removals or what.

https://github.com/rpm-software-management/dnf-plugins-extras/pull/163 at least fixes the simple reproducer I found. I'm about, like, 80% sure it shouldn't cause anything else to blow up horribly? :) That whole 'actually allow us to serialize transactions properly' thing sure sounds like a good idea though.

+1 blocker

+1 blocker, because dnf system-upgrade doesn't work correctly when executed properly (the way it is supposed to be used)

There is something very wrong with bugzilla interface caching. Revert status change.

I still don't think this violates the criteria, as AFAIK it does not affect any default release-blocking package set of F29 or F30.

Note the fix is still somewhat speculative. We now take all the removals from the `download` phase and add them to the transaction, then take all the installs from the `download` phase and add them to the transaction, then resolve that transaction. I'm not gonna sit here and swear that this will always reproduce the `download` transaction exactly and not do anything surprising, it would not surprise me very much at all if there turns out to be a different corner case where *this* produces an unexpected result.

-1 blocker as it doesn't meet critera, but +1 FE as it would be very nice to fix it.

-1 blocker (doesn't violate criteria, but I'd love to see this fixed), +1 FE

-1 Blocker, +1 FE

FE doesn't make any sense here as the change would be for F29 and F30. We'd send it to F31 too, but that would only be for upgrades *from* F31 so it's not urgent. This is either an AcceptedPreviousReleaseBlocker or nothing.

-1 blocker

If we become aware that it doesn't allow the installation of a release-blocking package set for F29 or F30, we can reconsider.

-1 blocker

Change me to -1 blocker, +1 FE based on the comments that it doesn't affect default package sets

Errr, ignore the "+1 FE part"

FEDORA-2019-daa0986c6d has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-daa0986c6d

Discussed during the 2019-10-14 blocker review meeting: [0]

The decision to classify this bug as a "RejectedBlocker" was made as we are not aware of any case where this problem affects one of the release-blocking F30 package sets, so it does not violate the criterion stated.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-10-14/f31-blocker-review.2019-10-14-16.01.txt

FEDORA-2019-daa0986c6d has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-daa0986c6d

dnf-plugins-extras-4.0.7-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-daa0986c6d

FEDORA-2019-e8db3bef68 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8db3bef68

dnf-plugins-extras-4.0.7-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8db3bef68

dnf-plugins-extras-4.0.7-2.fc30 did not resolve my problem noted on the test mailing list.

Still getting

2019-10-18T23:28:00Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-free-058b175644bc1430/packages/rpmfusion-free-appstream-data-30-1.20181021.fc30.noarch.rpm
2019-10-18T23:28:00Z CRITICAL Package "rpmfusion-free-appstream-data-30-1.20181021.fc30.noarch" from repository "rpmfusion-free" has incorrect checksum
2019-10-18T23:28:00Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-nonfree-c253272f7b309f17/packages/rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch.rpm
2019-10-18T23:28:00Z CRITICAL Package "rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch" from repository "rpmfusion-nonfree" has incorrect checksum

Created attachment 1627498 [details]
Logs of upgrade

Logs requested by Adam

Thanks, Ed.

So, your case seems a bit different. We'll need to dig into it in a bit more detail, but broadly the issue is that a couple of rpmfusion packages show up to be "reinstalled" in the `upgrade` transaction but are not in the `download` transaction:

2019-10-20T15:42:45Z DEBUG ---> Package rpmfusion-free-appstream-data.noarch 30-1.20181021.fc30 will be reinstalled
2019-10-20T15:42:45Z DEBUG ---> Package rpmfusion-nonfree-appstream-data.noarch 30-1.20181021.fc30 will be reinstalled
...
Reinstalling:
 rpmfusion-free-appstream-data               noarch30-1.20181021.fc30                  rpmfusion-free    303 k
 rpmfusion-nonfree-appstream-data            noarch30-1.20181021.fc30                  rpmfusion-nonfree 62 k

We're still in the same category of issue here - the `download` transaction is coming out different to the `upgrade` transaction - but the details seem a bit different. I'm not sure yet *why* the upgrade transaction decides to reinstall those packages.

dnf-plugins-extras-4.0.7-2.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

For the RPMFUSION case : 
https://bugzilla.redhat.com/show_bug.cgi?id=1764169

Clearing CommonBugs as we fixed this ahead of Final.