Bug 1758588 - dnf system-upgrade reboot fails due to depresolv difference with download
Summary: dnf system-upgrade reboot fails due to depresolv difference with download
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf-plugins-extras
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-04 14:43 UTC by Brownsen
Modified: 2019-10-29 01:17 UTC (History)
26 users (show)

Fixed In Version: dnf-plugins-extras-4.0.7-2.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-25 17:01:52 UTC


Attachments (Terms of Use)
dnf logs from dnf system-upgrade -v reboot --debugsolver (979.11 KB, application/gzip)
2019-10-05 13:24 UTC, Brownsen
no flags Details
Logs of upgrade (242.33 KB, application/gzip)
2019-10-20 15:50 UTC, Ed Greshko
no flags Details

Description Brownsen 2019-10-04 14:43:36 UTC
Description of problem:

dnf system-upgrade reboot does not come to the same depresolv conclusion than dnf system-upgrade download. It therefore fails.

Version-Release number of selected component (if applicable):
4.2.11
  Installed: dnf-0:4.2.11-2.fc30.noarch at Thu 03 Oct 2019 10:33:35 AM GMT
  Built    : Fedora Project at Tue 01 Oct 2019 02:12:41 PM GMT

  Installed: rpm-0:4.14.2.1-5.fc30.x86_64 at Sat 31 Aug 2019 12:44:13 PM GMT
  Built    : Fedora Project at Thu 29 Aug 2019 10:46:16 AM GMT

How reproducible:

It is difficult to reproduce as the issue seems to only occur in very rare depresolv constellations.

Steps to Reproduce:
1. dnf system-upgrade clean
2. dnf system-upgrade download --refresh --allowerasing --releasever=31
3. dnf system-upgrade reboot

Actual results:

Step three reboots and starts the upgrade. It fails with the following:

2019-10-03T11:28:47Z INFO Downloading Packages:
2019-10-03T11:28:51Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/updates-testing-9640951d8a5b54c9/packages/plasma-integration-5.16.5
-2.fc31.x86_64.rpm
2019-10-03T11:28:51Z CRITICAL Package "plasma-integration-5.16.5-2.fc31.x86_64" from repository "updates-testing" has incorrect checksum
2019-10-03T11:28:53Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/fedora-3589ee8a7ee1691d/packages/LabPlot-2.5.0-5.fc31.x86_64.rpm
2019-10-03T11:28:53Z CRITICAL Package "LabPlot-2.5.0-5.fc31.x86_64" from repository "fedora" has incorrect checksum
2019-10-03T11:28:54Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/fedora-3589ee8a7ee1691d/packages/cantor-19.04.3-2.fc31.x86_64.rpm
2019-10-03T11:28:54Z CRITICAL Package "cantor-19.04.3-2.fc31.x86_64" from repository "fedora" has incorrect checksum
2019-10-03T11:28:54Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/fedora-3589ee8a7ee1691d/packages/cantor-libs-19.04.3-2.fc31.x86_64.
rpm
2019-10-03T11:28:54Z CRITICAL Package "cantor-libs-19.04.3-2.fc31.x86_64" from repository "fedora" has incorrect checksum
2019-10-03T11:29:01Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/fedora-3589ee8a7ee1691d/packages/plasma-desktop-5.16.4-1.fc31.x86_6
4.rpm
2019-10-03T11:29:01Z CRITICAL Package "plasma-desktop-5.16.4-1.fc31.x86_64" from repository "fedora" has incorrect checksum
2019-10-03T11:29:01Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/fedora-3589ee8a7ee1691d/packages/plasma-lookandfeel-fedora-5.16.4-1
.fc31.noarch.rpm
2019-10-03T11:29:01Z CRITICAL Package "plasma-lookandfeel-fedora-5.16.4-1.fc31.noarch" from repository "fedora" has incorrect checksum
2019-10-03T11:29:01Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/fedora-3589ee8a7ee1691d/packages/plasma-workspace-5.16.4-1.fc31.x86
_64.rpm
2019-10-03T11:29:01Z CRITICAL Package "plasma-workspace-5.16.4-1.fc31.x86_64" from repository "fedora" has incorrect checksum
2019-10-03T11:29:01Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/fedora-3589ee8a7ee1691d/packages/sddm-breeze-5.16.4-1.fc31.noarch.rpm
2019-10-03T11:29:01Z CRITICAL Package "sddm-breeze-5.16.4-1.fc31.noarch" from repository "fedora" has incorrect checksum
2019-10-03T11:29:03Z DDEBUG Cleaning up.
2019-10-03T11:29:03Z SUBDEBUG
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 65, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 98, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 122, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 166, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 224, in do_transaction
    self.download_packages(install_pkgs, self.output.progress, total_cb)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 1133, in download_packages
    remote_pkgs, local_repository_pkgs = self._select_remote_pkgs(pkglist)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 2474, in _select_remote_pkgs
    _('Some packages have invalid cache, but cannot be downloaded due to '
dnf.exceptions.Error: Some packages have invalid cache, but cannot be downloaded due to "--cacheonly" option
2019-10-03T11:29:03Z CRITICAL Error: Some packages have invalid cache, but cannot be downloaded due to "--cacheonly" option

The reason for it is that the pkgs have not been downloaded during step 2. In fact the packages were deemed as to be removed. Partial output from system-upgrade download:

Removing dependent packages:
....
plasma-desktop 5.15.5-1.fc30
sddm-breeze 5.15.5-1.fc30
plasma-workspace 5.15.5-1.fc30
plasma-lookandfeel-fedora 5.15.5-1.fc30
cantor-libs 18.12.3-1.fc30
cantor 18.12.3-1.fc30
LibPlot 2.5.0-3.fc30
.....

And no newer versions to be installed.


Expected results:
Depresolv results should be identical between system-upgrade download and reboot and the system upgrade should be completed.

Additional info:

A similar bug was opened, but not investigated further: #1644439 

In system-upgrade.py there is a note in the run_upgrade function line 520:

# NOTE: We *assume* that depsolving here will yield the same
# transaction as it did during the download, but we aren't doing
# anything to *ensure* that; if the metadata changed, or if depsolving
# is non-deterministic in some way, we could end up with a different
# transaction and then the upgrade will fail due to missing packages.
#
# One way to *guarantee* that we have the same transaction would be
# to save & restore the Transaction object, but there's no documented
# way to save a Transaction to disk.
#
# So far, though, the above assumption seems to hold. So... onward!

So it seems I have hit a very rare condition, in which the depresolv differs between download and reboot.

Comment 1 Brownsen 2019-10-05 13:24:34 UTC
Created attachment 1622735 [details]
dnf logs from dnf system-upgrade -v reboot --debugsolver

Comment 2 Brownsen 2019-10-05 13:25:15 UTC
The output (./debugdata) from

dnf system-upgrade -v download --refresh --allowerasing --releasever=31 --debugsolver

is in the file Bug_1758588_debugadata_download.tar.gz shared here:

https://drive.google.com/file/d/1D9Q0TD-xZAenCulTkD9SRLr6GFIBuLVb/view?usp=sharing

Running 

dnf system-upgrade -v reboot --debugsolver

did not create a debugdata dir anywhere. Therefore I am attaching the dnf logs from /var/log/ in file Bug_1758588_dnf_logs.tar.gz

Comment 3 Brownsen 2019-10-06 10:47:10 UTC
Possible workaround is to dnf remove the pkgs causing it and rerun system-upgrade download and reboot

Comment 4 Marek Blaha 2019-10-08 07:52:19 UTC
Yes, it really looks like the transaction was resolved differently for "download" and "upgrade" phases and the root cause of it is here:

download command was run with --allowerasing and this switch gives solver more options how to solve the transaction:
2019-10-04T20:53:08Z DDEBUG Command: dnf system-upgrade -v download --refresh --allowerasing --releasever=31 --debugsolver

after reboot, upgrade command was run without this switch:
2019-10-04T21:38:04Z DDEBUG Command: dnf system-upgrade upgrade 

Dnf needs to remember switches used in the download phase and reuse them with upgrade. This is bug on the dnf side and will be fixed.

Comment 5 Adam Williamson 2019-10-08 15:39:14 UTC
Well, it may be a bit more complicated. The plugin definitely tries to store the allowerasing state and load it before running the upgrade:

https://github.com/rpm-software-management/dnf-plugins-extras/blob/master/plugins/system_upgrade.py#L578 (stores it after download)
https://github.com/rpm-software-management/dnf-plugins-extras/blob/master/plugins/system_upgrade.py#L416 (loads it in configure_upgrade, before the upgrade runs)

what I wonder, though, is whether this being a part of `self.cli.demands` rather than `self.base.conf` like most of the other options is a problem? I'm wondering if the design of dnf itself is that the CLI code passes that value to the base functions - like at https://github.com/rpm-software-management/dnf/blob/master/dnf/cli/commands/shell.py#L246 , for e.g. - rather than the base functions reading it directly. However in this system_upgrade plugin we aren't running through the CLI code, we just set `self.cli.demands.allow_erasing` and then call `self.base.install(pkgspec, reponame=repo_id)` for each package in the list directly.

I'm not sure how `base.resolve()` ultimately gets run on the system upgrade path - it must be plugin magic? - but my guess is that, when it does get run, the value of `self.cli.demands.allow_erasing` is not passed to it.

Comment 6 Fedora Blocker Bugs Application 2019-10-08 16:11:42 UTC
Proposed as a Blocker for 31-final by Fedora user chrismurphy using the blocker tracking app because:

 Propose as a "0Day/PreviousRelease" blocker; while it can be fixed at any time, if accepted it means a stable fix must be available in F29/F30 by F31 release day.
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Normal.2C_0-Day_and_Previous_Release_blockers

Per beta criterion:
"For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from a fully updated, clean default installation of each of the last two stable Fedora releases with that package set installed. "

Seeing as the user is not flagged at the download step that the upgrade will fail, and then the reboot/upgrade step does fail, I propose that violates the criterion.

Comment 7 Samuel Sieb 2019-10-08 18:19:12 UTC
This is a regression because --allowerasing has worked in previous releases.

Comment 8 Adam Williamson 2019-10-09 01:08:39 UTC
So, I've been poking at this today. So far:

1. I can reproduce it. My minimal reproducer so far: do a minimal install of F30, do 'dnf -y install cantor', do 'dnf --releasever=31 --allowerasing system-upgrade download', do 'dnf system-upgrade reboot'

2. When doing the above, /var/lib/dnf/system-upgrade.json has "allow_erasing": true

As a side note, one of the actual deps issue here, and the one that installing cantor triggers for testing, is our old friend https://bugzilla.redhat.com/show_bug.cgi?id=1747408 , AKA the libgit2 module upgrade bug: kf5-ktexteditor requires libgit, which triggers that bug. So the 'real' way to fix the upgrade for that issue would be 'dnf module reset libgit2'. The OP has a few other issues, though, including one to do with dnf-yum and dnf package versions in F30 and F31 that I'll file separately.

I'll continue digging into this tomorrow, I gotta go for dinner. I'll start logging what actually happens in the upgrade transaction and try to see why it's including cantor.

Comment 9 Adam Williamson 2019-10-09 19:17:01 UTC
Hum. AFAICS, allowerasing is always being enabled here. I added logging statements, and it's properly set during the upgrade attempt, and it is set True when resolve() is called.

My next target is: exactly how system-upgrade implements the upgrade process. It basically takes the list of packages that was found by the `download` step and calls dnf Base method `install()` on each of them. I'm guessing perhaps cantor gets pulled into the set during that process, and somehow allowerasing doesn't result in it being kicked out again at resolve().

So I threw some logging in there, and it appears that when the actual upgrade step is trying to happen, cantor appears in the transaction when libqalculate does:

Oct 09 11:57:59 localhost.localdomain dnf[640]: XXX adding pkg spec kf5-sonnet-ui-5.61.0-1.fc31.x86_64
Oct 09 11:57:59 localhost.localdomain dnf[640]: XXX resolve called with allowerasing True
Oct 09 11:58:00 localhost.localdomain dnf[640]: XXX adding pkg spec libpwquality-1.4.1-1.fc31.x86_64
Oct 09 11:58:00 localhost.localdomain dnf[640]: XXX resolve called with allowerasing True
Oct 09 11:58:00 localhost.localdomain dnf[640]: XXX adding pkg spec kf5-syntax-highlighting-5.61.0-1.fc31.x86_64
Oct 09 11:58:00 localhost.localdomain dnf[640]: XXX resolve called with allowerasing True
Oct 09 11:58:01 localhost.localdomain dnf[640]: XXX adding pkg spec libqalculate-3.1.0-2.fc31.x86_64
Oct 09 11:58:01 localhost.localdomain dnf[640]: XXX resolve called with allowerasing True
Oct 09 11:58:01 localhost.localdomain dnf[640]: XXX cantor now in install set!

to do that I made base.install() resolve (with allowerasing=True) every time it is called and then check whether 'cantor' appears in self.transaction.install_set.

Looks like cantor-libs requires libqalculate, and there's an soname bump in F31, so that makes sense as far it goes: the new libqalculate soname is pulled into the transaction, so DNF notices that the installed 'cantor-libs' relies on the old soname and pulls the newer one from F31 into the transaction. So that's how cantor makes it into the transaction even though it's not listed directly in /var/lib/dnf/system-upgrade.json . The next question is, why is it kicked out of the transaction (or never included in it) at the `download` stage, but not at the `upgrade` stage?

Comment 10 Adam Williamson 2019-10-09 19:21:21 UTC
On that question, I noticed a strange thing about the initial `download` phase transaction. Here's what it decides:

=====

Upgrading:
 libgit2                                  x86_64     0.27.8-1.module_f31+3326+1efaac5b       fedora-modular     412 k
[... hundreds of other things ...]
Removing dependent packages:
 cantor                                   x86_64     18.12.3-1.fc30                          @updates           3.1 M
 cantor-libs                              x86_64     18.12.3-1.fc30                          @updates           3.3 M
 kf5-ktexteditor                          x86_64     5.59.0-1.fc30                           @updates            14 M
Downgrading:
 python3-unbound                          x86_64     1.9.3-1.fc31                            fedora             100 k
 unbound-libs                             x86_64     1.9.3-1.fc31                            fedora             522 k

Transaction Summary
======================================================================================================================
Install     51 Packages
Upgrade    537 Packages
Remove       3 Packages
Downgrade    2 Packages

=====

so, when you look at it carefully, that's *odd*, isn't it? Why is it *only* kicking cantor, cantor-libs and kf5-ktexteditor out of the transaction? cantor and cantor-libs go because they depend on kf5-ktexteditor, okay. But why is kf5-ktexteditor being kicked out?

I'm...wondering if somehow dnf is getting confused about whether libgit2 is excluded or not? At some point it decides libgit2 is excluded and therefore kicks its dependency - kf5-ktexteditor - out of the transaction, but then somehow decides to include libgit2 after all but *not* pull kf5-ktexteditor back in?

Comment 11 Adam Williamson 2019-10-09 19:47:23 UTC
unfortunately can't find anything useful in the solver debug info to explain exactly what goes on in the download transaction wrt kf5-ktexteditor , and haven't drunk enough yet today to start understanding libdnf code, so...I'll see if the devs can take it from here.

Comment 12 Adam Williamson 2019-10-09 19:53:17 UTC
Oh, one more nugget - the issue here seems to be specific to distro-sync mode (which system-upgrade uses by default). If I do `dnf --releasever=31 --allowerasing upgrade`, I get a different result:

 Problem: cannot install the best update candidate for package kf5-ktexteditor-5.59.0-1.fc30.x86_64
  - problem with installed package kf5-ktexteditor-5.59.0-1.fc30.x86_64
  - package kf5-ktexteditor-5.61.0-1.fc31.x86_64 requires libgit2.so.28()(64bit), but none of the providers can be installed
  - package libgit2-0.28.2-2.module_f31+5411+fa1856a4.x86_64 is excluded
  - package libgit2-0.28.3-1.fc31.x86_64 is excluded
...
Skipping packages with broken dependencies:
 kf5-ktexteditor                                        x86_64                   5.61.0-1.fc31                                         fedora                           2.6 M

Transaction Summary
==============================================================================================================================================================================
Install   51 Packages
Upgrade  537 Packages
Skip       1 Package

which I believe means it plans to just leave kf5-ktexteditor at its F30 package state. I'm not sure what it plans to do about the dependency on libqalculate, but anyway.

By comparison if I do `dnf --releasever=31 --allowerasing distro-sync`, I get the same result as with `dnf --releasever=31 --allowerasing system-upgrade download` - it wants to erase cantor and kf5-ktexteditor.

Comment 13 Adam Williamson 2019-10-09 20:17:23 UTC
OOH! Okay, I think I have figured this one out. At least I have a plausible theory.

So, okay, let's go over it from square 1. It is important to understand how system-upgrade operates.

The system-upgrade `download` phase sets various options, then calls (by default) `self.base.distro_sync()` to populate a transaction. i.e. it does a distro-sync *operation*. To dnf/libdnf, distro-sync is not a configuration option or anything, it's a particular *operation*. Then that transaction gets resolved and run, which results in all the packages being download, and system-upgrade reads out the list of packages that was included in the transaction and stashes it in a state file, along with all the configuration options like gpgcheck, allowerasing, best and so on.

The system-upgrade `upgrade` phase does *not* do a distro-sync operation. What it does is read in all the options and the list of packages from the state file, set all the options, and then call `self.base.install()` on each of the packages in the list.

So, here's what I think happens. During the *download* phase, we are doing a `distro-sync` operation, which essentially adds a constraint to the rules for solving the transaction - a constraint that libdnf/libsolv call `RULE_DISTUPGRADE`, and is defined as follows:

*SOLVER_DISTUPGRADE*::
Update the matching installed packages to the best version included in one of the repositories. After this operation, all come from one of the available repositories except orphaned packages. Orphaned packages are packages that have no relation to the packages in the repositories, i.e. no package in the repositories have the same name or obsolete the orphaned package. This action brings the installed packages in sync with the ones in the repository. By default it also turns of arch/vendor/version locking for the affected packages to simulate a fresh installation. This means that distupgrade can actually downgrade packages if only lower versions of a package are available in the repositories. You can tweak this behavior with the SOLVER_FLAG_DUP_ solver flags.

Note especially "After this operation, all come from one of the available repositories except orphaned packages." So, I believe kf5-ktexteditor is excluded from the `download` phase transaction because of this constraint. As we can see in #c12, its F31 build cannot be included in the transaction (basically Because Modularity Stream Issues - we don't have a libgit2 0.28 stream enabled). Because we're operating under SOLVER_DISTUPGRADE rules we're not allowed to just keep the fc30 kf5-ktexteditor, because SOLVER_DISTUPGRADE requires that if a newer version is available in the distupgrade repo we *must* update to it. However, because we have --allowerasing, libsolv/libdnf decide not to just fail on this, but to 'resolve' it by kicking kf5-ktexteditor and its dependencies out of the transaction entirely. This results in a transaction that meets all the requirements. So this transaction goes ahead and runs, and the system-upgrade plugin writes out its list of packages to the state file - a list that does not include kf5-ktexteditor or cantor. However, there's no mechanism that records the fact that those packages were actively *excluded* from the transaction, or anything like that.

We now head to the `upgrade` phase. As we noted earlier, this phase does *not* run a `distro-sync` operation, and there is no such thing as a distro-sync flag or config setting at the dnf level. What this phase does instead is create the transaction simply by calling `install()` on each package listed in the state file, then resolve that transaction. Because we're not doing a `distro-sync` operation, this time the transaction is resolved **WITHOUT** the `SOLVER_DISTUPGRADE` constraint.

kf5-ktexteditor and cantor and cantor-libs are never explicitly added to the transaction. But, as we saw in #c9, they get pulled in via a bumped library dependency - libqalculate *is* listed in the state file, so it gets pulled into the transaction, and libdnf notices that this breaks the dependencies of cantor-libs, so it pulls the updated F31 cantor-libs and cantor into the transaction. Now unlike at the `download` stage, there is no `SOLVER_DISTUPGRADE` constraint in operation, so it's valid for libdnf to keep the *old F30* kf5-ktexteditor package to satisfy cantor-libs' dependency on kf5-ktexteditor. Thus at the `upgrade` phase we wind up with a valid transaction that *includes* cantor and cantor-libs.

And voila, that's the explosion. The `download` transaction's additional constraint wound up in cantor, cantor-libs and kf5-ktexteditor being excluded, so they weren't downloaded. The `upgrade` transaction doesn't have that constraint, pulls them in, and so it blows up because the packages aren't there and it can't download them.

Comment 14 Adam Williamson 2019-10-09 20:24:21 UTC
assigning back to the plugin as if my theory is correct, the problem here is ultimately in the way the plugin runs these two different transactions, libdnf is only doing what it's being asked to do.

Comment 15 Marek Blaha 2019-10-10 07:28:42 UTC
I believe that the proper fix of the system-upgrade should be:

1. prepare transaction in the `download` phase and store it as a whole into the history database. (At the moment system-upgrade stores data into system-upgrade.json file and stores only "forward" operations - list of the packages that should be installed on the upgraded system. It doesn't trace any removals.)

2. after reboot do something like redo on this saved transaction - load it from the database and run.

This approach will also fix the problem with incorrect reasons of newly installed dependencies - bz#1574930 and will enable "offline" updates (or basically any transactions could be run offline after reboot).
OTOH the proper solution would require fixing it on all levels - system-upgrade, dnf itself and probably also libdnf enhancements will be required. Attempting to do this after the beta is too risky.

There might be a temporary workaround to this bug in storing also package removals in system-upgrade.json. This should be relatively easy to implement and can be done in the system-upgrade plugin itself.

Comment 16 Adam Williamson 2019-10-10 15:10:08 UTC
Yeah, I was gonna suggest the same thing (track removals as well as installs). I even got started trying it yesterday, but shadowing the install code *exactly* doesn't work as attempting to use `pkg.repo.id` in the dict fails with "KeyError: '@System'". I didn't have time to debug that yesterday, not sure if the fix is just to use a simple list instead of a dict for removals or what.

Comment 17 Adam Williamson 2019-10-10 16:47:41 UTC
https://github.com/rpm-software-management/dnf-plugins-extras/pull/163 at least fixes the simple reproducer I found. I'm about, like, 80% sure it shouldn't cause anything else to blow up horribly? :) That whole 'actually allow us to serialize transactions properly' thing sure sounds like a good idea though.

Comment 18 Ben Cotton 2019-10-10 17:01:17 UTC
+1 blocker

Comment 19 Kamil Páral 2019-10-10 17:03:02 UTC
+1 blocker, because dnf system-upgrade doesn't work correctly when executed properly (the way it is supposed to be used)

Comment 20 Kamil Páral 2019-10-10 17:04:57 UTC
There is something very wrong with bugzilla interface caching. Revert status change.

Comment 21 Adam Williamson 2019-10-10 17:09:09 UTC
I still don't think this violates the criteria, as AFAIK it does not affect any default release-blocking package set of F29 or F30.

Comment 22 Adam Williamson 2019-10-10 17:10:54 UTC
Note the fix is still somewhat speculative. We now take all the removals from the `download` phase and add them to the transaction, then take all the installs from the `download` phase and add them to the transaction, then resolve that transaction. I'm not gonna sit here and swear that this will always reproduce the `download` transaction exactly and not do anything surprising, it would not surprise me very much at all if there turns out to be a different corner case where *this* produces an unexpected result.

Comment 23 Kevin Fenzi 2019-10-10 17:21:21 UTC
-1 blocker as it doesn't meet critera, but +1 FE as it would be very nice to fix it.

Comment 24 František Zatloukal 2019-10-10 17:41:05 UTC
-1 blocker (doesn't violate criteria, but I'd love to see this fixed), +1 FE

Comment 25 Mohan Boddu 2019-10-10 17:43:16 UTC
-1 Blocker, +1 FE

Comment 26 Adam Williamson 2019-10-10 17:53:34 UTC
FE doesn't make any sense here as the change would be for F29 and F30. We'd send it to F31 too, but that would only be for upgrades *from* F31 so it's not urgent. This is either an AcceptedPreviousReleaseBlocker or nothing.

Comment 27 Geoffrey Marr 2019-10-10 18:46:58 UTC
-1 blocker

If we become aware that it doesn't allow the installation of a release-blocking package set for F29 or F30, we can reconsider.

Comment 28 Julen Landa Alustiza 2019-10-11 06:42:50 UTC
-1 blocker

Comment 29 Ben Cotton 2019-10-11 19:35:32 UTC
Change me to -1 blocker, +1 FE based on the comments that it doesn't affect default package sets

Comment 30 Ben Cotton 2019-10-11 19:36:37 UTC
Errr, ignore the "+1 FE part"

Comment 31 Fedora Update System 2019-10-14 13:18:15 UTC
FEDORA-2019-daa0986c6d has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-daa0986c6d

Comment 32 Geoffrey Marr 2019-10-14 19:11:37 UTC
Discussed during the 2019-10-14 blocker review meeting: [0]

The decision to classify this bug as a "RejectedBlocker" was made as we are not aware of any case where this problem affects one of the release-blocking F30 package sets, so it does not violate the criterion stated.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-10-14/f31-blocker-review.2019-10-14-16.01.txt

Comment 33 Fedora Update System 2019-10-15 10:27:55 UTC
FEDORA-2019-daa0986c6d has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-daa0986c6d

Comment 34 Fedora Update System 2019-10-15 22:57:57 UTC
dnf-plugins-extras-4.0.7-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-daa0986c6d

Comment 35 Fedora Update System 2019-10-18 10:34:17 UTC
FEDORA-2019-e8db3bef68 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8db3bef68

Comment 36 Fedora Update System 2019-10-18 21:00:24 UTC
dnf-plugins-extras-4.0.7-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e8db3bef68

Comment 37 Ed Greshko 2019-10-18 23:30:43 UTC
dnf-plugins-extras-4.0.7-2.fc30 did not resolve my problem noted on the test mailing list.

Still getting

2019-10-18T23:28:00Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-free-058b175644bc1430/packages/rpmfusion-free-appstream-data-30-1.20181021.fc30.noarch.rpm
2019-10-18T23:28:00Z CRITICAL Package "rpmfusion-free-appstream-data-30-1.20181021.fc30.noarch" from repository "rpmfusion-free" has incorrect checksum
2019-10-18T23:28:00Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-nonfree-c253272f7b309f17/packages/rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch.rpm
2019-10-18T23:28:00Z CRITICAL Package "rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch" from repository "rpmfusion-nonfree" has incorrect checksum

Comment 38 Ed Greshko 2019-10-20 15:50:49 UTC
Created attachment 1627498 [details]
Logs of upgrade

Logs requested by Adam

Comment 39 Adam Williamson 2019-10-20 16:22:14 UTC
Thanks, Ed.

So, your case seems a bit different. We'll need to dig into it in a bit more detail, but broadly the issue is that a couple of rpmfusion packages show up to be "reinstalled" in the `upgrade` transaction but are not in the `download` transaction:

2019-10-20T15:42:45Z DEBUG ---> Package rpmfusion-free-appstream-data.noarch 30-1.20181021.fc30 will be reinstalled
2019-10-20T15:42:45Z DEBUG ---> Package rpmfusion-nonfree-appstream-data.noarch 30-1.20181021.fc30 will be reinstalled
...
Reinstalling:
 rpmfusion-free-appstream-data               noarch30-1.20181021.fc30                  rpmfusion-free    303 k
 rpmfusion-nonfree-appstream-data            noarch30-1.20181021.fc30                  rpmfusion-nonfree 62 k

We're still in the same category of issue here - the `download` transaction is coming out different to the `upgrade` transaction - but the details seem a bit different. I'm not sure yet *why* the upgrade transaction decides to reinstall those packages.

Comment 40 Fedora Update System 2019-10-25 17:01:52 UTC
dnf-plugins-extras-4.0.7-2.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 41 dherault 2019-10-25 23:13:38 UTC
For the RPMFUSION case : 
https://bugzilla.redhat.com/show_bug.cgi?id=1764169

Comment 42 Adam Williamson 2019-10-29 01:17:45 UTC
Clearing CommonBugs as we fixed this ahead of Final.


Note You need to log in before you can comment on or make changes to this bug.