Hide Forgot
Description of problem: I tried to upgrade from Fedora 30 to Fedora 31. The upgrade download phase succeeded, but after reboot the transaction immediately failed with: 2019-10-22T11:21:59Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-free-058b175644bc1430/packages/rpmfusion-free-appstream-data-30-1.20181021.fc30.noarch.rpm 2019-10-22T11:21:59Z CRITICAL Package "rpmfusion-free-appstream-data-30-1.20181021.fc30.noarch" from repository "rpmfusion-free" has incorrect checksum 2019-10-22T11:21:59Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-nonfree-c253272f7b309f17/packages/rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch.rpm 2019-10-22T11:21:59Z CRITICAL Package "rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch" from repository "rpmfusion-nonfree" has incorrect checksum 2019-10-22T11:22:05Z DDEBUG Cleaning up. 2019-10-22T11:22:05Z SUBDEBUG Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 65, in main return _main(base, args, cli_class, option_parser_class) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 98, in _main return cli_run(cli, base) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 122, in cli_run ret = resolving(cli, base) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 166, in resolving base.do_transaction(display=displays) File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 224, in do_transaction self.download_packages(install_pkgs, self.output.progress, total_cb) File "/usr/lib/python3.7/site-packages/dnf/base.py", line 1133, in download_packages remote_pkgs, local_repository_pkgs = self._select_remote_pkgs(pkglist) File "/usr/lib/python3.7/site-packages/dnf/base.py", line 2474, in _select_remote_pkgs _('Some packages have invalid cache, but cannot be downloaded due to ' dnf.exceptions.Error: Some packages have invalid cache, but cannot be downloaded due to "--cacheonly" option 2019-10-22T11:22:05Z CRITICAL Error: Some packages have invalid cache, but cannot be downloaded due to "--cacheonly" option The two packages in question do not exist in /var/lib/dnf/system-upgrade/ at all. It seems dnf simply forgot to download them. By no accident, these two packages are the only ones that are marked for reinstallation in the transaction: Reinstalling: rpmfusion-free-appstream-data noarch30-1.20181021.fc30 rpmfusion-free 303 k rpmfusion-nonfree-appstream-data noarch30-1.20181021.fc30 rpmfusion-nonfree 62 k (They are not mentioned in the download phase transaction). After talking to Jaroslav Mracek, the packages get reinstalled when their NEVRA is the same, but their contents changed. That should never happen in Fedora, because Koji doesn't allow to use the same NEVRA for multiple builds, but it can happen in third-party repositories using different build methods, like rpmfusion above. A fix in dnf is likely needed to make sure such packages are downloaded and available for the offline transaction. The system itself doesn't seem to be negatively impacted in any way by this bug, the transaction didn't perform any changes to the system. Version-Release number of selected component (if applicable): dnf-4.2.11-2.fc30.noarch dnf-data-4.2.11-2.fc30.noarch dnf-plugins-core-4.0.10-1.fc30.noarch dnf-utils-4.0.10-1.fc30.noarch libdnf-0.35.5-2.fc30.x86_64 libsolv-0.7.7-1.fc30.x86_64 python3-dnf-4.2.11-2.fc30.noarch python3-dnf-plugins-core-4.0.10-1.fc30.noarch python3-dnf-plugins-extras-common-4.0.7-2.fc30.noarch python3-dnf-plugin-system-upgrade-4.0.7-2.fc30.noarch python3-libdnf-0.35.5-2.fc30.x86_64 How reproducible: likely 100% at this very point of time with the two aforementioned packages installed, but I haven't tested multiple times yet Steps to Reproduce: 1. have all rpmfusion repos (main, updates, testing) enabled and rpmfusion-free-appstream-data and rpmfusion-nonfree-appstream-data packages installed on F30 2. sudo dnf system-upgrade download --releasever 31 3. sudo dnf system-upgrade reboot 4. see the transaction failed immediately on reboot
Created attachment 1627972 [details] dnf.log (download and upgrade)
Created attachment 1627973 [details] journal during upgrade
This bug is not likely to happen with Fedora packages, so I'm not going to propose it as an F31 blocker. The fix needs to happen in F29 and F30, so no point in proposing it as F31 freeze exception either. Because it is likely a high-profile bug, though, I'm marking it as CommonBugs and proposing as a PrioritizedBug.
Created attachment 1627974 [details] rpm -qa output
FTR, downgrading libsolv-0.7.7-1.fc30.x86_64 to libsolv-0.7.5-1.fc30.x86_64 did not help.
Hmm, this is going to be more convoluted than I thought. I installed a minimal F30 in a VM, enabled rmpfusion repos and installed rpmfusion-free-appstream-data and rpmfusion-nonfree-appstream-data. The upgrade completed fine, no error. The transaction didn't touch the two packages at all (didn't try to reinstall them). So I have two theories: a) those two rpm packages changed their contents during F30 lifecycle, I still have an older version, and the repo now contains a newer version. Dnf doesn't notice this during standard 'dnf update', but it does notice it during system-upgrade for some reason. That's why a fresh system installation doesn't exhibit the error, but my installation does. However, I downloaded the package and compared the files included with my system files, and there's no difference reported on file level. There's also no difference in metadata printed by rpm -qi when comparing the installed package and the downloaded package. b) there is some specific trigger condition on my system, but I have no idea what it could be
I retract nominations because it is not clear what is happening here and how many people it might affect.
I am sorry, but I am unable to reproduce the issue. I installed nearly same packageset like you have on your system, but everything works like expected. I believe that we have two options: 1. Try to fix your system but we will lose the reproducer ("dnf distrosync" before system upgrade could help before because it will reinstall packages with different content but same NEVRA) 2. Try to investigate and compare debugsolver-data from download and reboot system-upgrades sub-commands.
(In reply to Jaroslav Mracek from comment #8) > 1. Try to fix your system but we will lose the reproducer ("dnf distrosync" > before system upgrade could help before because it will reinstall packages > with different content but same NEVRA) I always use distrosyc, but it doesn't want to perform any changes. > 2. Try to investigate and compare debugsolver-data from download and reboot > system-upgrades sub-commands. I'll be happy to provide it if you tell me how :)
*** Bug 1761671 has been marked as a duplicate of this bug. ***
Kamil, You're not alone ;) Can't upgrade to F31 too oct. 25 00:10:51 localhost.localdomain dnf[809]: Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-free-058b175644bc1430/packages/rpmfusion-free-appstream-data-30-1.20181021.fc30.noarch.rpm oct. 25 00:10:51 localhost.localdomain dnf[809]: Le paquet "rpmfusion-free-appstream-data-30-1.20181021.fc30.noarch" du dépôt "rpmfusion-free" a une somme de contrôle incorrecte oct. 25 00:10:51 localhost.localdomain dnf[809]: Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-nonfree-c253272f7b309f17/packages/rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch.rpm oct. 25 00:10:51 localhost.localdomain dnf[809]: Le paquet "rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch" du dépôt "rpmfusion-nonfree" a une somme de contrôle incorrecte oct. 25 00:10:54 localhost.localdomain dnf[809]: Erreur : Certains paquets ont un cache invalide, mais ne peuvent pas être téléchargés à cause de l’option « --cacheonly » oct. 25 00:10:54 localhost.localdomain systemd[1]: dnf-system-upgrade.service: Main process exited, code=exited, status=1/FAILURE oct. 25 00:10:54 localhost.localdomain systemd[1]: dnf-system-upgrade.service: Failed with result 'exit-code'. oct. 25 00:10:54 localhost.localdomain systemd[1]: Failed to start System Upgrade using DNF.
FYI: I submitted a bug report with this same issue which was found to duplicate this. Happy to help test any proposed solutions. DNF log below. Transaction Summary ================================================================================ Install 45 Packages Upgrade 3552 Packages Remove 14 Packages Downgrade 24 Packages 2019-10-23T05:09:35Z INFO Total size: 4.0 G 2019-10-23T05:09:35Z INFO Total download size: 62 k 2019-10-23T05:09:35Z INFO Downloading Packages: 2019-10-23T05:09:36Z CRITICAL Error opening file for checksum: /var/lib/dnf/system-upgrade/rpmfusion-nonfree-c253272f7b309f17/packages/rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch.rpm 2019-10-23T05:09:36Z CRITICAL Package "rpmfusion-nonfree-appstream-data-30-1.20181021.fc30.noarch" from repository "rpmfusion-nonfree" has incorrect checksum 2019-10-23T05:11:04Z DDEBUG Cleaning up.
OK, I'll mark this as CommonBugs again. We'll document this and say that the fix is still not available, but we're working on this. And we'll try to resolve this with Jaroslav next week. In the mean time, I have a suspicion that a simple workaround is available. If you do "sudo dnf reinstall rpmfusion-free-appstream-data rpmfusion-nonfree-appstream-data" and then follow up with the usual "system-upgrade download" and "system-upgrade reboot", I think it's highly likely that this problem is gone. I could try it but I then I wouldn't be able to reproduce it anymore when working on a proper fix. Could one of you test that theory and then report back? Thanks.
I tried to transfer files owned by rpmfusion-free-appstream-data and rpmfusion-nonfree-appstream-data using rsync to a VM, but the upgrade still starts just fine. So the problem is probably not triggered by the actual files stored on disk. It might be something stored in RPMDB.
I have installed a VM containing the same set of RPMs as I have on my laptop. Then I copied over /var/lib/rpm* and all files from rpmfusion-{free,nonfree}-appstream-data from my laptop to the VM. The dnf system-upgrade still worked as expected, no error. Sigh.
I've tried "sudo dnf reinstall rpmfusion-free-appstream-data rpmfusion-nonfree-appstream-data" And after the usual system-upgrade download , and reboot. Unfortunately same issue, can't upgrade.
I too am unable to upgrade. If one is interested, my logs are contained in https://bugzilla.redhat.com/show_bug.cgi?id=1758588
I tried deleting and purging the rpmfusion-free-appstream-data file off my system. Ran DNF upgrade --refresh. Verified again the file was gone. Started the system upgrade again. For reasons known to those above my pay grade it insisted on installing the file back and resulted in the same error.
(In reply to Hugh M. from comment #18) > I tried deleting and purging the rpmfusion-free-appstream-data file off my > system. Ran DNF upgrade --refresh. Verified again the file was gone. Started > the system upgrade again. For reasons known to those above my pay grade it > insisted on installing the file back and resulted in the same error. You should run "dnf system-upgrade clean" or "dnf clean packages" before retrying. FWIW, I do both only because I'm paranoid. :-)
(In reply to Ed Greshko from comment #19) > (In reply to Hugh M. from comment #18) > > I tried deleting and purging the rpmfusion-free-appstream-data file off my > > system. Ran DNF upgrade --refresh. Verified again the file was gone. Started > > the system upgrade again. For reasons known to those above my pay grade it > > insisted on installing the file back and resulted in the same error. > > You should run "dnf system-upgrade clean" or "dnf clean packages" before > retrying. > FWIW, I do both only because I'm paranoid. :-) Sorry I didn't put it in my comment. I am a fellow paranoid also and run "dnf package clean" prior to any dnf updates or upgrades. I also verified the /var/lib/dnf/system-upgrade/ directory was empty before starting the system upgrade. My thought is the system-upgrade saw the rpmfusion packages and sees this file as a required dependency. So it installs it.
I runned dnf system-upgrade clean and dnf clean package after the udo dnf reinstall rpmfusion-free-appstream-data rpmfusion-nonfree-appstream-data. Same issue.
I've actually removed 2 packages before running the "dnf system-upgrade download" so it will be a part of the transaction, re-ran dnf system-upgrade download without a cleanup, it downloaded the missing packages and then upgrade completed successfully on reboot. I'm not sure what's wrong with a packages, maybe they really changed the checksum of the packages while maintaining the version?
I've had exactly the same problem. I removed both rpmfusion-free-appstream-data and rpmfusion-nonfree-appstream-data packages before starting dnf system-upgrade download --releasever 31, then was able to start the upgrade to F31. I'm now under F31. The 2 packages were reinstalled during the upgrade to F31, in the following versions : Nom : rpmfusion-free-appstream-data Version : 30 Publication : 1.20181021.fc30 and Nom : rpmfusion-nonfree-appstream-data Version : 30 Publication : 1.20181021.fc30 Note that the OS version is still F30 ...
(In reply to xvr from comment #23) > I've had exactly the same problem. > I removed both rpmfusion-free-appstream-data and > rpmfusion-nonfree-appstream-data packages before starting dnf system-upgrade > download --releasever 31, then was able to start the upgrade to F31. > I'm now under F31. > The 2 packages were reinstalled during the upgrade to F31, in the following > versions : > Nom : rpmfusion-free-appstream-data > Version : 30 > Publication : 1.20181021.fc30 > > and > > Nom : rpmfusion-nonfree-appstream-data > Version : 30 > Publication : 1.20181021.fc30 > > Note that the OS version is still F30 ... Confirm this works, I'm now on 31.
Also upgraded after temporarily removing those packages. I've just seen updates that should resolve it: rpmfusion-free-appstream-data noarch 31-1.fc31 rpmfusion-free-updates 378 k rpmfusion-nonfree-appstream-data noarch 31-1.fc31 rpmfusion-nonfree-updates 101 k
Can somebody who was affected and still hasn't upgraded confirm whether the updates from comment 25 fix this issue out of the box (no workarounds needed anymore)? Thanks.
(In reply to Kamil Páral from comment #26) > Can somebody who was affected and still hasn't upgraded confirm whether the > updates from comment 25 fix this issue out of the box (no workarounds needed > anymore)? Thanks. Yes. I just upgraded a system that was affected and the upgrade worked as expected without any workaround. I didn't remove anything prior to the upgrade and at the end [egreshko@acer ~]$ rpm -q rpmfusion-free-appstream-data rpmfusion-nonfree-appstream-data rpmfusion-free-appstream-data-31-1.fc31.noarch rpmfusion-nonfree-appstream-data-31-1.fc31.noarch as expected.
Just a side note: Even on a "pure" fedora system, dnf system-upgrade wants to downgrade several packages. This should not happen with our strict policies. In particular, it should not happen with: dnf dnf-data dnf-plugins-core libdnf python3-dnf python3-libdnf But it does. While this should not impact the transaction right after upgrade it does not improve the confidence in "dnf stability" in terms of versions and transactions.
That has nothing to do with this. Also, it's not really a problem, and is a consequence of one of our strict policies - the release freeze. F31 was frozen for final release while the newer dnf versions went into F30, so of course they did not go into F31 during the freeze. That's why dnf is ahead of F31 in F30 at present. Downgrading it on upgrade to F31 is the expected consequence and doesn't cause any problems.
Hi, not sure if this will help, but I also had the problem so I'm sharing my experience. Just during the upgrade reboot, if I hit the arrow keys I would see a transaction preparing: https://ibb.co/yfpbFbS https://ibb.co/DY8hWSc But in the end the process stopped (although sometimes it showed the process complete at 100%, after hitting the arrow keys and see the transaction list, then it would go back to 0%) and after the reboot I'm still in 30.
pierre.blarre, that's a different problem. I'd suggest first trying to reach one of user help channels, where others can help you figure out whether the upgrade really failed or not. If there is a problem, please report it as a new bugzilla issue, thanks.
I have disabled rpmfusion-{free,nonfree}-updates{,-testing} repos to avoid the bumped NEVRA packages. Then I completely wiped /var/cache/dnf/* and /var/lib/dnf/system-upgrade*. That didn't have any effect, I can still reproduce the problem.
I have discovered a difference when set clean_requirements_on_remove=False. It looks like that it resolves the side effect during the reboot. Here is the PR: https://github.com/rpm-software-management/dnf-plugins-extras/pull/168. Please can anyone confirm that it resolves the issue?
I have patched the file and system-upgrade was performed successfully. (And I have lost access to the reproducer, so no more verification from me). The patch might be a bit of overkill, but it seems to fix/work around the issue.
> After talking to Jaroslav Mracek, the packages get reinstalled when their > NEVRA is the same, but their contents changed. That should never happen in > Fedora, because Koji doesn't allow to use the same NEVRA for multiple > builds, but it can happen in third-party repositories using different build > methods, like rpmfusion above. That's the one odd thing in all this. RPM Fusion DOES use Koji. (https://koji.rpmfusion.org/) So, something must've gone wrong in the build process to cause this. OOC, was this ever reported to RPM Fusion? This is the first I'm hearing of it (because I happened to be looking over Common Bugs). I can't find a report in RPM Fusion's bugzilla, and I don't see any of the usual suspects here either. Don't tell me you were all discussing this for 2 solid weeks without notifying the repo about the problem?
Frank, that was one of possible theories, but it turned out to be false. The problem occurred even when affected people reinstalled those packages beforehand. So no, this was not directly related to rpmfusion (even though the problem occurred with their packages). Jaroslav can explain the actual problem in detail, I can't. But what you quoted was not the cause.
Gotcha, thanks Kamil. Perhaps Jaroslav or someone could update the Common Bugs entry titled: "Upgrade to Fedora 31 may fail if DNF decides packages need to be reinstalled (often related to RPM Fusion third-party repository)"? https://fedoraproject.org/wiki/Common_F31_bugs#upgrade-reinstall-failure
*** Bug 1770703 has been marked as a duplicate of this bug. ***
The fix was already released in dnf-plugins-extras-4.0.8-1.fc30.
I would like top drop the mention of rpmfusion of the common bugs section related to this issue. I'm very concerned that none escalated the issue with us. https://fedoraproject.org/wiki/Bugs/Common#upgrade-reinstall-failure My understanding is that when rpmfusion-free-appstream-data*fc30 where signed with fc30 keys as provided in the f30 repos everything went fine, but for the f31 case, the maintainer didn't updated the package for fc31 so the package was still using the rpmfusion-free-appstream-data*fc30, but as provided with the fc31 key the checksum didn't match. This is no different that regular fedora processes where packages not rebuilt for any Fedora N will keep the Fedora N-1 dist tag (or older), but with a There is really no reason why this would affect rpmfusion and not fedora unless the system-upgrade process is seriously broken by design. Instead it is expected that system-upgrade to fetch rpmfusion packages for f32 when the target release to install is fedora 32 no matter what is the base distro. (and that process should be followed for any repository, not only fedora ones). @kamil, can you please verify that a sane process is followed. Please also remove the mention of rpmfusion for this bugs. Thanks in advance.
(missing text) .. dist tag (or older), but with another GPG key.
(In reply to Nicolas Chauvet (kwizart) from comment #40) > I would like top drop the mention of rpmfusion of the common bugs section > related to this issue. Can you explain why? I don't understand it. > I'm very concerned that none escalated the issue with us. There was no need. It was a bug in DNF that we needed to resolve. If you rebuilt the packages, we would have lost the reproducer. I was actually hoping you wouldn't rebuild those packages soon. > My understanding is that when rpmfusion-free-appstream-data*fc30 where > signed with fc30 keys as provided in the f30 repos everything went fine, but > ... Jaroslav can be more specific, but I believe we established that this was not a problem with rpmfusion infrastructure and build process. See comment 36. It was a bug in DNF that for some reason manifested with specific rpmfusion packages. I'm not sure if we ever found out the root cause, but we at least found the fix. > There is really no reason why this would affect rpmfusion and not fedora > unless the system-upgrade process is seriously broken by design. Yes, that's why it wasn't reported as an rpmfusion bug. At the same time, we only saw it with rpmfusion, so the common bugs entry was correct in saying that we only saw it with rpmfusion but it could affect other repositories as well, and that temporarily uninstalling rpmfusion packages before upgrade worked around the problem (until DNF team provided a fix, so now it should not be necessary). Please understand this is not about doing bad PR to a third party repo. It was a general problem (at least that's what we believe) which we only saw with rpmfusion. Now that the commonbugs entry is marked as resolved, so people who update their systems don't need to perform the workarounds, is that a good enough solution for you? If not, please help me understand why you consider the description to be problematic.
(In reply to Kamil Páral from comment #42) > It > was a general problem (at least that's what we believe) which we only saw > with rpmfusion. You do HEAR the apparent contradiction in that statement, though, right?
(In reply to "FeRD" (Frank Dana) from comment #43) > You do HEAR the apparent contradiction in that statement, though, right? There is no contradiction and I tried to explain it. We believe it was a generic problem, and we only saw it with rpmfusion. That doesn't mean it can't appear elsewhere, of course it can, we just didn't see it. It ultimately doesn't matter whether it was related to package signing or something else, whether it was caused by some rpmfusion maintainer action or a bad DNF codepath. The same situation can occur with other repos. DNF needs to be able to handle it or produce a reasonable error message.
Fair enough with this. My issue is more a Communication issue where the name "RPM Fusion" can be miss-interpreted as the (root) cause of an upgrade problem. This is enforced by the proposed workaround, that suggests to remove rpmfusion during the upgrade which is a severe disablement for end-users to keep all rpmfusion packages in shape across upgrade. Either the wording is intentional or not is not something I'm interested that much. Now that the common bugs entry have shift with f32, this is less an issue. But I'd like to drop the mention of rpmfusion in the f31 case, still. I'm suggesting to replace drop the paragraph that mention the workaround and all rpmfusion mention from this topic. (eventually to keep 3rd part repository). Can we agree on this ?
If the bug was still unresolved, I wouldn't agree, because mentioning rpmfusion and the only known workaround was a substantial help for readers to identify and fix the problem (I would be of course happy to re-word as necessary). But because it was resolved, I don't think it matters. I have reworded the commonbugs entry, made it generic and removed rpmfusion mentions from it.
Thanks a lot for theses changes and your kind answers Kamil!
Agreed, the updates and detailed responses are much appreciated.