Bug 1260989 - system upgrade using distro-sync fails - dnf tries to remove all kernels
Summary: system upgrade using distro-sync fails - dnf tries to remove all kernels
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 22
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 1266589 1266949 1269178 (view as bug list)
Depends On:
Blocks: F23FinalBlocker 1266589
TreeView+ depends on / blocked
 
Reported: 2015-09-08 11:15 UTC by Kamil Páral
Modified: 2016-01-08 21:11 UTC (History)
22 users (show)

Fixed In Version: libsolv-0.6.14-2 dnf-1.1.3-1.fc23 dnf-1.1.3-1.fc22
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-28 16:19:21 UTC


Attachments (Terms of Use)
dnf.log (795.37 KB, text/plain)
2015-09-08 11:54 UTC, Kamil Páral
no flags Details
upgrade.journal (314.83 KB, text/plain)
2015-09-08 11:54 UTC, Kamil Páral
no flags Details
rpm-qa (51.63 KB, text/plain)
2015-09-08 11:54 UTC, Kamil Páral
no flags Details
solver debugdata when running system-upgrade with --allowerasing (4.92 MB, application/x-7z-compressed)
2015-09-08 12:31 UTC, Kamil Páral
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1244175 None CLOSED distro-sync removes orphaned packages in certain cases (kernel update) 2019-09-16 23:22:46 UTC
Red Hat Bugzilla 1263677 None CLOSED it's very easy to end up with a partially-upgraded system 2019-09-16 23:22:46 UTC

Internal Links: 1244175 1263677

Description Kamil Páral 2015-09-08 11:15:03 UTC
Description of problem:
I tried to upgrade from F22 to F23 using this command:
$ sudo dnf system-upgrade --distro-sync download --releasever=23
$ sudo dnf system-upgrade reboot

The system downloaded all packages and rebooted, but the upgrade process exited immediately. The log contains this error:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 84, in main
    return _main(base, args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 146, in _main
    ret = resolving(cli, base)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 158, in resolving
    base.resolve(cli.demands.allow_erasing)
  File "/usr/lib/python2.7/site-packages/dnf/base.py", line 530, in resolve
    raise exc
DepsolveError: package kernel-core-4.1.6-200.fc22.x86_64 requires systemd >= 200, but none of the providers can be installed
Sep 08 12:38:09 CRITICAL Error: package kernel-core-4.1.6-200.fc22.x86_64 requires systemd >= 200, but none of the providers can be installed
Sep 08 12:38:09 INFO (try to add '--allowerasing' to command line to replace conflicting packages)

I have no idea why it speaks about kernel 4.1.6. Kernel 4.2.0 should have been installed, kernel 4.0.7 should have been removed. Kernel 4.1.6 should have been kept untouched. Furthermore, systemd-222-2.fc23 should have been installed, which satisfies systemd >= 200 condition. No idea what's wrong. Maybe it's related to the fact that kernel 4.1.6 is the currently booted kernel?

Version-Release number of selected component (if applicable):
python2-dnf-plugin-system-upgrade-0.4.0-1.fc22.noarch
python3-dnf-1.1.0-2.fc22.noarch
python-dnf-1.1.0-2.fc22.noarch
python-dnf-plugins-core-0.1.10-1.fc22.noarch
python-dnf-plugins-extras-common-0.0.9-1.fc22.noarch
python-dnf-plugins-extras-migrate-0.0.9-1.fc22.noarch


How reproducible:
always

Steps to Reproduce:
1. probably have a few kernels installed (might be related)
2. sudo dnf system-upgrade --distro-sync download --releasever=23
3. sudo dnf system-upgrade reboot

Comment 1 Kamil Páral 2015-09-08 11:54:38 UTC
Created attachment 1071304 [details]
dnf.log

Comment 2 Kamil Páral 2015-09-08 11:54:41 UTC
Created attachment 1071305 [details]
upgrade.journal

Comment 3 Kamil Páral 2015-09-08 11:54:44 UTC
Created attachment 1071306 [details]
rpm-qa

Comment 4 Kamil Páral 2015-09-08 11:57:54 UTC
When I tried running the same command without --distro-sync, it worked perfectly. Does --distro-sync require --allowerasing? In all cases? How is it related to kernel 4.1.6?

Comment 5 Kamil Páral 2015-09-08 12:30:02 UTC
This is interesting:

$ sudo dnf system-upgrade --distro-sync download --releasever=23 --allowerasing
Fedora 23 - x86_64                                                              71 MB/s |  43 MB     00:00    
Fedora 23 - x86_64 - Updates                                                    79 kB/s | 257  B     00:00    
Last metadata expiration check performed 0:00:00 ago on Tue Sep  8 14:22:28 2015.
Dependencies resolved.
Error: The operation would result in removing the booted kernel: kernel-core-4.1.6-200.fc22.x86_64.

It seems weird it tried to remove the current kernel, when 2 older ones are installed:
$ rpm -q kernel
kernel-4.0.7-300.fc22.x86_64
kernel-4.1.4-200.fc22.x86_64
kernel-4.1.6-200.fc22.x86_64

When I run it with -v, I see this in the output:
---> Package kernel.x86_64 4.0.7-300.fc22 will be erased
---> Package kernel.x86_64 4.1.4-200.fc22 will be erased
---> Package kernel.x86_64 4.1.6-200.fc22 will be erased
---> Package kernel-core.x86_64 4.0.7-300.fc22 will be erased
---> Package kernel-core.x86_64 4.1.4-200.fc22 will be erased
---> Package kernel-core.x86_64 4.1.6-200.fc22 will be erased
---> Package kernel-modules.x86_64 4.0.7-300.fc22 will be erased
---> Package kernel-modules.x86_64 4.1.4-200.fc22 will be erased
---> Package kernel-modules.x86_64 4.1.6-200.fc22 will be erased

It tries to remove ALL the kernels. I have a feeling this is going to be related to bug 1244175. Due to that bug, distro-sync is trying to remove orphans. And when performing a system upgrade, it seems to consider all old kernels as orphans... probably due to that fact that they have a special flag making them installable in parallel.

Comment 6 Kamil Páral 2015-09-08 12:31:43 UTC
Created attachment 1071333 [details]
solver debugdata when running system-upgrade with --allowerasing

Comment 7 Kamil Páral 2015-09-08 12:33:44 UTC
Reassigning to dnf, it seems to be another dupe of (or closely related to) bug 1244175.

Comment 8 Honza Silhan 2015-09-16 09:14:04 UTC
We'll look whether it's a duplicate of bug 1244175 and try to fix it.

Comment 9 Kamil Páral 2015-09-17 16:22:36 UTC
There's a workaround for this - install a F23 kernel (one that is currently in F23 repos) on F22, boot it, and run the system-upgrade again. It will remove all your F22 kernels, but it will not complain about the booted F23 one, and let you continue.

Comment 10 Kamil Páral 2015-09-18 11:23:20 UTC
Another workaround is to have just a single kernel installed on the system. Then it does not hit the issue mentioned in comment 0 and the upgrade proceeds. However, this can't be combined with --allowerasing, because then issue from comment 5 kicks in and prevents you from downloading the packages.

Comment 11 Boricua 2015-09-23 12:37:25 UTC
(In reply to Kamil Páral from comment #9)
> There's a workaround for this - install a F23 kernel (one that is currently
> in F23 repos) on F22, boot it, and run the system-upgrade again. It will
> remove all your F22 kernels, but it will not complain about the booted F23
> one, and let you continue.

Perhaps I did something wrong, but this tip didn't work for me. I installed kernel 4.2.1-300.fc23, along with its kernel-module on my up-to-date Fedora 22 desktop. Then rebooted and everything seems to be working OK, except that I get the same error report when trying to upgrade:

[francisco@hogar ~]$ sudo dnf system-upgrade download --releasever=23 [--distro-sync] [--best] [--allowerasing]
[sudo] password for francisco: 
Traceback (most recent call last):
  File "/bin/dnf", line 36, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 198, in user_main
    errcode = main(args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 84, in main
    return _main(base, args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 117, in _main
    cli.configure(map(ucd, args))
  File "/usr/lib/python2.7/site-packages/dnf/cli/cli.py", line 1029, in configure
    self.command.configure(self.base.extcmds)
  File "/usr/lib/python2.7/site-packages/dnf-plugins/system_upgrade.py", line 263, in configure
    self.opts = self.parse_args(args)
  File "/usr/lib/python2.7/site-packages/dnf-plugins/system_upgrade.py", line 254, in parse_args
    opts = p.parse_args(extargs)
  File "/usr/lib64/python2.7/argparse.py", line 1704, in parse_args
    self.error(msg % ' '.join(argv))
  File "/usr/lib/python2.7/site-packages/dnf-plugins/system_upgrade.py", line 206, in error
    raise AttributeError(message)
AttributeError: unrecognized arguments: [--distro-sync] [--best] [--allowerasing]

By the way, installing the F23 kernel caused my system to uninstall all my F22 kernels.

Comment 12 Honza Horak 2015-09-25 09:10:59 UTC
(In reply to Boricua from comment #11)
> AttributeError: unrecognized arguments: [--distro-sync] [--best]
> [--allowerasing]

Brackets [ ... ] are used to say these options are optional, if you want to use them, you should use:
$ sudo dnf system-upgrade download --releasever=23 --distro-sync --best --allowerasing

Comment 13 Boricua 2015-09-26 14:08:17 UTC
(In reply to Honza Horak from comment #12)
> (In reply to Boricua from comment #11)
> > AttributeError: unrecognized arguments: [--distro-sync] [--best]
> > [--allowerasing]
> 
> Brackets [ ... ] are used to say these options are optional, if you want to
> use them, you should use:
> $ sudo dnf system-upgrade download --releasever=23 --distro-sync --best
> --allowerasing

Understood. My bad; apologies for a wrong report. I am proceeding with the upgrade right now.

Comment 14 Bruno Wolff III 2015-09-28 16:52:02 UTC
As reported in bug 1266949, adding --exclude='kernel*' got distro-sync --allowerasing to work, though all kernels ended up being removed. If you know this is going to happen, you will most likely be abke to install a kernel when it's done and things will be OK. If something goes wrong you may need to do a rescue to install a kernel.

Comment 15 Stephen Gallagher 2015-09-30 19:18:07 UTC
I can confirm the above. I just installed a VM with F22 Server, upgraded all packages (so I have two kernels after a reboot) and then attempted to do:

`dnf system-upgrade download --distro-sync --allowerasing --releasever=23`

It failed with the error: Error: The operation would result in removing the booted kernel: kernel-core-4.1.7-200.fc22.x86_64.

Adding `--exclude "*kernel*"` resulted in a staged upgrade that included:


Removing:
 abrt-addon-kerneloops        x86_64 2.6.1-5.fc22                 @System  79 k
 abrt-addon-pstoreoops        x86_64 2.6.1-5.fc22                 @System  14 k
 abrt-addon-vmcore            x86_64 2.6.1-5.fc22                 @System  42 k
 abrt-cli                     x86_64 2.6.1-5.fc22                 @System   0  
 kernel                       x86_64 4.0.4-301.fc22               @System   0  
 kernel                       x86_64 4.1.7-200.fc22               @System   0  
 kernel-core                  x86_64 4.0.4-301.fc22               @System  41 M
 kernel-core                  x86_64 4.1.7-200.fc22               @System  42 M
 kernel-modules               x86_64 4.0.4-301.fc22               @System  17 M
 kernel-modules               x86_64 4.1.7-200.fc22               @System  17 M
 libreport-plugin-kerneloops  x86_64 2.6.2-4.fc22                 @System  38 k
 pm-utils                     x86_64 1.4.1-30.fc22                @System 364 k



I'm proposing this as a blocker to Fedora 23 due to today's FESCo ruling[1] on upgrade behavior:
AGREED: "Upgrades use distro-sync by default. On resulting conflicts, the
 user is presented with the list of problematic packages and the choice of
 removing them as part of the upgrade or aborting. " (+7,0,0) (nirik,
 18:33:46)

[1] https://fedorahosted.org/fesco/ticket/1481#comment:11

Comment 16 Bruno Wolff III 2015-10-02 16:13:18 UTC
Bug 1253264 (which bug 1255463 was marked a duplicate of, despite having a more specific RFE request) covers the removing the running kernel causes the distro-sync to fail. It might be a better target than this one for a blocker. Though if this one has the same root cause as 1244175, it would be nice to get fixed, before fthe release of f23.
Note that we are looking for these fixes to happen in F21 and F22, which means they don't block f23 composes, but rather the release process (assuming they are accepted as blockers).

Comment 17 Adam Williamson 2015-10-05 22:26:08 UTC
Discussed at 2015-10-05 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-10-05/f23-blocker-review.2015-10-05-16.00.html . Accepted as a blocker per rationale in #c15 and #c16, and indeed this is a 'special' blocker (requiring updates to be pushed for all stable releases before F23 final release date).

Comment 18 Honza Silhan 2015-10-13 13:24:06 UTC
*** Bug 1266949 has been marked as a duplicate of this bug. ***

Comment 19 Honza Silhan 2015-10-13 14:16:59 UTC
*** Bug 1269178 has been marked as a duplicate of this bug. ***

Comment 20 Robin Stocker 2015-10-14 04:12:32 UTC
Any updates on this?

In previous Fedora releases, I used to upgrade as soon as the beta was out. This time, that's not yet possible, and I fear that there won't be much testing of upgraded systems. (The workarounds scare me a bit, though I might try the one with the F23 kernel.)

Comment 21 Honza Silhan 2015-10-14 11:50:26 UTC
We are working on it.

Part of the bug (comment 15) when it tries to remove running excluded kernel and protected_packages plugin allows that was fixed in the upstream [1]. It will never remove running kernel now.

We would like to get it fixed in libsolv if possible [2]. We work on DNF temporary fix though.

[1] https://github.com/rpm-software-management/hawkey/commit/e875ea61aad686e91833c50245fb03998b7674e7
[2] https://github.com/openSUSE/libsolv/issues/109

Comment 22 Fedora Update System 2015-10-14 17:00:19 UTC
dnf-1.1.3-1.fc23 dnf-plugins-extras-0.0.11-1.fc23 dnf-plugins-core-0.1.13-1.fc23 hawkey-0.6.2-1.fc23 libsolv-0.6.14-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-a011ac001a

Comment 23 Fedora Update System 2015-10-14 17:01:54 UTC
dnf-1.1.3-1.fc22 dnf-plugins-extras-0.0.11-1.fc22 dnf-plugins-core-0.1.13-1.fc22 hawkey-0.6.2-1.fc22 libsolv-0.6.14-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-02efefba0e

Comment 24 Honza Silhan 2015-10-14 17:13:07 UTC
* It was fixed in libsolv-0.6.14-2. builds for f21+ has been made.

Comment 25 Fedora Update System 2015-10-14 22:52:26 UTC
dnf-1.1.3-1.fc23, dnf-plugins-core-0.1.13-1.fc23, dnf-plugins-extras-0.0.11-1.fc23, hawkey-0.6.2-1.fc23, libsolv-0.6.14-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update hawkey dnf-plugins-core dnf-plugins-extras dnf libsolv'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-a011ac001a

Comment 26 Fedora Update System 2015-10-15 05:51:41 UTC
dnf-1.1.3-1.fc22, dnf-plugins-core-0.1.13-1.fc22, dnf-plugins-extras-0.0.11-1.fc22, hawkey-0.6.2-1.fc22, libsolv-0.6.14-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update hawkey dnf-plugins-core dnf-plugins-extras dnf libsolv'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-02efefba0e

Comment 27 Kamil Páral 2015-10-15 13:10:23 UTC
I have tested new builds and this problem seems to be gone. I have tested both --distro-sync and --distro-sync --allowerasing. In both cases I had to use --enablerepo=updates-testing, because since this update is still not pushed stable, there were some dependency issues in the meantime:

package python-dnf-1.1.3-1.fc22.noarch requires dnf-conf = 1.1.3-1.fc22, but none of the providers can be installed

Both upgrades went fine, so I think this is now fixed. Great job, Jan, thanks.

Comment 28 Adam Jackson 2015-10-15 13:37:51 UTC
I'm not sure this is really fixed, it doesn't work for me still:

hyoscyamine:~% sudo dnf --releasever 23 distro-sync --allowerasing
Last metadata expiration check performed 0:20:58 ago on Thu Oct 15 09:15:19 2015.
Dependencies resolved.
Error: The operation would result in removing the booted kernel: kernel-core-4.1.8-200.fc22.x86_64.
hyoscyamine:~% rpm -qa \*dnf\*
dnf-plugins-core-0.1.12-3.fc22.noarch
python-dnf-plugins-core-0.1.12-3.fc22.noarch
dnf-conf-1.1.3-1.fc22.noarch
dnf-yum-1.1.3-1.fc22.noarch
dnf-plugin-system-upgrade-0.4.1-1.fc22.noarch
dnf-langpacks-conf-0.15.1-1.fc22.noarch
python2-dnf-plugin-system-upgrade-0.4.1-1.fc22.noarch
python-dnf-1.1.3-1.fc22.noarch
dnf-langpacks-0.15.1-1.fc22.noarch
python-dnf-langpacks-0.15.1-1.fc22.noarch
python-dnf-plugins-extras-common-0.0.9-1.fc22.noarch
dnf-1.1.3-1.fc22.noarch
python-dnf-plugins-extras-migrate-0.0.9-1.fc22.noarch
hyoscyamine:~% rpm -qa kernel\*
kernel-modules-4.1.4-200.fc22.x86_64
kernel-devel-4.1.8-200.fc22.x86_64
kernel-4.1.4-200.fc22.x86_64
kernel-modules-4.1.10-200.fc22.x86_64
kernel-headers-4.1.10-200.fc22.x86_64
kernel-4.1.8-200.fc22.x86_64
kernel-4.1.10-200.fc22.x86_64
kernel-devel-4.1.10-200.fc22.x86_64
kernel-modules-4.1.8-200.fc22.x86_64
kernel-core-4.1.10-200.fc22.x86_64
kernel-devel-4.0.4-202.fc21.x86_64
kernel-devel-4.1.4-200.fc22.x86_64
kernel-core-4.1.4-200.fc22.x86_64
kernel-core-4.1.8-200.fc22.x86_64

Comment 29 Adam Jackson 2015-10-15 18:10:55 UTC
(In reply to Adam Jackson from comment #28)
> I'm not sure this is really fixed, it doesn't work for me still:

... but I hadn't updated hawkey or libsolv, nevermind.

Comment 30 Robin Stocker 2015-10-17 06:48:45 UTC
Thanks for the updates. It failed a few times more but I finally got it working with these options:

sudo dnf system-upgrade download --releasever=23 --distro-sync --allowerasing --enablerepo=updates-testing

Comment 31 Fedora Update System 2015-10-17 15:51:26 UTC
dnf-1.1.3-1.fc23, dnf-plugins-core-0.1.13-1.fc23, dnf-plugins-extras-0.0.11-1.fc23, hawkey-0.6.2-1.fc23, libsolv-0.6.14-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 32 Kamil Páral 2015-10-20 07:55:24 UTC
Reopening until the following update is pushed stable:
https://bodhi.fedoraproject.org/updates/FEDORA-2015-02efefba0e

Also, we're missing an update for F21. Jan, can you please create an update ASAP? Thanks!

Comment 33 Fedora Update System 2015-10-20 09:52:26 UTC
hawkey-0.5.3-3.fc21 libsolv-0.6.14-2.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-b3a8c0e68d

Comment 34 Kamil Páral 2015-10-22 07:51:46 UTC
*** Bug 1266589 has been marked as a duplicate of this bug. ***

Comment 35 Fedora Update System 2015-10-23 17:19:23 UTC
dnf-1.1.3-1.fc22, dnf-plugins-core-0.1.13-1.fc22, dnf-plugins-extras-0.0.11-1.fc22, hawkey-0.6.2-1.fc22, libsolv-0.6.14-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Comment 36 Fedora Update System 2015-10-26 10:28:58 UTC
hawkey-0.5.3-3.fc21, libsolv-0.6.14-2.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update hawkey libsolv'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-b3a8c0e68d

Comment 37 Fedora Update System 2015-10-28 16:19:12 UTC
hawkey-0.5.3-3.fc21, libsolv-0.6.14-2.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.

Comment 38 Kaan Cappon 2016-01-08 21:11:16 UTC
"I had the same error." See bug 1261569.

"cat /etc/system-release fedora release 23 but uname -r fc22"

I have two disk on my laptop and others Linux distributions. The boot sequence of the two discs was generated by other Linux distributions.
I started on the distribution that generated the file, and I restarted the laptop. Everything is in order.
Hope this can help.
Best regards.
Kaan.


Note You need to log in before you can comment on or make changes to this bug.