Bug 1406829 - Upgrade hangs on "Starting Switch Root..."
Summary: Upgrade hangs on "Starting Switch Root..."
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 24
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Will Woods
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-21 15:15 UTC by Maciej Sitarz
Modified: 2017-08-08 19:29 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-08 19:29:07 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Maciej Sitarz 2016-12-21 15:15:23 UTC
Description of problem:
I'm not able to upgrade from Fedora 24 to 25. After rebooting the system with 'dnf system-upgrade reboot' the system boots, but the upgrade does not start. Interactive console or UI does not start. Last message on the screen is "Starting Switch Root..."

Version-Release number of selected component (if applicable):
# rpm -q dnf-plugin-system-upgrade systemd
dnf-plugin-system-upgrade-0.7.1-3.fc24.noarch
systemd-229-16.fc24.x86_64


How reproducible:
Always reproducible. I'm still on Fedora 24, didn't found a way to upgrade to Fedora 25 yet.

Steps to Reproduce:
1. dnf upgrade --refresh -y
2. dnf install dnf-plugin-system-upgrade -y
3. dnf system-upgrade download -y --refresh --releasever=25 --allowerasing 
4. dnf system-upgrade reboot


Actual results:
Upgrade process does not start. There's no way to login on any TTY (except of those debug ones).
System seems to be processing something (CPU fans go 100%), but no progress after few hours.

systemd process PID 1 is using 100% of one CPU.



Expected results:
Upgrade process should start after reboot.


Additional info:

# dnf system-upgrade log
Contacting OpenClient Router for restricted repository information
Added restricted repo: Fedora-24-x86_64-Licensed
The following boots appear to contain upgrade logs:
-- no logs were found --

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.8.14-200.fc24.x86_64 root=/dev/mapper/vg_ssd_msitarz-lv_root ro rd.md=0 rd.dm=0 rd.lvm.lv=vg_ssd_msitarz/lv_swap rd.lvm.lv=vg_ssd_msitarz/lv_root KEYTABLE=pl2 rd.luks.uuid=luks-50a340b3-e92f-4e48-8927-ffbd7ee2fe5f SYSFONT=True LANG=en_US.UTF-8 rd.driver.blacklist=nouveau resume=/dev/mapper/vg_ssd_msitarz-lv_swap rhgb quiet i915.modeset=1 pcie_aspm=force i915.i915_enable_rc6=7 i915.i915_enable_fbc=1 i915.semaphores=1 i915.lvds_downclock=1 acpi_osi=Linux video.use_native_backlight=1

# rpm -qa "plymouth*"
plymouth-0.8.9-17.2013.08.14.fc24.x86_64
plymouth-scripts-0.8.9-17.2013.08.14.fc24.x86_64
plymouth-core-libs-0.8.9-17.2013.08.14.fc24.x86_64
plymouth-system-theme-0.8.9-17.2013.08.14.fc24.x86_64
plymouth-plugin-two-step-0.8.9-17.2013.08.14.fc24.x86_64
plymouth-plugin-label-0.8.9-17.2013.08.14.fc24.x86_64
plymouth-graphics-libs-0.8.9-17.2013.08.14.fc24.x86_64
plymouth-theme-charge-0.8.9-17.2013.08.14.fc24.x86_64

# rpm -V systemd|grep etc
S.5....T.  c /etc/sysctl.conf
S.5....T.  c /etc/systemd/journald.conf

# gdb --pid=1 --batch -ex "bt"

warning: the debug information found in "/usr/lib/debug//lib64/libselinux.so.1.debug" does not match "/lib64/libselinux.so.1" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//lib64/librt-2.23.so.debug" does not match "/lib64/librt.so.1" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//usr/lib64/librt-2.23.so.debug" does not match "/lib64/librt.so.1" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//lib64/libpthread-2.23.so.debug" does not match "/lib64/libpthread.so.0" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//usr/lib64/libpthread-2.23.so.debug" does not match "/lib64/libpthread.so.0" (CRC mismatch).
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: the debug information found in "/usr/lib/debug//lib64/libc-2.23.so.debug" does not match "/lib64/libc.so.6" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//usr/lib64/libc-2.23.so.debug" does not match "/lib64/libc.so.6" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//lib64/ld-2.23.so.debug" does not match "/lib64/ld-linux-x86-64.so.2" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//usr/lib64/ld-2.23.so.debug" does not match "/lib64/ld-linux-x86-64.so.2" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//lib64/libpcre.so.1.2.7.debug" does not match "/lib64/libpcre.so.1" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//usr/lib64/libpcre.so.1.2.7.debug" does not match "/lib64/libpcre.so.1" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//lib64/libdl-2.23.so.debug" does not match "/lib64/libdl.so.2" (CRC mismatch).
warning: the debug information found in "/usr/lib/debug//usr/lib64/libdl-2.23.so.debug" does not match "/lib64/libdl.so.2" (CRC mismatch).
0x0000562b472f5393 in hashmap_iterate_entry.lto_priv.355 (h=h@entry=0x562b48681690, i=0x7ffede2f6c30) at src/basic/hashmap.c:679
679     }
#0  0x0000562b472f5393 in hashmap_iterate_entry.lto_priv.355 (h=h@entry=0x562b48681690, i=0x7ffede2f6c30) at src/basic/hashmap.c:679
#1  0x0000562b472f5672 in internal_hashmap_iterate (h=<optimized out>, i=<optimized out>, value=<optimized out>, key=<optimized out>, key=<optimized out>, value=<optimized out>) at src/basic/hashmap.c:686
#2  0x0000562b47377087 in unit_gc_sweep.lto_priv.975 (u=0x562b487ce180, gc_marker=gc_marker@entry=1732) at src/core/manager.c:833
#3  0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487d0750, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#4  0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487d15d0, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#5  0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487ccbc0, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#6  0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487cad30, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#7  0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487cd300, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#8  0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487ca2a0, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#9  0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487cb840, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#10 0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487ce8c0, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#11 0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487d1d10, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#12 0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487cc480, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#13 0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487d0e90, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#14 0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487dc930, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#15 0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487f0250, gc_marker=gc_marker@entry=1732) at src/core/manager.c:834
#16 0x0000562b4737705c in unit_gc_sweep.lto_priv.975 (u=0x562b487ba440, gc_marker=1732) at src/core/manager.c:834
#17 0x0000562b473663a6 in manager_dispatch_gc_queue (m=0x562b48655c30) at src/core/manager.c:881
#18 manager_loop (m=0x562b48655c30) at src/core/manager.c:2035
#19 0x0000562b472a309d in main (argc=<optimized out>, argv=<optimized out>) at src/core/main.c:1833



How to recover from this situation:
- Enabling debug shell before starting the upgrade is a good idea (systemctl add-wants system-update.target debug-shell.service)
- Removing /system-update and a hard reset is enough to go back to old working system (rm /system-update)

Comment 1 Maciej Sitarz 2017-01-18 10:34:32 UTC
Today (Wed Jan 18 11:32:57 CET 2017) I tried the upgrade procedure one more time.
Fedora 24 got some updates, also there were some new packages to download for Fedora 25.
The problem still exists and the symptoms are the same.

Comment 2 Maciej Sitarz 2017-02-28 12:37:34 UTC
I once again tried the upgrade procedure (Tue Feb 28 13:31:02 CET 2017).
Fedora 24 got some updates, also there were some new packages to download for Fedora 25.
The problem still exists and the symptoms are the same.

(I'm changing the Component to: systemd)

Any ideas how to fix it or a workaround?
Maybe I could somehow get the latest systemd installed and retest?

I wanted to submit the problem to systemd GitHub, but the only "support" two last releases of systemd and suggest submitting to distributions bugtracker for older versions.

Comment 3 Maciej Sitarz 2017-05-29 08:16:27 UTC
I managed to upgrade the system.
What I did is:
1. Rebuild systemd from SRPM package for Fedora 25 using systemd-231-14.fc25.src.rpm on Fedora 24
2. Install the systemd package built above (previously removed samba-client-libs skipping dependency check because it conflicted)
3. Rebooted to upgrade process
4. After upgrade finished reinstalled samba-client-libs


After that systemd update the system upgrade went smoothly.


I'm not closing this bug report, maybe the developer would like to describe this problem somewhere in the docs.

Comment 4 Fedora End Of Life 2017-07-26 00:07:30 UTC
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 5 Fedora End Of Life 2017-08-08 19:29:07 UTC
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.