Bug 1160079 - during reboot after fedup upgrade systemd hangs because /sysroot cannot be unmounted
Summary: during reboot after fedup upgrade systemd hangs because /sysroot cannot be un...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: fedup
Version: 19
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Will Woods
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-03 23:57 UTC by Zbigniew Jędrzejewski-Szmek
Modified: 2015-04-05 08:12 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-18 11:52:33 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
screenshot of hang (26.84 KB, image/png)
2014-11-03 23:57 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details
smartctl -x /dev/sda (10.72 KB, text/plain)
2014-12-15 15:57 UTC, Philip
no flags Details

Description Zbigniew Jędrzejewski-Szmek 2014-11-03 23:57:58 UTC
Created attachment 953358 [details]
screenshot of hang

Description of problem:
KVM machine with encrypted root. I run fedup --network 21 --instrepo
https://dl.fedoraproject.org/pub/alt/stage/21_Beta_RC4/Server/x86_64/os
--product=server, and rebooted. After the initial transaction was calculated, rpm said that I don't have enough free space and I was dropped into an emergency shell. I removed some cruft from the disk, pressed ^D, and the upgrade commenced and finished successfully. But during the reboot systemd said:

Failed unmounting /sysroot/proc
...
Unmounted /mnt
Unmounted /sysroot/dev
Unmounted /sysroot/sys/fs/cgroup
Failed unmounting /sysroot/sys
Failed unmounting /sysroot
...
A stop job is running for /dev/disk/by-uuid/xxxx-yyy-zzzz
A stop job is running for Cryptography setup for luks-zzz-yyy-xxx (.../no limit)

I waited 16 minutes, ctrl-alt-del caused systemd to cycle jobs but it always go to the same point. Hard reset was required. After reboot system booted fine, the upgrade was successful.

Most likely this is a systemd issue.


Version-Release number of selected component (if applicable):
Fully updated F19 system as of today.
fedup-0.9.0-2.fc19.
systemd-216-5.fc21 in the upgrade image. 

I made a snapshot before the upgrade so I'll try to retest this.

Comment 1 Philip 2014-12-12 18:26:20 UTC
I've had the same issue (without the not-enough-space part):

Fresh Fedora 19 installation, installed fedup, upgraded to Fedora 20. Preparation worked, reboot, upgrade finished without an error but the automatic reboot didn't work because it kept running a stop job which never finished. Ctrl + Alt + Del didn't help.

Then upgraded to Fedora 21, same thing - everything worked until the upgrade was complete, but the shutdown got stuck at "A stop job is running for Cryptography... (10min / no limit)". Also errors claiming that several /sysroot/* filesystems cannot be unmounted (like "Failed unmounting /sysroot/proc.").

The root filesystem is encrypted, which seems to be the trigger for this issue.

Right after upgrading to Fedora 21:
# rpm -qa | grep -E "(fedup|systemd|sysv)"
systemd-libs-216-12.fc21.x86_64
fedup-0.9.0-2.fc21.noarch
systemd-python3-216-12.fc21.x86_64
kcm_systemd-0.7.0-2.fc21.x86_64
systemd-216-12.fc21.x86_64
systemd-compat-libs-216-12.fc21.x86_64
systemd-python-216-12.fc21.x86_64
ntsysv-1.3.63-1.fc21.x86_64

Comment 2 Will Woods 2014-12-12 20:43:52 UTC
(In reply to Philip from comment #1)
> 
> The root filesystem is encrypted, which seems to be the trigger for this
> issue.

Encryption isn't sufficient to trigger this bug; this didn't happen on the encrypted systems I tested (e.g. my laptop), which had sufficient disk space. So there's another as-yet-undiscovered factor at work on your systems.

Philip, can you give some more details about your system? Is there anything unusual about it (low disk space, virtual disks, etc) that it would have in common with Zbigniew's system but not mine?

Comment 3 Philip 2014-12-15 15:51:58 UTC
(In reply to Will Woods from comment #2)
> Philip, can you give some more details about your system? Is there anything
> unusual about it (low disk space, virtual disks, etc) that it would have in
> common with Zbigniew's system but not mine?

It's a laptop with one hdd, which looks like this:
# parted /dev/sda print
Model: ATA HGST HTS725050A7 (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  525MB   524MB   primary  ext4         boot
 2      525MB   11.0GB  10.5GB  primary
 3      11.0GB  493GB   482GB   primary

sda1 is the boot partition, sda2 is swap, sda3 is the root partition (btrfs with a subvolume for home):
UUID=[dm-0] /                       btrfs   subvol=root,x-systemd.device-timeout=0 1 1
UUID=[sda1] /boot                   ext4    defaults        1 2
UUID=[dm-0] /home                   btrfs   subvol=home,x-systemd.device-timeout=0 1 2
/dev/mapper/luks-[sda2] swap                    swap    defaults,x-systemd.device-timeout=0 0 0
(I've replaced the actual UUIDs with [name] where name is the UUID according to /dev/disk/by-uuid/.)

The btrfs volume has about 436 GB of free space, so that's not it.

So the root volume (where / is mounted) is not directly on a hdd partition, but rather a btrfs volume inside an encrypted container (which is on sda3).

As for the comparison with Zbigniew's system, I'm not using KVM, but other than that, I'm not sure what to compare.

If there something specific I could check?

Comment 4 Philip 2014-12-15 15:57:42 UTC
Created attachment 969110 [details]
smartctl -x /dev/sda

My hdd is not defective either, here's the smartctl output.

Comment 5 David H. Gutteridge 2014-12-30 06:10:48 UTC
I just encountered this same issue when upgrading my laptop from 20 to 21 using Fedup. (After a hard reset, I found the installation was successful.) I can provide details as needed, but there's nothing exotic about my setup as far as I know. (There was no disk space issue, and I'm not using any virtualization. All my partitions are ext4.)

Comment 6 Robin Laing 2015-01-02 04:44:10 UTC
Upgrading from Fedora 19 to Fedora 21 using FedUp.

Procedure from the wiki, went well until the shutdown.  Left it as we were busy and it kept "A stop job is running for Cryptography... (... / no limit)" for over 90 minutes.

http://fedoraproject.org/wiki/FedUp

Followed comment 5 for the hard reboot.

Finished the procedures after the reboot as per the wiki.  Maybe the comment about this could be added to the wiki.


fedup --network 21 --poduct=nonproduct

ASUS laptop fully encrypted partitions.  No LVM.

Comment 7 Fedora End Of Life 2015-01-09 21:32:43 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 8 Fedora End Of Life 2015-02-18 11:52:33 UTC
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 9 Axel Sommerfeldt 2015-04-05 08:12:45 UTC
Same happened to me when updating F21 to F22 with Fedup, the update process halted with "Failed unmounting /sysroot/proc". However pressing Ctrl-Alt-Del solved this and except this the update went fine.


Note You need to log in before you can comment on or make changes to this bug.