Bug 1645499

Summary: Unmounting file systems fail on shutdown
Product: Red Hat Enterprise Linux 7 Reporter: Marko Myllynen <myllynen>
Component: systemdAssignee: systemd-maint
Status: CLOSED WONTFIX QA Contact: qe-baseos-daemons
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.6CC: dtardon, katherinewallitonejohn, otheus.uibk, redhat-bugzilla, systemd-maint-list
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-15 07:31:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marko Myllynen 2018-11-02 11:25:14 UTC
Description of problem:
Seen on RHEL 7.6 minimal installation during system poweroff/reboot:

...
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Stopped Security Auditing Service.
[  OK  ] Stopped Create Volatile Files and Directories.
[  OK  ] Stopped Import network configuration from initramfs.
[  OK  ] Stopped target Local File Systems.
         Unmounting Configuration File System...
         Unmounting /boot/efi...
[FAILED] Failed unmounting Configuration File System.
[  OK  ] Unmounted /boot/efi.
[  OK  ] Reached target Unmount All Filesystems.
...

Version-Release number of selected component (if applicable):
systemd-219-62.el7.x86_64

Comment 2 Marko Myllynen 2018-11-02 13:27:14 UTC
Occasionally on the same test UEFI VM also seeing:

...
[  OK  ] Unmounted /boot/efi.
         Unmounting /boot...
[FAILED] Failed unmounting /boot.
[  OK  ] Unmounted /run/user/0.
...

And if using a storage layout as follows:

root@localhost:~# dmsetup ls
luks-fe535275-d351-4518-91e4-4360031179fe	(253:3)
luks-6b68857c-bb05-45ea-b76a-f4ff4d6d90e5	(253:2)
rhel-swap	(253:1)
rhel-root	(253:0)
root@localhost:~# lsblk -i  
NAME                                        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda                                         252:0    0    8G  0 disk  
|-vda1                                      252:1    0  200M  0 part  /boot/efi
|-vda2                                      252:2    0  512M  0 part  /boot
`-vda3                                      252:3    0  7.3G  0 part  
  |-rhel-root                               253:0    0  6.3G  0 lvm   
  | `-luks-fe535275-d351-4518-91e4-4360031179fe
                                            253:3    0  6.3G  0 crypt /
  `-rhel-swap                               253:1    0    1G  0 lvm   
    `-luks-6b68857c-bb05-45ea-b76a-f4ff4d6d90e5
                                            253:2    0 1022M  0 crypt [SWAP]
root@localhost:~# 

Then also the following error is seen (something for dm-2, sometimes for dm-3):

...
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Stopped target Swap.
         Deactivating swap /dev/disk/by-id/d...c-bb05-45ea-b76a-f4ff4d6d90e5...
[  OK  ] Deactivated swap /dev/disk/by-uuid/...fb4-59e5-4f74-95ad-cc529e429711.
[  OK  ] Deactivated swap /dev/disk/by-id/dm...57c-bb05-45ea-b76a-f4ff4d6d90e5.
[  OK  ] Deactivated swap /dev/disk/by-id/dm...57c-bb05-45ea-b76a-f4ff4d6d90e5.
[  OK  ] Deactivated swap /dev/dm-3.
[FAILED] Stopped (with error) /dev/dm-3.
[  OK  ] Deactivated swap /dev/mapper/luks-6...57c-bb05-45ea-b76a-f4ff4d6d90e5.
[  OK  ] Stopped Monitoring of LVM2 mirrors,...ng dmeventd or progress polling.
         Stopping LVM2 metadata daemon...
[  OK  ] Stopped LVM2 metadata daemon.
[  OK  ] Closed LVM2 metadata daemon socket.
[  OK  ] Stopped /dev/mapper/luks-6b68857c-bb05-45ea-b76a-f4ff4d6d90e5.
[  OK  ] Stopped /dev/disk/by-uuid/98397fb4-59e5-4f74-95ad-cc529e429711.
[  OK  ] Stopped /dev/disk/by-id/dm-uuid-CRY...57c-bb05-45ea-b76a-f4ff4d6d90e5.
[  OK  ] Stopped /dev/disk/by-id/dm-name-luk...57c-bb05-45ea-b76a-f4ff4d6d90e5.
[  OK  ] Stopped /sys/devices/virtual/block/dm-3.
[  OK  ] Stopped Cryptography Setup for luks...57c-bb05-45ea-b76a-f4ff4d6d90e5.
...

Comment 3 Jan Synacek 2018-11-05 10:01:55 UTC
With this amount of information, there's nothing we can do about that. Also, I can't reproduce this.

Comment 4 Marko Myllynen 2018-11-05 13:43:23 UTC
(In reply to Jan Synacek from comment #3)
> With this amount of information, there's nothing we can do about that. Also,
> I can't reproduce this.

Then you request the data you need, not close the bug report. Thanks.

Comment 5 Marko Myllynen 2018-11-06 07:25:37 UTC
I've now uploaded a kickstart to my GitHub repo that can be used to reproduce this.

1) Setup environment for installing kickstarted VMs on RHEL 7.6 host
2) Use https://github.com/myllynen/misc/blob/master/rhel-7-base.ks
3) Observe the issues on the guest console

I used 'virt-install --connect qemu:///system --name test --virt-type kvm --arch x86_64 --vcpus 2 --cpu host --ram 2048 --os-type linux --os-variant rhel7.6 --disk pool=default,format=qcow2,cache=none,io=native,size=8 --network network=default --graphics vnc --noreboot --location http://192.168.122.1/mnt/ --extra-args "ip=dhcp inst.ks=http://192.168.122.1/ks/rhel-7-base.ks console=tty0 console=ttyS0,115200 quiet systemd.show_status=yes" --noautoconsole' to install the guest, then 'virsh console --domain test --devname serial0' to attach to its console, log in as root, and reboot the guest to see the "Failed unmounting Configuration File System" message.

For the other two messages, I replaced autopartitioning with:

part /boot --fstype=xfs --ondisk=vda --size=512
part pv.01 --fstype="lvmpv" --ondisk=vda --size=1 --grow
volgroup rhel pv.01
logvol / --fstype=xfs --grow --size=1024 --name=root --vgname=rhel --encrypted
logvol swap  --fstype=swap --size=1024 --name=swap --vgname=rhel --encrypted

The "Stopped (with error) /dev/dm-3" error shows up every time, however "Failed unmounting /boot" shows up rather randomly (both on BIOS or UEFI).

Comment 6 Marko Myllynen 2018-11-12 08:17:22 UTC
And when installing with something like "virt-install  ... --location rhel-server-7.6-x86_64-dvd.iso --initrd-inject rhel-7-base.ks --extra-args "inst.ks=file:/rhel-7-base.ks" we see this when the installer is shutting down:

...
[  OK  ] Stopped Configure read-only root support.
[  OK  ] Unmounted /mnt/sysimage/dev/shm.
[  OK  ] Unmounted Configuration File System.
[  OK  ] Unmounted /mnt/sysimage/proc.
[  OK  ] Unmounted Temporary Directory.
[FAILED] Failed unmounting /run/install/repo.
[  OK  ] Unmounted /mnt/sysimage/run.
[  OK  ] Stopped target Swap.
[  OK  ] Unmounted /mnt/sysimage/sys/fs/selinux.
...

So looks like systemd does not handle unmounting of filesystems properly in many occasions.

Comment 7 Robert Scheck 2019-05-23 12:59:52 UTC
I'm seeing the same here, it feels like systemd is unable to umount configfs properly. This bug could be similar like bug #1630378.

Comment 8 Otheus 2020-05-27 20:56:37 UTC
I experienced the same problem, and it was definitvely linked to biosdevname=0. The failure at shutdown was
 [FAILED] Failed unmounting Configuration File System.
which delayed the shutdown/reboot by 30 seconds. This is a virual machine running under RHEV-M.

Comment 11 RHEL Program Management 2021-03-15 07:31:06 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 13 zehenna 2023-05-27 04:24:20 UTC Comment hidden (spam)