Bug 1303700

I have tried this:

touch /.autorelabel
reboot
add rd.break=pre-shutdown to kernel cmdline
boot

With old initscript package, after relabel, the machine is rebooted -> shutdown dracut was not run.
With new initscript package, after relabel, I got a shell in dracut -> shutdown dracut was run.

Can you please also test this on your machines? If you get a same result, then there is some other bug.

Lukas, thanks for a hint with pre-shutdown break. The real problem was my root is on standard partition. I've tried new installation where root is on xfs over lvm and magically pre-shutdown break, root unmount before reboot, and clean mount works as expected.
I'm not able to recognize if it's still problem of initscripts. If not, switch this bug back to ON_QA, and I'll file new one on this.

=Logs from lvm system:=
...after selinux relabel
[   26.677414] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[   26.691238] systemd-journald[499]: Received SIGTERM from PID 1 (systemd-shutdow).
[   26.704429] type=1305 audit(1457512049.800:12): audit_pid=0 old=658 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   26.719510] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[   26.726746] systemd-shutdown[1]: Unmounting file systems.
[   26.776541] systemd-shutdown[1]: Unmounting /boot.
[   26.781222] XFS (vda1): Unmounting Filesystem
[   26.843621] systemd-shutdown[1]: All filesystems unmounted.
[   26.846079] systemd-shutdown[1]: Deactivating swaps.
[   26.848564] systemd-shutdown[1]: Deactivating swap /dev/dm-1.
[   26.850877] systemd-shutdown[1]: All swaps deactivated.
[   26.853486] systemd-shutdown[1]: Detaching loop devices.
[   26.856749] systemd-shutdown[1]: All loop devices detached.
[   26.860029] systemd-shutdown[1]: Detaching DM devices.
[   26.862707] systemd-shutdown[1]: Detaching DM 253:1.
[   26.875426] systemd-shutdown[1]: Not all DM devices detached, 1 left.
[   26.877910] systemd-shutdown[1]: Detaching DM devices.
[   26.883574] systemd-shutdown[1]: Not all DM devices detached, 1 left.
[   26.887338] systemd-shutdown[1]: Cannot finalize remaining DM devices, continuing.
[   26.895830] systemd-shutdown[1]: Successfully changed into root pivot.
[   26.898130] systemd-shutdown[1]: Returning to initrd...


[   27.020938] dracut Warning: Break before pre-shutdown
dracut Warning: Break before pre-shutdown


Generating "/run/initramfs/rdsosreport.txt"
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

To get more debug information in the report,
reboot with "rd.debug" added to the kernel command line.

Dropping to debug shell.

pre-shutdown:/# [   29.561688] dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
[   29.629610] XFS (dm-0): Unmounting Filesystem
[   29.663234] dracut Warning: Unmounted /oldroot.
[   29.683074] dracut: Disassembling device-mapper devices
Rebooting.
[   29.716186] Unregister pv shared memory for cpu 1
[   29.719215] Unregister pv shared memory for cpu 0
[   29.749885] Restarting system.
[   29.750611] reboot: machine restart
<snip>
[    2.713568] XFS (dm-0): Mounting V4 Filesystem
[    2.816095] XFS (dm-0): Ending clean mount

# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda           252:0    0    9G  0 disk 
├─vda1        252:1    0  500M  0 part /boot
└─vda2        252:2    0  8.5G  0 part 
  ├─rhel-root 253:0    0  7.6G  0 lvm  /
  └─rhel-swap 253:1    0  924M  0 lvm  [SWAP]



=Logs from standard partition system:=
...after selinux relabel
[   27.565751] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[   27.573607] systemd-journald[383]: Received SIGTERM from PID 1 (systemd-shutdow).
[   27.587367] type=1305 audit(1457512073.307:12): audit_pid=0 old=528 auid=4294967295 ses=4294967295 subj=system_u:s
ystem_r:auditd_t:s0 res=1
[   27.618825] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[   27.623113] systemd-shutdown[1]: Unmounting file systems.
[   27.633162] systemd-shutdown[1]: Unmounting /boot.
[   27.636813] XFS (vda1): Unmounting Filesystem
[   27.720848] systemd-shutdown[1]: All filesystems unmounted.
[   27.722769] systemd-shutdown[1]: Deactivating swaps.
[   27.725581] systemd-shutdown[1]: Deactivating swap /dev/vda2.
[   27.728484] systemd-shutdown[1]: All swaps deactivated.
[   27.731143] systemd-shutdown[1]: Detaching loop devices.
[   27.735062] systemd-shutdown[1]: All loop devices detached.
[   27.738523] systemd-shutdown[1]: Detaching DM devices.
[   27.740761] systemd-shutdown[1]: All DM devices detached.
[   27.745738] systemd-shutdown[1]: Rebooting.
[   27.747210] Unregister pv shared memory for cpu 1
[   27.748884] Unregister pv shared memory for cpu 0
[   27.778396] Restarting system.
<snip>
[    2.127519] XFS (vda3): Mounting V4 Filesystem
[    2.135680] XFS (vda3): Starting recovery (logdev: internal)
[    2.150651] XFS (vda3): Ending recovery (logdev: internal)

# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    253:0    0    9G  0 disk 
├─vda1 253:1    0  500M  0 part /boot
├─vda2 253:2    0  922M  0 part [SWAP]
└─vda3 253:3    0  7.6G  0 part /

In systemd-shutdown we first try to remount filesystem read-only and then we attempt to unmount the filesystem. In case of root fs this fails, but that is expected. After we unmounted everything we could, we then call sync and reboot/shutdown/kexec depending on options.

I don't see any obvious problem with approach described above. I suspect that XFS log message on next boot is probably a bug/known-issue in XFS. I don't know, let's try to ask Dave.

(In reply to Michal Sekletar from comment #9)
> In systemd-shutdown we first try to remount filesystem read-only and then we
> attempt to unmount the filesystem. In case of root fs this fails, but that
> is expected. After we unmounted everything we could, we then call sync and
> reboot/shutdown/kexec depending on options.
> 
> I don't see any obvious problem with approach described above. I suspect
> that XFS log message on next boot is probably a bug/known-issue in XFS. I
> don't know, let's try to ask Dave.

Sync doesn't provide any guarantees that the log is clean - all it guarantees is that after a subsequent mount all the data and metadata is present in the filesystem. XFS, like ext4, only writes dirty metadata to the log when sync runs, hence requiring log recovery to run to provide the requirements of sync.

Unmount is different - it requires us to write back all the dirty metadata to it's allocated blocks, hence leaving the log clean and not requiring recovery on teh next mount. This is why unmount often takes a lot longer to execute than sync.

IOWs, XFS is behaving exactly as we expect it to here. sync + recovery gives the same result as unmount once the subsequent mount is completed. The only difference is where we take the performance hit for writing back dirty metadata (unmount or log recovery)....

-Dave.

So it looks like we've done everything in the initscripts scope, I'm going to switch this bug to verified and file new one on kernel as Lukas suggested.

(In reply to Michal Sekletar from comment #9)
> In systemd-shutdown we first try to remount filesystem read-only and then we
> attempt to unmount the filesystem. In case of root fs this fails, but that
> is expected.

Why is this unique to the setenforce / relabel scenario?  A normal reboot does not cause a log replay on the root fs.

(In reply to Eric Sandeen from comment #12)
> (In reply to Michal Sekletar from comment #9)
> > In systemd-shutdown we first try to remount filesystem read-only and then we
> > attempt to unmount the filesystem. In case of root fs this fails, but that
> > is expected.
> 
> Why is this unique to the setenforce / relabel scenario?  A normal reboot
> does not cause a log replay on the root fs.

I think there is no difference if / is on XFS on standard partition. Such setup doesn't require pivot_root to initramfs during shutdown and we only sync() the filesystem. Hence log replay on next mount. But as Dave already pointed out, this is fine, because we are guaranteed that *all* data and meta-data is on disk.

Previously in relabel scenario, we would never restore content of initramfs and pivot_root there even in cases this is necessary. I assume this can cause filesystem corruption in some cases because filesystem *thinks* that data are actually on disk but they are not. Instead are queued for write-back somewhere in DM layer. But as mentioned, we would never pivot_root so we can actually unmount old root fs and detach underlying DM device. This is now fixed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0526.html