Bug 1961659

Description of problem:

We have some customers getting the emergency prompt below because the unpack of the initramfs isn't complete at the time the "shutdown" script in the initramfs executes "reboot":
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
/shutdown: line 115: reboot: command not found
[  145.951127] dracut Warning: reboot failed!
dracut Warning: reboot failed!


[...]

Dropping to debug shell.

shutdown:/# 
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

This happens rarely (something like once over 100 times), but it's quite critical since this leaves the system unusable and requires administrator's intervention.

Digging into this, I found out that there was an ordering cycle happening during shutdown, when using a generic initramfs (or any initramfs having the "nfs" module installed):
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
[  117.274806] systemd[1]: local-fs.target: Found ordering cycle on var-lib-nfs-rpc_pipefs.mount/stop
[  117.276183] systemd[1]: local-fs.target: Found dependency on systemd-tmpfiles-setup.service/stop
[  117.277666] systemd[1]: local-fs.target: Found dependency on local-fs.target/stop
[  117.278777] systemd[1]: local-fs.target: Job var-lib-nfs-rpc_pipefs.mount/stop deleted to break ordering cycle starting with local-fs.target/stop
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

When the ordering cycle is resolved by killing another job than "local-fs.target/stop" (as shown below), this doesn't create the condition (at least we observed that "local-fs.target/stop" was deleted when emergency was entered).

On my system, even having systemd delete "local-fs.target/stop" isn't enough to reproduce because it looks like it's due to the dynamic as well, in particular how long it takes for /usr/lib/dracut/dracut-initramfs-restore to restore the initramfs.

The ordering cycle appears because because /var/lib/nfs/rpc_pipefs mount isn't in sync with the mount unit defined as /usr/lib/systemd/system/var-lib-nfs-rpc_pipefs.mount:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
# systemctl status var-lib-nfs-rpc_pipefs.mount 
● var-lib-nfs-rpc_pipefs.mount - RPC Pipe File System
   Loaded: loaded (/proc/self/mountinfo; static; vendor preset: disabled)
   Active: active (mounted)
    Where: /var/lib/nfs/rpc_pipefs
     What: rpc_pipefs
    Tasks: 0 (limit: 10840)
   Memory: 0B
   CGroup: /system.slice/var-lib-nfs-rpc_pipefs.mount

# systemctl show var-lib-nfs-rpc_pipefs.mount | egrep "Before|After"
Before=local-fs.target rpc_pipefs.target
After=systemd-journald.socket system.slice systemd-tmpfiles-setup.service -.mount
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

From above, we see systemd didn't use /usr/lib/systemd/system/var-lib-nfs-rpc_pipefs.mount but generated a mount unit based on mountinfo.
This is due to having /var/lib/nfs/rpc_pipefs mounted IN the initramfs already.
Due to this, systemd automatically generates the mount unit and adds Before=local-fs.target which creates the ordering cycle on shutdown.


Version-Release number of selected component (if applicable):

systemd-239


How reproducible:

Sometimes on customer system, wasn't able to reproduce locally

Steps to Reproduce:
1. Install "dracut-config-generic" and rebuild the initramfs

  # yum -y install dracut-config-generic
  # dracut -f

2. Check the mount point after rebooting

  # systemctl show var-lib-nfs-rpc_pipefs.mount | egrep "Before|After"

Actual results:

  Before=local-fs.target rpc_pipefs.target
  After=systemd-journald.socket system.slice systemd-tmpfiles-setup.service -.mount

Expected results:

  No "Before=local-fs.target" dependency


Additional info:

This is closely related to BZ#1924587 (RFE) but happens in another condition (there is no timeout here).