Bug 2129834
| Summary: | Ordering cycle at shutdown due to /var/lib/nfs/rpc_pipefs mount | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Renaud Métrich <rmetrich> |
| Component: | dracut | Assignee: | dracut-maint-list |
| Status: | CLOSED MIGRATED | QA Contact: | qe-baseos-daemons |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 8.6 | CC: | dtardon, myamazak, ralston |
| Target Milestone: | rc | Keywords: | Bugfix, MigratedToJIRA, Triaged |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-09-20 20:25:52 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This is a duplicate of bug 2014502. But, in this particular case, it looks like we could workaround the problem by passing "-o _netdev" to the mount call. *** Bug 2181425 has been marked as a duplicate of this bug. *** Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: When the system is booting with the NFS dracut modules, e.g. when using the generic initramfs (due to presence of dracut-config-generic package), the following ordering cycle is seen at shutdown: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found ordering cycle on var-lib-nfs-rpc_pipefs.mount/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on systemd-tmpfiles-setup.service/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on local-fs.target/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Job var-lib-nfs-rpc_pipefs.mount/stop deleted to break ordering cycle starting with local-fs.target/stop -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- After much digging, I was able to find out that the reason was having an issue with /var/lib/nfs/rpc_pipefs mount: the unit has the following ordering dependencies (as seen by "systemd-analyze dump"): -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Before: local-fs.target (origin-mountinfo-implicit) Before: rpc_pipefs.target (destination-file) Before: umount.target (origin-mountinfo-default) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- whereas it should not and just have those: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Before: rpc_pipefs.target (destination-file) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Additionally (but unrelated), in such case, we can see the mount is not correct, it is mounted as: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- whereas it should be mounted as (see *What*): -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- The root cause is the NFS dracut module: 1. firstly mounting with What == *rpc_pipefs* instead of What == *sunrpc* (this causes the discrepancy in mount parameters): -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- # diff -u /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh.orig /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh --- /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh.orig 2022-09-26 13:34:11.416891907 +0200 +++ /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh 2022-09-26 13:34:19.152877777 +0200 @@ -4,7 +4,7 @@ if modprobe sunrpc || strstr "$(cat /proc/filesystems)" rpc_pipefs; then [ ! -d /var/lib/nfs/rpc_pipefs/nfs ] && \ - mount -t rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs + mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs # Start rpcbind or rpcbind # FIXME occasionally saw 'rpcbind: fork failed: No such device' -- why? -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- 2. secondly the cleanup doing a bind-mount of /var/lib/nfs/rpc_pipefs to /sysroot/var/lib/nfs/rpc_pipefs, which leads to having the unexpected "Before=local-fs.target umount.target" dependencies and "origin-mountinfo-implicit/origin-mountinfo-default" -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- # diff -u /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh.orig /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh --- /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh.orig 2022-09-26 13:43:19.441194953 +0200 +++ /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh 2022-09-26 13:43:59.199659403 +0200 @@ -21,7 +21,7 @@ if [ -d $NEWROOT/$rpcpipefspath ]; then # mount --move does not seem to work??? - mount --bind /var/lib/nfs/rpc_pipefs $NEWROOT/$rpcpipefspath + #mount --bind /var/lib/nfs/rpc_pipefs $NEWROOT/$rpcpipefspath umount /var/lib/nfs/rpc_pipefs 2>/dev/null else umount /var/lib/nfs/rpc_pipefs 2>/dev/null -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Note: here above, not remounting to /sysroot is probably NOT correct when having a NFS root; this hack is just to show what fixes the issue on regulard systems actually. Version-Release number of selected component (if applicable): dracut-network-049-202.git20220511.el8_6.x86_64 and before How reproducible: Always Steps to Reproduce: 1. Install dracut-config-generic package # yum -y install dracut-config-generic 2. Rebuild the initramfs # dracut -f 3. Setup persistent journal and boot twice # mkdir -p /var/log/journal # reboot ... # reboot 4. Check ordering cycles at shutdown # journalctl -b -1 | grep local-fs.target Actual results: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found ordering cycle on var-lib-nfs-rpc_pipefs.mount/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on systemd-tmpfiles-setup.service/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on local-fs.target/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Job var-lib-nfs-rpc_pipefs.mount/stop deleted to break ordering cycle starting with local-fs.target/stop -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Expected results: No ordering cycle