Bug 1552242
| Summary: | VDO mounts in fstab can cause "Breaking order cycle" event when permissive or enforcing mode is enabled | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Dennis Keefe <dkeefe> |
| Component: | vdo | Assignee: | Dennis Keefe <dkeefe> |
| Status: | CLOSED NOTABUG | QA Contact: | Jakub Krysl <jkrysl> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.5 | CC: | awalsh, bgurney, jkrysl, limershe, loberman, mvollmer |
| Target Milestone: | rc | ||
| Target Release: | 7.6 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-07-10 23:00:18 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1546181 | ||
It has been proposed that either using a mount file unit file, such as: ----------------------------- [Unit] Description = Mount file system that lives on /VDO name = VDO.mount Requires = vdo.service Conflicts = umount.target [Mount] What = /dev/mapper/vdo Where = /VDO Type = xfs Options = discard [Install] WantedBy = multi-user.target ----------------------------- instead of the fstab entry will correct this issue or, modifying the vdo.service file to: ----------------------------- [Unit] Description=VDO volume services After=systemd-remount-fs.service DefaultDependencies=no Conflicts=shutdown.target Before=local-fs-pre.target [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/bin/vdo start --all --confFile /etc/vdoconf.yml ExecStop=/usr/bin/vdo stop --all --confFile /etc/vdoconf.yml [Install] WantedBy=local-fs-pre.target ----------------------------- With this change to the vdo.service file there is no longer a need for adding x-systemd.requires=vdo.service in the fstab file. It is important that VDO doesn't delay or prevent the system from starting up if there is a rebuild of the volume. If the process of VDO rebuilding prevents other services from starting using a different "Before" target then the solution might not be favorable. Here is the current vdo.service configuration # /usr/lib/systemd/system/vdo.service [Unit] Description=VDO volume services After=systemd-remount-fs.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/bin/vdo start --all --confFile /etc/vdoconf.yml ExecStop=/usr/bin/vdo stop --all --confFile /etc/vdoconf.yml [Install] WantedBy=multi-user.target ----------------------- Some testing will be required. Note: the vdo.service unit file has not been tested yet. Adding this change to vdo.service file makes mounting VDO volumes easier and fixes the problem of "Breaking order cycle", but adds a "cost" of delaying the boot cycle if the VDO volume has not been cleanly shutdown. VDO volumes perform a recovery after unclean shutdowns. If the vdo.service file is changed to use "local-fs-pre.target", then the system will wait until VDO has completed it's recovery before proceeding. The time it takes to finish a recovery can depend on the VDO volume size, speed of the CPU and media. Simulating this could be done by creating a VDO volume on a USB thumb drive (slow media), writing data to that device, then crashing the system. After the system has been rebooted, run systemd-analyze blame to see how long VDO took to start the volume(s). In my test it took about 90 seconds. During the recovery process the system can't be accessed, since systemd hasn't reached the point of starting SSH or other services to manage the system. Using the mount unit file instead of changing the vdo.service fill will mount vdo volumes after the system has booted and doesn't cause the "Breaking order cycle". Pro/Con 1. vdo.service - would make mounting VDO volumes easier for many users - But VDO recoveries can extend the boot cycle 2. mount unit - adds additional steps, which are not intuitive to everyone - VDO does not get in the way of the boot up process or system access Adding the proposed vdo.service file can delay the boot cycle when VDO volumes need to recover, therefore we should not implement this feature. Users should create mount unit files for VDO volumes unless there is an alternate method of initializing vdo volume that doesn't delay the boot or break ordering. What about adding both x-systemd.requires=vdo.service and _netdev to /etc/fstab? That should also get rid of the cycle and doesn't require people (and Cockpit) to switch from /etc/fstab to something new. _netdev may allow for this to work, but we're not really a network device, so it doesn't fit the definition exactly. I think it would be incorrect for that reason. Maybe we can add a documentation note in the Storage Admin guide to specify that option noting that we're not a network device, but it achieves the goal for SELinux compliance? If someone had iscsi storage and created a VDO volume on top, then a filesystem, then NFS, would using _netdev have problems figuring out which network device / service to start first (nfs, vdo, iscsi)? This issue is resolved by creating a mount unit file https://bugzilla.redhat.com/show_bug.cgi?id=1552242 While using the "Before" option mentioned above can also resolve this issue it can have the side effect of extending the boot process. Closing ticket |
Description of problem: Adding a mount option in fstab for VDO, such as: /dev/mapper/vdo /VDO xfs defaults,discards,x-systemd.requires=vdo.service 0 0 Can cause events of "Breaking ordering cycle" journal -b |egrep "Breaking|Found" This is caused by systemd adding a dependency for local-fs.target the VDO.mount unit file. Version-Release number of selected component (if applicable): vdo-6.1.0.146-16.x86_64 How reproducible: 1. Create a VDO volume (block device needed) # vdo create --name=vdo1 --device=/dev/vda 2. Format VDO device # mkfs.xfs -K /dev/mapper/vdo1 3. Create mount directory # mkdir /VDO 4. Create fstab entry for VDO volume /dev/mapper/vdo1 /VDO xfs defaults,x-systemd.requires=vdo.service 0 0 5. Set the SELINUX mode to permissive in /etc/selinux/config 6. # reboot 7. After system boots # journalctl -b |egrep "Breaking|Found" Actual results: # journalctl -b|egrep "Breaking|Found" .. Mar 06 14:20:01 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job systemd-journal-catalog-update.service/start Mar 06 14:20:01 localhost.localdomain systemd[1]: Found ordering cycle on basic.target/start Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on VDO.mount/start Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on vdo.service/start Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on basic.target/start Mar 06 14:20:01 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start .. Expected results: Breaking ordering cycle is not reported Additional info: