RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1552242 - VDO mounts in fstab can cause "Breaking order cycle" event when permissive or enforcing mode is enabled
Summary: VDO mounts in fstab can cause "Breaking order cycle" event when permissive or...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: vdo
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 7.6
Assignee: Dennis Keefe
QA Contact: Jakub Krysl
URL:
Whiteboard:
Depends On:
Blocks: 1546181
TreeView+ depends on / blocked
 
Reported: 2018-03-06 19:29 UTC by Dennis Keefe
Modified: 2019-03-06 02:36 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-10 23:00:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Dennis Keefe 2018-03-06 19:29:19 UTC
Description of problem:
Adding a mount option in fstab for VDO, such as:
/dev/mapper/vdo /VDO xfs  defaults,discards,x-systemd.requires=vdo.service 0 0

Can cause events of "Breaking ordering cycle" 
journal -b |egrep "Breaking|Found"

This is caused by systemd adding a dependency for local-fs.target 
the VDO.mount unit file.

Version-Release number of selected component (if applicable):
vdo-6.1.0.146-16.x86_64

How reproducible:

1. Create a VDO volume (block device needed)
     # vdo create --name=vdo1 --device=/dev/vda
2. Format VDO device
     # mkfs.xfs -K /dev/mapper/vdo1
3. Create mount directory
     # mkdir /VDO
4. Create fstab entry for VDO volume
     /dev/mapper/vdo1  /VDO  xfs  defaults,x-systemd.requires=vdo.service 0 0
5.  Set the SELINUX mode to permissive in /etc/selinux/config
6.  # reboot
7.  After system boots
    # journalctl -b |egrep "Breaking|Found"


Actual results:
# journalctl -b|egrep "Breaking|Found"
..
Mar 06 14:20:01 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job systemd-journal-catalog-update.service/start
Mar 06 14:20:01 localhost.localdomain systemd[1]: Found ordering cycle on basic.target/start
Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on VDO.mount/start
Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on vdo.service/start
Mar 06 14:20:01 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Mar 06 14:20:01 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start
..

Expected results:
Breaking ordering cycle is not reported

Additional info:

Comment 2 Dennis Keefe 2018-03-06 19:42:26 UTC
It has been proposed that either using a mount file unit file, such as: 
-----------------------------
[Unit]
Description = Mount file system that lives on /VDO
name = VDO.mount
Requires = vdo.service 
Conflicts = umount.target
 
[Mount]
What = /dev/mapper/vdo
Where = /VDO
Type = xfs
Options = discard
 
[Install]
WantedBy = multi-user.target
-----------------------------

instead of the fstab entry will correct this issue

or, modifying the vdo.service file to:

-----------------------------
[Unit]
Description=VDO volume services
After=systemd-remount-fs.service
DefaultDependencies=no
Conflicts=shutdown.target
Before=local-fs-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/vdo start --all --confFile /etc/vdoconf.yml
ExecStop=/usr/bin/vdo stop --all --confFile /etc/vdoconf.yml

[Install]
WantedBy=local-fs-pre.target
-----------------------------

With this change to the vdo.service file there is no longer a need for 
adding x-systemd.requires=vdo.service in the fstab file.

It is important that VDO doesn't delay or prevent the system from starting up
if there is a rebuild of the volume.  If the process of VDO rebuilding prevents other services from starting using a different "Before" target then the solution might not be favorable.   

Here is the current vdo.service configuration

# /usr/lib/systemd/system/vdo.service
[Unit]
Description=VDO volume services
After=systemd-remount-fs.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/vdo start --all --confFile /etc/vdoconf.yml
ExecStop=/usr/bin/vdo stop --all --confFile /etc/vdoconf.yml

[Install]
WantedBy=multi-user.target

-----------------------

Some testing will be required.

Comment 3 Dennis Keefe 2018-03-07 13:57:31 UTC
Note: the vdo.service unit file has not been tested yet.

Comment 5 Dennis Keefe 2018-05-29 20:56:39 UTC
Adding this change to vdo.service file makes mounting VDO volumes easier and
fixes the problem of "Breaking order cycle", but adds a "cost" of delaying the
boot cycle if the VDO volume has not been cleanly shutdown.

VDO volumes perform a recovery after unclean shutdowns.  If the vdo.service file
is changed to use "local-fs-pre.target", then the system will wait until VDO has
completed it's recovery before proceeding.   The time it takes to finish a
recovery can depend on the VDO volume size, speed of the CPU and media.

Simulating this could be done by creating a VDO volume on a USB thumb drive
(slow media), writing data to that device, then crashing the system.  After
the system has been rebooted, run systemd-analyze blame to see how long VDO
took to start the volume(s).  In my test it took about 90 seconds. During the recovery process the system can't be accessed, since systemd hasn't reached the
point of starting SSH or other services to manage the system.

Using the mount unit file instead of changing the vdo.service fill will mount
vdo volumes after the system has booted and doesn't cause the "Breaking order cycle".

Pro/Con
1. vdo.service
   - would make mounting VDO volumes easier for many users
   - But VDO recoveries can extend the boot cycle
2. mount unit
   - adds additional steps, which are not intuitive to everyone
   - VDO does not get in the way of the boot up process or system access

Comment 6 Dennis Keefe 2018-06-08 19:25:32 UTC
Adding the proposed vdo.service file can delay the boot cycle when VDO volumes
need to recover, therefore we should not implement this feature.  
Users should create mount unit files for VDO volumes unless there is an alternate 
method of initializing vdo volume that doesn't delay the boot or break ordering.

Comment 7 Marius Vollmer 2018-06-15 11:39:19 UTC
What about adding both x-systemd.requires=vdo.service and _netdev to /etc/fstab?

That should also get rid of the cycle and doesn't require people (and Cockpit) to switch from /etc/fstab to something new.

Comment 8 Andy Walsh 2018-06-21 12:42:26 UTC
_netdev may allow for this to work, but we're not really a network device, so it doesn't fit the definition exactly.  I think it would be incorrect for that reason.

Maybe we can add a documentation note in the  Storage Admin guide to specify that option noting that we're not a network device, but it achieves the goal for SELinux compliance?

Comment 9 Dennis Keefe 2018-06-21 20:43:24 UTC
If someone had iscsi storage and created a VDO volume on top, then a filesystem, then NFS, would using _netdev have problems figuring out which network device / service to start first (nfs, vdo, iscsi)?

Comment 10 Dennis Keefe 2018-07-10 22:59:39 UTC
This issue is resolved by creating a mount unit file

 https://bugzilla.redhat.com/show_bug.cgi?id=1552242

While using the "Before" option mentioned above can also resolve this issue it can
have the side effect of extending the boot process.

Closing ticket


Note You need to log in before you can comment on or make changes to this bug.