Bug 1471922 - Breaking ordering cycle by deleting job when iSCSi used
Breaking ordering cycle by deleting job when iSCSi used
Status: NEW
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd (Show other bugs)
7.3
x86_64 Linux
unspecified Severity unspecified
: rc
: ---
Assigned To: systemd-maint
qe-baseos-daemons
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-17 12:38 EDT by bogdan.sh
Modified: 2017-09-01 04:53 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Two reboots with different failed jobs results. (394.72 KB, image/png)
2017-07-17 12:38 EDT, bogdan.sh
no flags Details

  None (edit)
Description bogdan.sh 2017-07-17 12:38:56 EDT
Created attachment 1299950 [details]
Two reboots with different failed jobs results.

Description of problem:
System is not able to process all boot jobs when iSCSi mount is used. Reproduced on different kernel versions. The issue seems to be related to systemd implementation.

After server reboot (full output):
# journalctl |egrep -i "cycle|skip|depend"
Jul 17 09:34:02 localhost.localdomain kernel: kvm-clock: using sched offset of 704511401346 cycles
Jul 17 09:34:02 localhost.localdomain kernel: Calibrating delay loop (skipped) preset value.. 4399.99 BogoMIPS (lpj=2199998)
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found ordering cycle on basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sockets.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on iscsid.socket/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network-online.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job sockets.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Job sockets.target/start deleted to break ordering cycle starting with basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found ordering cycle on paths.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on brandbot.path/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network-online.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on paths.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job brandbot.path/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Job brandbot.path/start deleted to break ordering cycle starting with paths.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found ordering cycle on basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network-online.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job systemd-tmpfiles-setup.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found ordering cycle on basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network-online.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network.service/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start
Jul 17 13:34:03 localhost.localdomain systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with basic.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found ordering cycle on sysinit.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network-online.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job systemd-tmpfiles-setup.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found ordering cycle on sysinit.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network-online.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found ordering cycle on var-lib-mysql.mount/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network-online.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job systemd-tmpfiles-setup.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with var-lib-mysql.mount/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found ordering cycle on var-lib-mysql.mount/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network-online.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on basic.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start
Jul 17 13:34:04 localhost.localdomain systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with var-lib-mysql.mount/start
Jul 17 13:34:04 localhost.localdomain acpid[405]: skipping incomplete file /etc/acpi/events/videoconf


Version-Release number of selected component (if applicable):
3.10.0-514.26.2.el7.x86_64

# rpm -qa | egrep "systemd|iscsi" | sort
iscsi-initiator-utils-6.2.0.873-35.el7.x86_64
iscsi-initiator-utils-iscsiuio-6.2.0.873-35.el7.x86_64
systemd-219-30.el7_3.9.x86_64
systemd-libs-219-30.el7_3.9.x86_64
systemd-sysv-219-30.el7_3.9.x86_64


Iscsi disk is mounted in /etc/fstab as:
UUID="d1b32b3a-6b2c-4773-85a1-d4ebe23c1700" /var/lib/mysql ext3 _netdev,usrquota,noatime,x-systemd.wants=network-online.target 0 0

# systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After var-lib-mysql.mount
Requires=-.mount
Requisite=
Wants=system.slice quotaon.service network-online.target systemd-quotacheck.service
BindsTo=dev-disk-by\x5cx2duuid-d1b32b3a\x5cx2d6b2c\x5cx2d4773\x5cx2d85a1\x5cx2dd4ebe23c1700.device
PartOf=
Before=quotaon.service umount.target remote-fs.target systemd-quotacheck.service
After=remote-fs-pre.target network.target -.mount systemd-journald.socket system.slice dev-disk-by\x5cx2duuid-d1b32b3a\x5cx2d6b2c\x5cx2d4773\x5cx2d85a1\x5cx2dd4ebe23c1700.device network-online.target


The issue is always happening, no matter of options x-systemd in /etc/fstab . As well DefaultDependencies=no makes nothing. 


How reproducible:
100% reproducible all the time with different kernels.

Steps to Reproduce:
1. Install and update CentOS to latest
2. create iscsi target for examle with openfiler
3. configure CentOS to mount iscsi on boot over fstab
4. reboot
5. check with:
 journalctl |egrep -i "cycle|skip|depend"


Actual results:
A lot of 'deleting job' in the output which means services were not started properly.

Expected results:
Clear outout after reboot and journalctl command

Additional info:
Usually failed jobs are different from different reboots (check screenshot).
Comment 2 pavel.z 2017-08-07 05:13:17 EDT
Hello,

We have the same problem with mount iSCSI disks during system boot.
This issue does not allow us to upgrade our CentOS servers from 6 to 7 release.
Comment 3 pavel.z 2017-09-01 04:53:47 EDT
Hi RedHat team,

Do you have progress on solution for this this bug?

Note You need to log in before you can comment on or make changes to this bug.