Bug 1471922
| Summary: | Breaking ordering cycle by deleting job when iSCSi used | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | bogdan.sh <bshyshka> | ||||||
| Component: | systemd | Assignee: | systemd-maint | ||||||
| Status: | CLOSED WONTFIX | QA Contact: | qe-baseos-daemons | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 7.3 | CC: | je.fomin, johannespfau, patdung100+redhat, pavel.z, redhat, systemd-maint-list | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2021-01-15 07:39:59 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Hello, We have the same problem with mount iSCSI disks during system boot. This issue does not allow us to upgrade our CentOS servers from 6 to 7 release. Hi RedHat team, Do you have progress on solution for this this bug? Created attachment 1473242 [details]
Having the problem with RHEL 7.5. Attached file is the related log
Having the problem with ISCSI & systemd to RHEL 7.5
................
[ SKIP ] Ordering cycle found, skipping Network
[ SKIP ] Ordering cycle found, skipping Network is Online
[ SKIP ] Ordering cycle found, skipping Remote File Systems (Pre)
[ SKIP ] Ordering cycle found, skipping File System Quota Check
[ SKIP ] Ordering cycle found, skipping Remote File Systems (Pre)
[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-41075ff7\x2d35f0\x2d4374\x2d9b92\x2d879ef2dde46f.device.
[DEPEND] Dependency failed for /home.
[DEPEND] Dependency failed for Remote File Systems.
[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-d04f649f\x2d558c\x2d4d7e\x2dbfc2\x2d836e4621fec9.device.
[DEPEND] Dependency failed for /installatron-backups.
[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-0a25dbd4\x2da28b\x2d4e0c\x2d9e81\x2dc129b735328a.device.
[DEPEND] Dependency failed for /var/lib/mysql.
[ INFO ] Network Manager is not active.
[DEPEND] Dependency failed for Network Manager Wait Online.
Starting Dynamic System Tuning Daemon...
...
Issue with initializing ISCSI
It on try mounted devices before start network service.
After starting network service does not mount some ISCSI devices.
If OS reboots ten time, problem show for the seventh time or show for the three time or five time , randomly. May boot ten time, and be All is ok!
We also had the same issue in a recent Centos 7. Like the original poster, we had `local-fs.target` in the loop, which got me thinking. In our case, adding _netdev to /etc/fstab for the relevant devices fixed this.
Interestingly, the systemd generator seems to partially detect that this is a network filesystem, even without _netdev: Quoting the systemd.mount manpage for _netdev:
> Network mount units are ordered between remote-fs-pre.target and remote-fs.target, instead of local-fs-pre.target and local-fs.target. They also pull in network-online.target and are ordered after it and network.target.
It seems only the second part happens, but part 1, ordering between remote-fs-pre.target and remote-fs.target, must be forced by specifying _netdev.
A duplicate Centos bug report is at https://bugs.centos.org/view.php?id=13561 I don't have an account to comment there though. After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. |
Created attachment 1299950 [details] Two reboots with different failed jobs results. Description of problem: System is not able to process all boot jobs when iSCSi mount is used. Reproduced on different kernel versions. The issue seems to be related to systemd implementation. After server reboot (full output): # journalctl |egrep -i "cycle|skip|depend" Jul 17 09:34:02 localhost.localdomain kernel: kvm-clock: using sched offset of 704511401346 cycles Jul 17 09:34:02 localhost.localdomain kernel: Calibrating delay loop (skipped) preset value.. 4399.99 BogoMIPS (lpj=2199998) Jul 17 13:34:03 localhost.localdomain systemd[1]: Found ordering cycle on basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sockets.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on iscsid.socket/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network-online.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job sockets.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Job sockets.target/start deleted to break ordering cycle starting with basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found ordering cycle on paths.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on brandbot.path/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network-online.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on paths.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job brandbot.path/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Job brandbot.path/start deleted to break ordering cycle starting with paths.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found ordering cycle on basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network-online.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job systemd-tmpfiles-setup.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found ordering cycle on basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network-online.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on network.service/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Found dependency on basic.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start Jul 17 13:34:03 localhost.localdomain systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with basic.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found ordering cycle on sysinit.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network-online.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on basic.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job systemd-tmpfiles-setup.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found ordering cycle on sysinit.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network-online.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on basic.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found ordering cycle on var-lib-mysql.mount/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network-online.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on basic.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-tmpfiles-setup.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job systemd-tmpfiles-setup.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with var-lib-mysql.mount/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found ordering cycle on var-lib-mysql.mount/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network-online.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on network.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on basic.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on sysinit.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on local-fs.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on quotaon.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on systemd-quotacheck.service/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Found dependency on var-lib-mysql.mount/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start Jul 17 13:34:04 localhost.localdomain systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with var-lib-mysql.mount/start Jul 17 13:34:04 localhost.localdomain acpid[405]: skipping incomplete file /etc/acpi/events/videoconf Version-Release number of selected component (if applicable): 3.10.0-514.26.2.el7.x86_64 # rpm -qa | egrep "systemd|iscsi" | sort iscsi-initiator-utils-6.2.0.873-35.el7.x86_64 iscsi-initiator-utils-iscsiuio-6.2.0.873-35.el7.x86_64 systemd-219-30.el7_3.9.x86_64 systemd-libs-219-30.el7_3.9.x86_64 systemd-sysv-219-30.el7_3.9.x86_64 Iscsi disk is mounted in /etc/fstab as: UUID="d1b32b3a-6b2c-4773-85a1-d4ebe23c1700" /var/lib/mysql ext3 _netdev,usrquota,noatime,x-systemd.wants=network-online.target 0 0 # systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After var-lib-mysql.mount Requires=-.mount Requisite= Wants=system.slice quotaon.service network-online.target systemd-quotacheck.service BindsTo=dev-disk-by\x5cx2duuid-d1b32b3a\x5cx2d6b2c\x5cx2d4773\x5cx2d85a1\x5cx2dd4ebe23c1700.device PartOf= Before=quotaon.service umount.target remote-fs.target systemd-quotacheck.service After=remote-fs-pre.target network.target -.mount systemd-journald.socket system.slice dev-disk-by\x5cx2duuid-d1b32b3a\x5cx2d6b2c\x5cx2d4773\x5cx2d85a1\x5cx2dd4ebe23c1700.device network-online.target The issue is always happening, no matter of options x-systemd in /etc/fstab . As well DefaultDependencies=no makes nothing. How reproducible: 100% reproducible all the time with different kernels. Steps to Reproduce: 1. Install and update CentOS to latest 2. create iscsi target for examle with openfiler 3. configure CentOS to mount iscsi on boot over fstab 4. reboot 5. check with: journalctl |egrep -i "cycle|skip|depend" Actual results: A lot of 'deleting job' in the output which means services were not started properly. Expected results: Clear outout after reboot and journalctl command Additional info: Usually failed jobs are different from different reboots (check screenshot).