Bug 1959720
| Summary: | NetworkManager does not start in Anaconda environment after switch root | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Radek Vykydal <rvykydal> | ||||
| Component: | hostname | Assignee: | Martin Osvald 🛹 <mosvald> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | František Hrdina <fhrdina> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 8.5 | CC: | acardace, bgalvani, carl, cesarb, efuller, hhei, jbastian, jose, jstodola, leiyang, lrintel, michele, pezhang, rduda, rkhan, sukulkar, thaller, till, xuli, yacao, yanghliu, yfu, yidliu, yuxisun | ||||
| Target Milestone: | beta | Keywords: | Regression | ||||
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-08-31 12:34:55 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
I think this is an issue in `hostname`. See http://pkgs.devel.redhat.com/cgit/rpms/hostname/commit/?h=rhel-8.5.0&id=e2e7d59e635bc10c73a205d66a749145cca6e9a6 Reassigning. See also bug 1956378. *** Bug 1960026 has been marked as a duplicate of this bug. *** The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is there any ETA of the fix? (In reply to Radek Vykydal from comment #3) > The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is > there any ETA of the fix? I've dropped broken version from the erratum. If it's still in the compose it means there's something wrong with the compose (In reply to Radek Vykydal from comment #3) > The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is > there any ETA of the fix? Sorry. My bad. I've dropped 9.0 build instead. Should be fixed now Not sure why, but for RHEL 8.5 development composes hostname-3.20-7.el8 is still used: http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/logs/x86_64/buildinstall-BaseOS-logs/lorax.log, it is also in the repository: http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/compose/BaseOS/x86_64/os/Packages/hostname-3.20-7.el8.x86_64.rpm (In reply to Radek Vykydal from comment #6) > Not sure why, but for RHEL 8.5 development composes hostname-3.20-7.el8 is > still used: > http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/ > logs/x86_64/buildinstall-BaseOS-logs/lorax.log, it is also in the repository: > http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/ > compose/BaseOS/x86_64/os/Packages/hostname-3.20-7.el8.x86_64.rpm I have no idea what's the development compose about [1] but it includes all packages with -candidate tag. You are testing against packages which will be never released. [1] http://docs.osci.redhat.com/rhel9_composing_guide/guide.html#_development_compose This bug apparently landed into centos-8 stream with hostname-3.20-7.el8.x86_64. it seems that random item in cycle can be deleted with different effects on server boot. The worse case I observed: * Job NetworkManager.service/start deleted to break ordering cycle - server node starts without networking: 2021-05-25 18:39:12 [ 11.880024] systemd[1]: NetworkManager-wait-online.service: Found ordering cycle on NetworkManager.service/start 2021-05-25 18:39:12 [ 11.890194] systemd[1]: NetworkManager-wait-online.service: Found dependency on dbus.service/start 2021-05-25 18:39:12 [ 11.899149] systemd[1]: NetworkManager-wait-online.service: Found dependency on sysinit.target/start 2021-05-25 18:39:12 [ 11.908283] systemd[1]: NetworkManager-wait-online.service: Found dependency on nis-domainname.service/start 2021-05-25 18:39:12 [ 11.918108] systemd[1]: NetworkManager-wait-online.service: Found dependency on network-online.target/start 2021-05-25 18:39:12 [ 11.927837] systemd[1]: NetworkManager-wait-online.service: Found dependency on NetworkManager-wait-online.service/start 2021-05-25 18:39:12 [ 11.938698] systemd[1]: NetworkManager-wait-online.service: Job NetworkManager.service/start deleted to break ordering cycle starting with NetworkManager-wait-online.service/start *** Bug 1959231 has been marked as a duplicate of this bug. *** > This bug apparently landed into centos-8 stream with hostname-3.20-7.el8.x86_64. Once this build was attached to errata [0], it was exported to git.centos.org [1] and built for CentOS Stream 8 [2]. Pavel, if we're withdrawing that update, please do it with positive release increase to update CentOS Stream users, not just by dropping it from errata. [0] bug 1956378 comment 1 [1] https://git.centos.org/rpms/hostname/c/9d520349ae5995de8a36a5e9e0f98d8586183123?branch=c8s [2] https://koji.mbox.centos.org/koji/buildinfo?buildID=17482 The corresponding issue on bugs.centos.org seems to be https://bugs.centos.org/view.php?id=18211 ("Package: hostname - SystemD Service File Problem Causing NetworkManager to Fail on Boot"). *** Bug 1973593 has been marked as a duplicate of this bug. *** Given the time that passed since anyone commented on this and the content of the latest comments (that centos 8 stream got fixed by 3.20-7.el8.0.1) from a year ago, I believe the regression introduced by bug 1956378 is fixed now, therefore closing this. Please, feel free to re-open if you think otherwise. |
Created attachment 1782271 [details] /tmp/syslog from reproducer With latest RHEL 8.5 builds (download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8/compose/BaseOS/x86_64/os/images/boot.iso) in our kickstart tests, in about 25 / 207 cases NetworkManager fails to start after switch root. Seems to be happening since NetworkManager-1.32.0-0.3.el8 Suspicious lines from the log: 07:37:15,652 WARNING systemd:NetworkManager.service: Found ordering cycle on dbus.socket/start 07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on sysinit.target/start 07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on nis-domainname.service/start 07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on network-online.target/start 07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on NetworkManager-wait-online.service/start 07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on NetworkManager.service/start 07:37:15,652 ERR systemd:NetworkManager.service: Job dbus.socket/start deleted to break ordering cycle starting with NetworkManager.service/start I am able to reproduce the issue running instrumented kickstart test locally which I need to get the syslog (from the nightly kickstart test runs we are getting only log messages since anaconda starts).