RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1959720 - NetworkManager does not start in Anaconda environment after switch root
Summary: NetworkManager does not start in Anaconda environment after switch root
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: hostname
Version: 8.5
Hardware: Unspecified
OS: Unspecified
urgent
unspecified
Target Milestone: beta
: ---
Assignee: Martin Osvald 🛹
QA Contact: František Hrdina
URL:
Whiteboard:
: 1959231 1960026 1973593 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-12 08:07 UTC by Radek Vykydal
Modified: 2022-08-31 12:35 UTC (History)
24 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-31 12:34:55 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
/tmp/syslog from reproducer (136.57 KB, text/plain)
2021-05-12 08:07 UTC, Radek Vykydal 		no flags Details
View All

Description Radek Vykydal 2021-05-12 08:07:51 UTC 
Created attachment 1782271 [details]
/tmp/syslog from reproducer

With latest RHEL 8.5 builds (download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8/compose/BaseOS/x86_64/os/images/boot.iso) in our kickstart tests, in about 25 / 207 cases NetworkManager fails to start after switch root.

Seems to be happening since NetworkManager-1.32.0-0.3.el8

Suspicious lines from the log:
07:37:15,652 WARNING systemd:NetworkManager.service: Found ordering cycle on dbus.socket/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on sysinit.target/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on nis-domainname.service/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on network-online.target/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on NetworkManager-wait-online.service/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on NetworkManager.service/start
07:37:15,652 ERR systemd:NetworkManager.service: Job dbus.socket/start deleted to break ordering cycle starting with NetworkManager.service/start

I am able to reproduce the issue running instrumented kickstart test locally which I need to get the syslog (from the nightly kickstart test runs we are getting only log messages since anaconda starts).
Comment 1 Thomas Haller 2021-05-12 08:16:50 UTC 
I think this is an issue in `hostname`.

See http://pkgs.devel.redhat.com/cgit/rpms/hostname/commit/?h=rhel-8.5.0&id=e2e7d59e635bc10c73a205d66a749145cca6e9a6


Reassigning.

See also bug 1956378.
Comment 2 Thomas Haller 2021-05-12 20:18:03 UTC 
*** Bug 1960026 has been marked as a duplicate of this bug. ***
Comment 3 Radek Vykydal 2021-05-20 13:13:44 UTC 
The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is there any ETA of the fix?
Comment 4 Pavel Zhukov 2021-05-20 18:19:05 UTC 
(In reply to Radek Vykydal from comment #3)
> The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is
> there any ETA of the fix?

I've dropped broken version from the erratum. If it's still in the compose it means there's something wrong with the compose
Comment 5 Pavel Zhukov 2021-05-21 04:02:30 UTC 
(In reply to Radek Vykydal from comment #3)
> The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is
> there any ETA of the fix?

Sorry. My bad. I've dropped 9.0 build instead. Should be fixed now
Comment 6 Radek Vykydal 2021-05-24 10:30:16 UTC 
Not sure why, but for RHEL 8.5 development composes hostname-3.20-7.el8 is still used:
http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/logs/x86_64/buildinstall-BaseOS-logs/lorax.log, it is also in the repository:
http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/compose/BaseOS/x86_64/os/Packages/hostname-3.20-7.el8.x86_64.rpm
Comment 7 Pavel Zhukov 2021-05-24 11:09:48 UTC 
(In reply to Radek Vykydal from comment #6)
> Not sure why, but for RHEL 8.5 development composes hostname-3.20-7.el8 is
> still used:
> http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/
> logs/x86_64/buildinstall-BaseOS-logs/lorax.log, it is also in the repository:
> http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/
> compose/BaseOS/x86_64/os/Packages/hostname-3.20-7.el8.x86_64.rpm

I have no idea what's the development compose about [1]  but it includes all packages with -candidate tag. You are testing against packages which will be never released. 

[1] http://docs.osci.redhat.com/rhel9_composing_guide/guide.html#_development_compose
Comment 10 Josef Dvoracek 2021-05-26 10:00:13 UTC 
This bug apparently landed into centos-8 stream with hostname-3.20-7.el8.x86_64.

it seems that random item in cycle can be deleted with different effects on server boot.

The worse case I observed:

* Job NetworkManager.service/start deleted to break ordering cycle - server node starts without networking:

2021-05-25 18:39:12 [   11.880024] systemd[1]: NetworkManager-wait-online.service: Found ordering cycle on NetworkManager.service/start
2021-05-25 18:39:12 [   11.890194] systemd[1]: NetworkManager-wait-online.service: Found dependency on dbus.service/start
2021-05-25 18:39:12 [   11.899149] systemd[1]: NetworkManager-wait-online.service: Found dependency on sysinit.target/start
2021-05-25 18:39:12 [   11.908283] systemd[1]: NetworkManager-wait-online.service: Found dependency on nis-domainname.service/start
2021-05-25 18:39:12 [   11.918108] systemd[1]: NetworkManager-wait-online.service: Found dependency on network-online.target/start
2021-05-25 18:39:12 [   11.927837] systemd[1]: NetworkManager-wait-online.service: Found dependency on NetworkManager-wait-online.service/start
2021-05-25 18:39:12 [   11.938698] systemd[1]: NetworkManager-wait-online.service: Job NetworkManager.service/start deleted to break ordering cycle starting with NetworkManager-wait-online.service/start
Comment 12 Eugenio Pérez Martín 2021-06-01 14:29:37 UTC 
*** Bug 1959231 has been marked as a duplicate of this bug. ***
Comment 13 Carl George 🤠 2021-06-02 18:09:51 UTC 
> This bug apparently landed into centos-8 stream with hostname-3.20-7.el8.x86_64.

Once this build was attached to errata [0], it was exported to git.centos.org [1] and built for CentOS Stream 8 [2].

Pavel, if we're withdrawing that update, please do it with positive release increase to update CentOS Stream users, not just by dropping it from errata.


[0] bug 1956378 comment 1
[1] https://git.centos.org/rpms/hostname/c/9d520349ae5995de8a36a5e9e0f98d8586183123?branch=c8s
[2] https://koji.mbox.centos.org/koji/buildinfo?buildID=17482
Comment 18 Cesar Eduardo Barros 2021-06-07 22:56:08 UTC 
The corresponding issue on bugs.centos.org seems to be https://bugs.centos.org/view.php?id=18211 ("Package: hostname - SystemD Service File Problem Causing NetworkManager to Fail on Boot").
Comment 19 Pavel Zhukov 2021-06-18 11:54:40 UTC 
*** Bug 1973593 has been marked as a duplicate of this bug. ***
Comment 23 Martin Osvald 🛹 2022-08-31 12:34:55 UTC 
Given the time that passed since anyone commented on this and the content of the latest comments (that centos 8 stream got fixed by 3.20-7.el8.0.1) from a year ago, I believe the regression introduced by bug 1956378 is fixed now, therefore closing this.

Please, feel free to re-open if you think otherwise.
Note You need to log in before you can comment on or make changes to this bug.