Bug 1959720 - NetworkManager does not start in Anaconda environment after switch root
Summary: NetworkManager does not start in Anaconda environment after switch root
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: hostname
Version: 8.5
Hardware: Unspecified
OS: Unspecified
urgent
unspecified
Target Milestone: beta
: ---
Assignee: Pavel Zhukov
QA Contact: František Hrdina
URL:
Whiteboard:
: 1959231 1960026 1973593 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-12 08:07 UTC by Radek Vykydal
Modified: 2021-06-21 08:24 UTC (History)
27 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
/tmp/syslog from reproducer (136.57 KB, text/plain)
2021-05-12 08:07 UTC, Radek Vykydal
no flags Details

Description Radek Vykydal 2021-05-12 08:07:51 UTC
Created attachment 1782271 [details]
/tmp/syslog from reproducer

With latest RHEL 8.5 builds (download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8/compose/BaseOS/x86_64/os/images/boot.iso) in our kickstart tests, in about 25 / 207 cases NetworkManager fails to start after switch root.

Seems to be happening since NetworkManager-1.32.0-0.3.el8

Suspicious lines from the log:
07:37:15,652 WARNING systemd:NetworkManager.service: Found ordering cycle on dbus.socket/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on sysinit.target/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on nis-domainname.service/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on network-online.target/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on NetworkManager-wait-online.service/start
07:37:15,652 WARNING systemd:NetworkManager.service: Found dependency on NetworkManager.service/start
07:37:15,652 ERR systemd:NetworkManager.service: Job dbus.socket/start deleted to break ordering cycle starting with NetworkManager.service/start

I am able to reproduce the issue running instrumented kickstart test locally which I need to get the syslog (from the nightly kickstart test runs we are getting only log messages since anaconda starts).

Comment 1 Thomas Haller 2021-05-12 08:16:50 UTC
I think this is an issue in `hostname`.

See http://pkgs.devel.redhat.com/cgit/rpms/hostname/commit/?h=rhel-8.5.0&id=e2e7d59e635bc10c73a205d66a749145cca6e9a6


Reassigning.

See also bug 1956378.

Comment 2 Thomas Haller 2021-05-12 20:18:03 UTC
*** Bug 1960026 has been marked as a duplicate of this bug. ***

Comment 3 Radek Vykydal 2021-05-20 13:13:44 UTC
The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is there any ETA of the fix?

Comment 4 Pavel Zhukov 2021-05-20 18:19:05 UTC
(In reply to Radek Vykydal from comment #3)
> The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is
> there any ETA of the fix?

I've dropped broken version from the erratum. If it's still in the compose it means there's something wrong with the compose

Comment 5 Pavel Zhukov 2021-05-21 04:02:30 UTC
(In reply to Radek Vykydal from comment #3)
> The issue is still breaking our nightly Anaconda kickstart tests. Pavel, is
> there any ETA of the fix?

Sorry. My bad. I've dropped 9.0 build instead. Should be fixed now

Comment 7 Pavel Zhukov 2021-05-24 11:09:48 UTC
(In reply to Radek Vykydal from comment #6)
> Not sure why, but for RHEL 8.5 development composes hostname-3.20-7.el8 is
> still used:
> http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/
> logs/x86_64/buildinstall-BaseOS-logs/lorax.log, it is also in the repository:
> http://download.eng.bos.redhat.com/rhel-8/development/RHEL-8/latest-RHEL-8.5/
> compose/BaseOS/x86_64/os/Packages/hostname-3.20-7.el8.x86_64.rpm

I have no idea what's the development compose about [1]  but it includes all packages with -candidate tag. You are testing against packages which will be never released. 

[1] http://docs.osci.redhat.com/rhel9_composing_guide/guide.html#_development_compose

Comment 10 Josef Dvoracek 2021-05-26 10:00:13 UTC
This bug apparently landed into centos-8 stream with hostname-3.20-7.el8.x86_64.

it seems that random item in cycle can be deleted with different effects on server boot.

The worse case I observed:

* Job NetworkManager.service/start deleted to break ordering cycle - server node starts without networking:

2021-05-25 18:39:12 [   11.880024] systemd[1]: NetworkManager-wait-online.service: Found ordering cycle on NetworkManager.service/start
2021-05-25 18:39:12 [   11.890194] systemd[1]: NetworkManager-wait-online.service: Found dependency on dbus.service/start
2021-05-25 18:39:12 [   11.899149] systemd[1]: NetworkManager-wait-online.service: Found dependency on sysinit.target/start
2021-05-25 18:39:12 [   11.908283] systemd[1]: NetworkManager-wait-online.service: Found dependency on nis-domainname.service/start
2021-05-25 18:39:12 [   11.918108] systemd[1]: NetworkManager-wait-online.service: Found dependency on network-online.target/start
2021-05-25 18:39:12 [   11.927837] systemd[1]: NetworkManager-wait-online.service: Found dependency on NetworkManager-wait-online.service/start
2021-05-25 18:39:12 [   11.938698] systemd[1]: NetworkManager-wait-online.service: Job NetworkManager.service/start deleted to break ordering cycle starting with NetworkManager-wait-online.service/start

Comment 12 Eugenio Pérez Martín 2021-06-01 14:29:37 UTC
*** Bug 1959231 has been marked as a duplicate of this bug. ***

Comment 13 Carl George 🤠 2021-06-02 18:09:51 UTC
> This bug apparently landed into centos-8 stream with hostname-3.20-7.el8.x86_64.

Once this build was attached to errata [0], it was exported to git.centos.org [1] and built for CentOS Stream 8 [2].

Pavel, if we're withdrawing that update, please do it with positive release increase to update CentOS Stream users, not just by dropping it from errata.


[0] bug 1956378 comment 1
[1] https://git.centos.org/rpms/hostname/c/9d520349ae5995de8a36a5e9e0f98d8586183123?branch=c8s
[2] https://koji.mbox.centos.org/koji/buildinfo?buildID=17482

Comment 18 Cesar Eduardo Barros 2021-06-07 22:56:08 UTC
The corresponding issue on bugs.centos.org seems to be https://bugs.centos.org/view.php?id=18211 ("Package: hostname - SystemD Service File Problem Causing NetworkManager to Fail on Boot").

Comment 19 Pavel Zhukov 2021-06-18 11:54:40 UTC
*** Bug 1973593 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.