Bug 1952229
| Summary: | After reboot, OvS interfaces become unmanaged in NetworkManager | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | David Critch <dcritch> |
| Component: | NetworkManager | Assignee: | NetworkManager Development Team <nm-team> |
| Status: | CLOSED NOTABUG | QA Contact: | Desktop QE <desktop-qa-list> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.3 | CC: | acardace, atragler, bgalvani, danw, fge, lrintel, phoracek, rkhan, sukulkar, till |
| Target Milestone: | beta | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-04-27 01:58:08 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1899057 | ||
|
Description
David Critch
2021-04-21 19:51:04 UTC
Notable timestamps: initial creation @ create @ Apr 21 19:11:52 reboot @ reboot @ Apr 21 19:15:11 and a snippet from the log after reboot: Apr 21 19:15:12 localhost NetworkManager[1965]: <trace> [1619032512.9305] ovsdb: added a bridge: brcnv, 6a343abd-06c6-4074-aacc-96b70815c37c Apr 21 19:15:12 localhost NetworkManager[1965]: <debug> [1619032512.9306] device[9cfd1f45bee63753] (brcnv): constructed (NMDeviceOvsBridge) Apr 21 19:15:12 localhost NetworkManager[1965]: <debug> [1619032512.9306] device[9cfd1f45bee63753] (brcnv): start setup of NMDeviceOvsBridge, kernel ifindex 0 Apr 21 19:15:12 localhost NetworkManager[1965]: <debug> [1619032512.9307] device[9cfd1f45bee63753] (brcnv): unmanaged: flags set to [platform-init,external-down=0x810/0x810/unmanaged/unrealized], set-unmanaged [external-down=0x800]) Apr 21 19:15:12 localhost NetworkManager[1965]: <debug> [1619032512.9307] device[9cfd1f45bee63753] (brcnv): unmanaged: flags set to [platform-init,external-down,!by-type=0x810/0x818/unmanaged/unrealized], set-managed [by-type=0x8]) Apr 21 19:15:12 localhost NetworkManager[1965]: <debug> [1619032512.9307] device[9cfd1f45bee63753] (brcnv): unmanaged: flags set to [platform-init,external-down,!by-type,!user-settings=0x810/0x858/unmanaged/unrealized], set-managed [user-settings=0x40]) Apr 21 19:15:12 localhost NetworkManager[1965]: <debug> [1619032512.9308] device[9cfd1f45bee63753] (brcnv): unmanaged: flags set to [external-down,!by-type,!platform-init,!user-settings=0x800/0x858/manageable/unrealized], set-managed [platform-init=0x10]) Apr 21 19:15:12 localhost NetworkManager[1965]: <debug> [1619032512.9308] device[9cfd1f45bee63753] (brcnv): unmanaged: flags set to [external-down,!sleeping,!by-type,!platform-init,!user-settings=0x800/0x859/manageable/unrealized], set-managed [sleeping=0x1]) Apr 21 19:15:12 localhost NetworkManager[1965]: <trace> [1619032512.9308] dbus-object[9cfd1f45bee63753]: export: "/org/freedesktop/NetworkManager/Devices/21" Apr 21 19:15:12 localhost NetworkManager[1965]: <info> [1619032512.9312] manager: (brcnv): new Open vSwitch Bridge device (/org/freedesktop/NetworkManager/Devices/21) It looks like after the reboot at 19:15:11 all connections have disappeared except this one: /etc/NetworkManager/system-connections-merged/default_connection.nmconnection David, do you know why? No idea. I just tested on a fresh node, and after running the script but before rebooting, that is still the only connection in that directory. That seems a problem in the underlying storage (which is, if I understand well, a overlay filesystem). Interesting... I went back to 4.7.0 (NM versions below, but might be irrelevant) NetworkManager-1.26.0-12.1.rhaos4.7.el8.x86_64 NetworkManager-ovs-1.26.0-12.1.rhaos4.7.el8.x86_64 NetworkManager-tui-1.26.0-12.1.rhaos4.7.el8.x86_64 NetworkManager-libnm-1.26.0-12.1.rhaos4.7.el8.x86_64 NetworkManager-team-1.26.0-12.1.rhaos4.7.el8.x86_64 All the files are there under /etc/NetworkManager/system-connections upon initial creation and after a reboot, and there is no `etc/NetworkManager/system-connections-merge` at all. I believe there was a change in RHCOS, making network profiles non persistent, so all of the day-1 configuration must be delivered through Ignition. Would it be possible to change the configuration script to run on every boot? As Ben said, that directory is mounted via overlayfs: overlay on /etc/NetworkManager/system-connections-merged type overlay (rw,relatime,seclabel,lowerdir=/etc/NetworkManager/system-connections,upperdir=/run/nm-system-connections,workdir=/run/nm-system-connections-work) After running the script, if I copy all the files from the upper directory to the lower one, it all persists after reboot, but I'm worried that isn't the right approach, or could get overwritten in an update. I can test running the script on every boot, but the issue is that even though NetworkManager forgets about them, they are still in the ovsdb. I suspect OVS will bark about it. I can modify to delete all the stuff in ovs first. I'll report back tomorrow. Okay, I tweaked the script slightly (just added a `ovs-vsctl --if-exists del-br brcnv` in there near the top) and that seems to work fine. I guess we can close this as NOTABUG since I'm figuring they had good reasons to make the change on the OpenShift/CoreOS side? Closing as previous comment. Please feel free to reopen if you think further work is required. *** Bug 1899745 has been marked as a duplicate of this bug. *** |