Bug 1875520
| Summary: | Install of Ovirt hosted engine fails to create ovirtmgmt bridge on vlan interface | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [oVirt] vdsm | Reporter: | Sverker Abrahamsson <sverker> | ||||||||
| Component: | Core | Assignee: | Ales Musil <amusil> | ||||||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Michael Burman <mburman> | ||||||||
| Severity: | unspecified | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 4.40.22 | CC: | bugs, dholler, mperina | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2021-10-20 08:05:27 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | Network | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | 1885857 | ||||||||||
| Bug Blocks: | |||||||||||
| Attachments: |
|
||||||||||
Created attachment 1713654 [details]
vdsm.log
Created attachment 1713655 [details]
supervdsm.log
Can you please attache the relevant log messages from journalctl? I still wonder why or who touches the non-vlanned interface (enp4s0). Is there a change in behavior, if IPv6 on the non-vlanned interface (enp4s0) is configured in a static way? Created attachment 1713793 [details]
Output from journalctl
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again. Is this still reproduable on oVirt 4.4.8 with latest CentOS Stream? If this appears again, please reopen this with more info. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |
Description of problem: When attempting to install Ovirt hosted engine on CentOS 8.2 the setup fails when creating ovirtmgmt bridge with the following error message: [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."} The network setup is a physical interface enp4s0 which receives external traffic untagged. On the same port there is a vlan interface, which should be used as the management network and hence on which the ovirtmgmt bridge should be created. [root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500 It appears vdsm attempts to create a network setup where enp4s0 does not have any address, but that fails because it has a ipv6 route. This is the output in vdsm log: MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution Version-Release number of selected component (if applicable): [root@h1-mgmt vdsm]# rpm -q vdsm vdsm-4.40.22-1.el8.x86_64 How reproducible: Create a physical network interface with a vlan interface, both having ip addresses. Select the vlan interface as bridge interface. Steps to Reproduce: 1. 2. 3. Actual results: If the physical interface have an ipv6 route it will fail as above, if it only have ipv4 address then probably enp4s0 will be without ip address. Expected results: enp4s0 will keep it's addresses and vlan interface enp4s0.4000 will be bridged with ovirtmgmt. Additional info: