Bug 1875520 - Install of Ovirt hosted engine fails to create ovirtmgmt bridge on vlan interface
Summary: Install of Ovirt hosted engine fails to create ovirtmgmt bridge on vlan inter...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.40.22
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Ales Musil
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On: 1885857
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-03 16:49 UTC by Sverker Abrahamsson
Modified: 2023-09-15 00:47 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-20 08:05:27 UTC
oVirt Team: Network
Embargoed:


Attachments (Terms of Use)
vdsm.log (327.18 KB, text/plain)
2020-09-03 17:11 UTC, Sverker Abrahamsson
no flags Details
supervdsm.log (99.90 KB, text/plain)
2020-09-03 17:12 UTC, Sverker Abrahamsson
no flags Details
Output from journalctl (3.00 MB, text/plain)
2020-09-04 15:04 UTC, Sverker Abrahamsson
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github nmstate nmstate issues 1339 0 None open Merge running routes to config for desired state when found interface is static address. 2021-01-11 09:01:50 UTC

Description Sverker Abrahamsson 2020-09-03 16:49:05 UTC
Description of problem:
When attempting to install Ovirt hosted engine on CentOS 8.2 the setup fails when creating ovirtmgmt bridge with the following error message:

[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors:   code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host.,    code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com.,    code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'},   fix accordingly and re-deploy."}

The network setup is a physical interface enp4s0 which receives external traffic untagged. On the same port there is a vlan interface, which should be used as the management network and hence on which the ovirtmgmt bridge should be created.

[root@h1-mgmt ~]# nmcli con show
NAME         UUID                                  TYPE      DEVICE
enp4s0       af7ccb53-011b-4c36-998a-1878b4ae7100  ethernet  enp4s0
enp4s0.4000  ecc8064d-18c1-99b7-3fe4-9c5a593ece6f  vlan      enp4s0.4000
[root@h1-mgmt ~]# nmstatectl show
---
dns-resolver:
  config:
    search: []
    server:
    - 213.133.98.98
  running:
    search: []
    server:
    - 213.133.98.98
route-rules:
  config: []
routes:
  config:
  - destination: 0.0.0.0/0
    metric: -1
    next-hop-address: 144.76.84.65
    next-hop-interface: enp4s0
    table-id: 0
  - destination: ::/0
    metric: -1
    next-hop-address: fe80::1
    next-hop-interface: enp4s0
    table-id: 0
  running:
  - destination: 0.0.0.0/0
    metric: 100
    next-hop-address: 144.76.84.65
    next-hop-interface: enp4s0
    table-id: 254
  - destination: 144.76.84.65/32
    metric: 100
    next-hop-address: ''
    next-hop-interface: enp4s0
    table-id: 254
  - destination: 172.27.1.0/24
    metric: 400
    next-hop-address: ''
    next-hop-interface: enp4s0.4000
    table-id: 254
  - destination: 2a01:4f8:192:1148::/64
    metric: 100
    next-hop-address: ''
    next-hop-interface: enp4s0
    table-id: 254
  - destination: ::/0
    metric: 100
    next-hop-address: fe80::1
    next-hop-interface: enp4s0
    table-id: 254
  - destination: fe80::/64
    metric: 100
    next-hop-address: ''
    next-hop-interface: enp4s0
    table-id: 254
  - destination: ff00::/8
    metric: 256
    next-hop-address: ''
    next-hop-interface: enp4s0
    table-id: 255
interfaces:
- name: ;vdsmdummy;
  type: linux-bridge
  state: down
  ipv4:
    enabled: false
  ipv6:
    enabled: false
  mac-address: B2:9E:E0:61:71:88
  mtu: 1500
- name: br-int
  type: unknown
  state: down
  ipv4:
    enabled: false
  ipv6:
    enabled: false
  mac-address: 6E:37:94:63:E0:4B
  mtu: 1500
- name: enp4s0
  type: ethernet
  state: up
  ethernet:
    auto-negotiation: true
    duplex: full
    speed: 1000
  ipv4:
    address:
    - ip: 144.76.84.73
      prefix-length: 32
    dhcp: false
    enabled: true
  ipv6:
    address:
    - ip: 2a01:4f8:192:1148::2
      prefix-length: 64
    - ip: fe80::62a4:4cff:fee9:4ac
      prefix-length: 64
    auto-dns: true
    auto-gateway: true
    auto-routes: true
    autoconf: true
    dhcp: true
    enabled: true
  mac-address: 60:A4:4C:E9:04:AC
  mtu: 1500
- name: enp4s0.4000
  type: vlan
  state: up
  ipv4:
    address:
    - ip: 172.27.1.1
      prefix-length: 24
    dhcp: false
    enabled: true
  ipv6:
    autoconf: false
    dhcp: false
    enabled: false
  mac-address: 60:A4:4C:E9:04:AC
  mtu: 1500
  vlan:
    base-iface: enp4s0
    id: 4000
- name: lo
  type: unknown
  state: down
  ipv4:
    enabled: false
  ipv6:
    enabled: false
  mtu: 65536
- name: ovs-system
  type: unknown
  state: down
  ipv4:
    enabled: false
  ipv6:
    enabled: false
  mac-address: A2:35:7A:6C:B7:EF
  mtu: 1500

It appears vdsm attempts to create a network setup where enp4s0 does not have any address, but that fails because it has a ipv6 route. This is the output in vdsm log:

MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False,
'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh
eck': 'true'}) {}
MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal
se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess':
True, 'connectivityCheck': 'true'}
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0
MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True
, 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']})
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None)
MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b''
MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate
MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'}
, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [
{'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config'
: {'server': ['213.133.98.98']}}}
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt']
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0']
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async
MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState>
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su'))
MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks
    _setup_networks(networks, bondings, options, net_info)
  File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks
    networks, bondings, options, net_info, in_rollback
  File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup
    _setup_nmstate(networks, bondings, options, in_rollback)
  File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate
    nmstate.setup(desired_state, verify_change=not in_rollback)
  File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup
    state_apply(desired_state, verify_change=verify_change)
  File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped
    ret = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply
    state.State(desired_state), verify_change, commit, rollback_timeout
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state
    con_profiles=ifaces_add_configs + ifaces_edit_configs,
  File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers
    mainloop.run(timeout=MAINLOOP_TIMEOUT)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run
    f"Unexpected failure of libnm when running the mainloop: {err}"
libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution

Version-Release number of selected component (if applicable):
[root@h1-mgmt vdsm]# rpm -q vdsm
vdsm-4.40.22-1.el8.x86_64

How reproducible:
Create a physical network interface with a vlan interface, both having ip addresses. Select the vlan interface as bridge interface. 

Steps to Reproduce:
1.
2.
3.

Actual results:
If the physical interface have an ipv6 route it will fail as above, if it only have ipv4 address then probably enp4s0 will be without ip address.

Expected results:
enp4s0 will keep it's addresses and vlan interface enp4s0.4000 will be bridged with ovirtmgmt.

Additional info:

Comment 1 Sverker Abrahamsson 2020-09-03 17:11:31 UTC
Created attachment 1713654 [details]
vdsm.log

Comment 2 Sverker Abrahamsson 2020-09-03 17:12:00 UTC
Created attachment 1713655 [details]
supervdsm.log

Comment 3 Dominik Holler 2020-09-04 08:03:35 UTC
Can you please attache the relevant log messages from journalctl?
I still wonder why or who touches the non-vlanned interface (enp4s0).


Is there a change in behavior, if IPv6 on the non-vlanned interface (enp4s0) is configured in a static way?

Comment 4 Sverker Abrahamsson 2020-09-04 15:04:43 UTC
Created attachment 1713793 [details]
Output from journalctl

Comment 5 RHEL Program Management 2020-12-01 14:05:18 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 6 Martin Perina 2021-10-06 07:31:04 UTC
Is this still reproduable on oVirt 4.4.8 with latest CentOS Stream?

Comment 7 Ales Musil 2021-10-20 08:05:27 UTC
If this appears again, please reopen this with more info.

Comment 8 Red Hat Bugzilla 2023-09-15 00:47:27 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.