Bug 1964342

Summary: Using nmstate inside a pod prevents any OVS operation
Product: Red Hat Enterprise Linux 8 Reporter: Quique Llorente <ellorent>
Component: nmstateAssignee: Gris Ge <fge>
Status: CLOSED ERRATA QA Contact: Mingyu Shi <mshi>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.5CC: ferferna, jiji, jishi, network-qe, till
Target Milestone: betaKeywords: Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: nmstate-1.1.0-0.1.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1964439 (view as bug list) Environment:
Last Closed: 2021-11-09 17:43:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1964439    

Description Quique Llorente 2021-05-25 09:20:45 UTC 
Description of problem:
When nmstate is used inside a pod delegating the OVS operation to the node trying to apply some desired state with OVS involved fail since nmstate try to run "systemctl status ovs" but systemctl is not working at those containers.


Version-Release number of selected component (if applicable):


How reproducible:

Steps to Reproduce:
1. git clone https://github.com/nmstate/nmstate
2. git reset --hard d67ba7f6b7c3036ceeca62545c95a1b10bdea513~1 # Before the fix
1. ./automation/run-tests.sh --k8s --centos-stream --test-type=integ_tier1 --pytest-args="-k ovs"


Actual results:
=================================== FAILURES ===================================
___________ test_create_and_remove_ovs_bridge_with_min_desired_state ___________

    @pytest.mark.tier1
    def test_create_and_remove_ovs_bridge_with_min_desired_state():
>       with Bridge(BRIDGE1).create() as state:

tests/integration/ovs_test.py:96: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.6/contextlib.py:81: in __enter__
    return next(self.gen)
tests/integration/testlib/ovslib.py:126: in create
    libnmstate.apply(desired_state)
/usr/lib/python3.6/site-packages/libnmstate/netapplier.py:74: in apply
    desired_state, plugins_capabilities(plugins)
/usr/lib/python3.6/site-packages/libnmstate/validator.py:46: in validate_capabilities
    validate_interface_capabilities(state.get(Interface.KEY, []), capabilities)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

ifaces_state = [{'bridge': {}, 'name': 'br1', 'state': 'up', 'type': 'ovs-bridge'}]
capabilities = ['team']

    def validate_interface_capabilities(ifaces_state, capabilities):
        ifaces_types = {iface_state.get("type") for iface_state in ifaces_state}
        has_ovs_capability = NmstatePlugin.OVS_CAPABILITY in capabilities
        has_team_capability = NmstatePlugin.TEAM_CAPABILITY in capabilities
        ovs_is_running = is_ovs_running()
        for iface_type in ifaces_types:
            is_ovs_type = iface_type in (
                InterfaceType.OVS_BRIDGE,
                InterfaceType.OVS_INTERFACE,
                InterfaceType.OVS_PORT,
            )
            if is_ovs_type and not has_ovs_capability:
                if not ovs_is_running:
                    raise NmstateDependencyError(
>                       "openvswitch service is not started."
                    )
E                   libnmstate.error.NmstateDependencyError: openvswitch service is not started.


Expected results:
Test passing


Additional info:
Comment 7 Mingyu Shi 2021-06-20 05:05:00 UTC 
Verified with versions:
nmstate-1.1.0-0.3.alpha3.el8.noarch
nispor-1.1.0-2.el8.x86_64
NetworkManager-1.32.0-0.5.el8.x86_64
openvswitch2.13-2.13.0-105.el8fdp.x86_64

1. Configuring ovs inside a container works.

2. If OVS is unusable(for example, stopping the openvswitch daemon in the host), nmstate will raise an error when trying to update anything of ovs:
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/context.py", line 213, in wait_all_finish
    raise tmp_error
libnmstate.error.NmstateDependencyError: Activate profile uuid:ae19ff6a-2c17-4e0e-a021-3e2f16e3390e iface:ovsbr1 type: ovs-interface failed: failed to communicating with Open vSwitch database, <enum NM_DEVICE_STATE_REASON_OVSDB_FAILED of type NM.DeviceStateReason>

3. Even though openvswitch daemon stopped, the existed ovs interfaces(created by nmstate or NM) still can be shown:
# nmstatectl show ovsbr0
Unhandled IFLA_INFO_DATA for iface type Other("Ipoib")
Unhandled IFLA_INFO_DATA for iface type Other("Ipoib")
2021-06-20 04:55:46,358 root         DEBUG    NetworkManager version 1.31.90
2021-06-20 04:55:46,359 root         DEBUG    Async action: Retrieve applied config: ethernet eno1 started
2021-06-20 04:55:46,359 root         DEBUG    Async action: Retrieve applied config: ovs-port ovs-port-ovsbr0 started
2021-06-20 04:55:46,360 root         DEBUG    Async action: Retrieve applied config: ovs-interface ovsbr0 started
2021-06-20 04:55:46,360 root         DEBUG    Async action: Retrieve applied config: ovs-bridge ovsbr0 started
2021-06-20 04:55:46,362 root         DEBUG    Async action: Retrieve applied config: ethernet eno1 finished
2021-06-20 04:55:46,363 root         DEBUG    Async action: Retrieve applied config: ovs-port ovs-port-ovsbr0 finished
2021-06-20 04:55:46,363 root         DEBUG    Async action: Retrieve applied config: ovs-interface ovsbr0 finished
2021-06-20 04:55:46,364 root         DEBUG    Async action: Retrieve applied config: ovs-bridge ovsbr0 finished
2021-06-20 04:55:46,368 root         DEBUG    Interface ovs-interface.br0 found. Merging the interface information.
2021-06-20 04:55:46,368 root         DEBUG    Interface ethernet.eno1 found. Merging the interface information.
2021-06-20 04:55:46,368 root         DEBUG    Interface ethernet.eno2 found. Merging the interface information.
2021-06-20 04:55:46,368 root         DEBUG    Interface ethernet.eno3 found. Merging the interface information.
2021-06-20 04:55:46,368 root         DEBUG    Interface ethernet.eno4 found. Merging the interface information.
2021-06-20 04:55:46,368 root         DEBUG    Interface ethernet.ens1f0 found. Merging the interface information.
2021-06-20 04:55:46,368 root         DEBUG    Interface ethernet.ens1f1 found. Merging the interface information.
2021-06-20 04:55:46,368 root         DEBUG    Interface ovs-interface.ovsbr0 found. Merging the interface information.
Unhandled IFLA_INFO_DATA for iface type Other("Ipoib")
Unhandled IFLA_INFO_DATA for iface type Other("Ipoib")
Unhandled IFLA_INFO_DATA for iface type Other("Ipoib")
Unhandled IFLA_INFO_DATA for iface type Other("Ipoib")
---
dns-resolver:
  config: {}
  running:
    search:
    - rhts.eng.pek2.redhat.com
    server:
    - 10.73.2.107
    - 10.73.2.108
    - 10.66.127.10
route-rules:
  config: []
routes:
  config: []
  running: []
interfaces:
- name: ovsbr0
  type: ovs-interface
  state: up
  accept-all-mac-addresses: false
  ethtool:
    feature:
      highdma: true
      rx-gro: true
      rx-gro-list: false
      tx-checksum-ip-generic: true
      tx-generic-segmentation: true
      tx-gre-csum-segmentation: true
      tx-gre-segmentation: true
      tx-ipxip4-segmentation: true
      tx-ipxip6-segmentation: true
      tx-nocache-copy: false
      tx-scatter-gather-fraglist: true
      tx-sctp-segmentation: true
      tx-tcp-ecn-segmentation: true
      tx-tcp-mangleid-segmentation: true
      tx-tcp-segmentation: true
      tx-tcp6-segmentation: true
      tx-udp_tnl-csum-segmentation: true
      tx-udp_tnl-segmentation: true
      tx-vlan-hw-insert: true
      tx-vlan-stag-hw-insert: true
  ipv4:
    enabled: false
    address: []
    dhcp: false
  ipv6:
    enabled: false
    address: []
    autoconf: false
    dhcp: false
  lldp:
    enabled: false
  mac-address: 82:51:53:76:59:49
  mtu: 1500
  ovs-db:
    external_ids: {}
- name: ovsbr0
  type: ovs-bridge
  state: up
  bridge:
    options:
      fail-mode: ''
      mcast-snooping-enable: false
      rstp: false
      stp: false
    port:
    - name: ovsbr0
  lldp:
    enabled: false
  ovs-db:
    external_ids: {}
Comment 9 errata-xmlrpc 2021-11-09 17:43:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4157