Bug 1931376

Hello,

We have same results with used "macvtap" peripheral. I used the updates-testing repos.

On my another Servers, this problem does not appear but I don't used the updates-testing repos.

Sylvain

Inbar,

Those logs are taken after the upgrade finished?

Both of those VMs vm-upgrade-a and vm-upgrade-b are on the same host. They report correct IP addresses. The bridge they are connected to is available on their node and seems healthy.

The one issue I see on your cluster is that VMs are not getting the MAC address they were allocated using the KubeMacPool. This is caused by NetworkAttachmentDefinition missing the `cnv-tuning` plugin. Please adjust your NetworkAttachmentDefinitions to mimick https://docs.openshift.com/container-platform/4.6/virt/virtual_machines/vm_networking/virt-attaching-vm-multiple-networks.html#virt-creating-bridge-nad-cli_virt-attaching-multiple-networks.

I would need access to the cluster to perform further debugging and see where the traffic gets stuck.

(In reply to Sylvain Réault from comment #4)
> Hello,
> 
> We have same results with used "macvtap" peripheral. I used the
> updates-testing repos.
> 
> On my another Servers, this problem does not appear but I don't used the
> updates-testing repos.
> 
> Sylvain

Thanks for reporting this. Unfortunately, macvtap is not supported in OpenShift Virtualization. If you have issues with it, please open an Issue on KubeVirt's GitHub https://github.com/kubevirt/kubevirt/issues.

(In reply to Petr Horáček from comment #5)
> Inbar,
> 
> Those logs are taken after the upgrade finished?
> 
> Both of those VMs vm-upgrade-a and vm-upgrade-b are on the same host. They
> report correct IP addresses. The bridge they are connected to is available
> on their node and seems healthy.
> 
> The one issue I see on your cluster is that VMs are not getting the MAC
> address they were allocated using the KubeMacPool. This is caused by
> NetworkAttachmentDefinition missing the `cnv-tuning` plugin. Please adjust
> your NetworkAttachmentDefinitions to mimick
> https://docs.openshift.com/container-platform/4.6/virt/virtual_machines/
> vm_networking/virt-attaching-vm-multiple-networks.html#virt-creating-bridge-
> nad-cli_virt-attaching-multiple-networks.
> 
> I would need access to the cluster to perform further debugging and see
> where the traffic gets stuck.

I enabled cnv-tuning, I will run tests again and hopefully that solves the issue

nmstate-handler version is: v2.6.0-21

After nmstate-handler pods restart I lost the veth interface for the bridge.


Before restart pods:
5: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master br1test state UP mode DEFAULT group default qlen 1000
88: br1test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
90: vethca31f423@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1test state UP mode DEFAULT group default


After restart pods:
5: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master br1test state UP mode DEFAULT group default qlen 1000
88: br1test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000

And no connectivity between the VMs.
 

The code from the pod:

def _bring_slave_up_if_not_in_desire(self):
        """
        When slave been included in master, automactially set it as state UP
        if not defiend in desire state
        """
        for iface in self._ifaces.values():
            if iface.is_up and iface.is_master:
                cur_iface = self.current_ifaces.get(iface.name)
                for slave_name in iface.slaves:
                    if cur_iface and slave_name in cur_iface.slaves:
                        # Nmstate should bring up the port interface if it has
                        # been added to the state not in all transactions
                        continue
                    slave_iface = self._ifaces[slave_name]
                    if not slave_iface.is_desired and not slave_iface.is_up:
                        slave_iface.mark_as_up()
                        slave_iface.mark_as_changed()

I can confirm. While the veths stayed attached on a simple bridge without any NIC attached, when the bridge is attached to host's physical NIC, veths get disconnected.

Sound NNCP:

    interfaces:
    - bridge:
        options:
          stp:
            enabled: false
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        enabled: false
      name: br1
      state: up
      type: linux-bridge

Failing one:

    interfaces:
    - bridge:
        options:
          stp:
            enabled: false
        port:
        - name: ens9
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        enabled: false
      name: br1
      state: up
      type: linux-bridge

Failed to verify the latest fix.
Failed to apply nncp.
nmstate-handler version is: v2.6.0-22


Traceback (most recent call last):
  File "/usr/bin/nmstatectl", line 11, in <module>
    load_entry_point('nmstate==0.3.4', 'console_scripts', 'nmstatectl')()
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 67, in main
    return args.func(args)
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 267, in apply
    args.save_to_disk,
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 289, in apply_state
    save_to_disk=save_to_disk,
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 71, in apply
    _apply_ifaces_state(plugins, net_state, verify_change, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 104, in _apply_ifaces_state
    plugin.apply_changes(net_state, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/plugin.py", line 174, in apply_changes
    nm_applier.apply_changes(self.context, net_state, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 84, in apply_changes
    for iface in net_state.ifaces.all_ifaces.values():
AttributeError: 'Ifaces' object has no attribute 'all_ifaces'



code from nmstate-handler pod:

/usr/lib/python3.6/site-packages/libnmstate/ifaces/ifaces.py
def _bring_slave_up_if_not_in_desire(self):
        """
        When slave been included in master, automactially set it as state UP
        if not defiend in desire state
        """
        for iface in self._ifaces.values():
            if iface.is_desired and iface.is_up and iface.is_master:
                cur_iface = self.current_ifaces.get(iface.name)
                for slave_name in iface.slaves:
                    if cur_iface and slave_name in cur_iface.slaves:
                        # Nmstate should bring up the port interface if it has
                        # been added to the state not in all transactions
                        continue
                    slave_iface = self._ifaces[slave_name]
                    if not slave_iface.is_desired and not slave_iface.is_up:
                        slave_iface.mark_as_up()
                        slave_iface.mark_as_changed()

def _remove_unknown_interface_type_slaves(self):
        """
        When master containing slaves with unknown interface type or down
        state, they should be removed from master slave list before verifying.
        """
        for iface in self._ifaces.values():
            if iface.is_up and iface.is_master and iface.slaves:
                for slave_name in iface.slaves:
                    slave_iface = self._ifaces[slave_name]
                    if (
                        slave_iface.type == InterfaceType.UNKNOWN
                        or slave_iface.state != InterfaceState.UP
                    ):
                        iface.remove_slave(slave_name)

Verified.
nmstate-handler version is: v2.6.0-23

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 2.6.0 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0799