Bug 2139701

Summary:	kubernetes-nmstate-operator deletes Virtual Functions created by sriov-fec-operator [rhel-8.4.0.z]
Product:	Red Hat Enterprise Linux 8	Reporter:	RHEL Program Management Team <pgm-rhel-tools>
Component:	nmstate	Assignee:	Fernando F. Mancera <ferferna>
Status:	CLOSED ERRATA	QA Contact:	Mingyu Shi <mshi>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.4	CC:	eglottma, ellorent, ferferna, fge, jiji, jishi, mori, mshi, network-qe, pibanezr, sfaye, till, tkimura
Target Milestone:	rc	Keywords:	Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	nmstate-1.0.2-20.el8_4	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2105141	Environment:
Last Closed:	2022-11-29 14:12:54 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2105141
Bug Blocks:

Comment 5 Mingyu Shi 2022-11-16 11:23:40 UTC

Verified with:
nmstate-1.0.2-20.el8_4.noarch
nispor-1.0.1-5.el8_4.x86_64
NetworkManager-1.30.0-13.el8_4.x86_64
Linux dell-per740-68.rhts.eng.pek2.redhat.com 4.18.0-305.25.1.el8_4.x86_64 #1 SMP Mon Oct 18 14:34:11 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
DISTRO=RHEL-8.4.0-updates-20211026.0

VFs are not touched when `sr-iov` subtree is not mentioned in the desired state.
But it needs to mention that if sriov configuration is written in NM connection:
sriov.total-vfs:                        4
sriov.vfs:                              0 mac=00:00:00:00:00:00 max-tx-rate=0 min-tx-rate=0 spoof-check=false trust=false, 1 mac=00:00:00:00:00:00 max-tx-rate=0 min-tx-rate=0 spoof-check=false trust=false, 2 mac=00:00:00:00:00:00 max-tx-rate=0 min-tx-rate=0 spoof-check=false trust=false, 3 mac=00:00:00:00:00:00 max-tx-rate=0 min-tx-rate=0 spoof-check=false trust=false

Removing the connection, or applied "state: absent", the VFs in the connection would be removed too.

[19:09:14@dell-per740-68 ~]0# echo 4 > /sys/class/net/ens3f0/device/sriov_numvfs 
[19:09:55@dell-per740-68 ~]0# echo 4 > /sys/class/net/ens3f1/device/sriov_numvfs 
[19:13:02@dell-per740-68 ~]0# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e4 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e5 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e6 brd ff:ff:ff:ff:ff:ff
5: ens1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:a0:94:b6 brd ff:ff:ff:ff:ff:ff
6: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e7 brd ff:ff:ff:ff:ff:ff
7: ens3f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:42:a1:5f:5c:58 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
8: ens3f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:42:a1:5f:5c:59 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
9: ens3f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether ae:df:ad:1b:d2:ab brd ff:ff:ff:ff:ff:ff
10: ens3f0v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether c2:fd:bb:f2:c4:31 brd ff:ff:ff:ff:ff:ff
11: ens3f0v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 36:06:14:69:78:c8 brd ff:ff:ff:ff:ff:ff
12: ens3f0v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 96:6f:6e:69:c8:0d brd ff:ff:ff:ff:ff:ff
13: ens3f1v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 52:73:22:d7:90:10 brd ff:ff:ff:ff:ff:ff
14: ens3f1v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0a:a4:5a:8b:85:63 brd ff:ff:ff:ff:ff:ff
15: ens3f1v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether da:8c:8d:b0:ca:e1 brd ff:ff:ff:ff:ff:ff
16: ens3f1v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 6a:26:07:99:5d:35 brd ff:ff:ff:ff:ff:ff
    [19:13:33@dell-per740-68 ~]0# nmstatectl set vf.yaml 
    /tmp/nmstatelog/2022-11-16-19:13:55-649423507.log
    Desired state applied: 
    ---
    interfaces:
    - name: ens3f0
      type: ethernet
      state: up
      description: vf test 1
      ipv4:
        enabled: true
        address:
        - ip: 192.168.199.1
          prefix-length: 24
      ipv6:
        enabled: true
        address:
        - ip: 192:168:199::1
          prefix-length: 64
      mtu: 2000
    - name: ens3f1
      type: ethernet
      state: up
      description: vf test 2
      ipv4:
        enabled: true
        address:
        - ip: 192.168.199.2
          prefix-length: 24
      ipv6:
        enabled: true
        address:
        - ip: 192:168:199::2
          prefix-length: 64
      mtu: 2000
/tmp/nmstatelog/2022-11-16-19:13:55-649423507.0.log nmstatectl set vf.yaml return 0
[19:14:10@dell-per740-68 ~]0# nmstatectl show ens3f[01]
2022-11-16 19:14:17,919 root         DEBUG    NetworkManager version 1.30.0
2022-11-16 19:14:17,920 root         DEBUG    Async action: Retrieve applied config: ethernet eno1 started
2022-11-16 19:14:17,921 root         DEBUG    Async action: Retrieve applied config: ethernet ens3f0 started
2022-11-16 19:14:17,921 root         DEBUG    Async action: Retrieve applied config: ethernet ens3f1 started
2022-11-16 19:14:17,923 root         DEBUG    Async action: Retrieve applied config: ethernet eno1 finished
2022-11-16 19:14:17,924 root         DEBUG    Async action: Retrieve applied config: ethernet ens3f0 finished
2022-11-16 19:14:17,924 root         DEBUG    Async action: Retrieve applied config: ethernet ens3f1 finished
---
dns-resolver:
  config: {}
  running:
    search:
    - rhts.eng.pek2.redhat.com
    server:
    - 10.11.5.19
    - 10.72.17.5
    - 10.68.5.26
route-rules:
  config: []
routes:
  config: []
  running:
  - destination: 192:168:199::/64
    metric: 101
    next-hop-address: '::'
    next-hop-interface: ens3f0
    table-id: 254
  - destination: 192:168:199::/64
    metric: 102
    next-hop-address: '::'
    next-hop-interface: ens3f1
    table-id: 254
  - destination: fe80::/64
    metric: 101
    next-hop-address: '::'
    next-hop-interface: ens3f0
    table-id: 254
  - destination: fe80::/64
    metric: 102
    next-hop-address: '::'
    next-hop-interface: ens3f1
    table-id: 254
  - destination: 192.168.199.0/24
    metric: 101
    next-hop-address: 0.0.0.0
    next-hop-interface: ens3f0
    table-id: 254
  - destination: 192.168.199.0/24
    metric: 102
    next-hop-address: 0.0.0.0
    next-hop-interface: ens3f1
    table-id: 254
interfaces:
- name: ens3f0
  type: ethernet
  state: up
  description: vf test 1
  ethernet:
    auto-negotiation: true
    duplex: full
    speed: 10000
    sr-iov:
      total-vfs: 4
      vfs:
      - _vf_iface_name: ens3f0v0
        id: 0
        mac-address: 00:00:00:00:00:00
        max-tx-rate: 0
        min-tx-rate: 0
        spoof-check: false
        trust: false
      - _vf_iface_name: ens3f0v1
        id: 1
        mac-address: 00:00:00:00:00:00
        max-tx-rate: 0
        min-tx-rate: 0
        spoof-check: false
        trust: false
      - _vf_iface_name: ens3f0v2
        id: 2
        mac-address: 00:00:00:00:00:00
        max-tx-rate: 0
        min-tx-rate: 0
        spoof-check: false
        trust: false
      - _vf_iface_name: ens3f0v3
        id: 3
        mac-address: 00:00:00:00:00:00
        max-tx-rate: 0
        min-tx-rate: 0
        spoof-check: false
        trust: false
  ipv4:
    enabled: true
    address:
    - ip: 192.168.199.1
      prefix-length: 24
    dhcp: false
  ipv6:
    enabled: true
    address:
    - ip: 192:168:199::1
      prefix-length: 64
    - ip: fe80::e42:a1ff:fe5f:5c58
      prefix-length: 64
    autoconf: false
    dhcp: false
  lldp:
    enabled: false
  mac-address: 0C:42:A1:5F:5C:58
  mtu: 2000
- name: ens3f1
  type: ethernet
  state: up
  description: vf test 2
  ethernet:
    auto-negotiation: true
    duplex: full
    speed: 10000
    sr-iov:
      total-vfs: 4
      vfs:
      - _vf_iface_name: ens3f1v0
        id: 0
        mac-address: 00:00:00:00:00:00
        max-tx-rate: 0
        min-tx-rate: 0
        spoof-check: false
        trust: false
      - _vf_iface_name: ens3f1v1
        id: 1
        mac-address: 00:00:00:00:00:00
        max-tx-rate: 0
        min-tx-rate: 0
        spoof-check: false
        trust: false
      - _vf_iface_name: ens3f1v2
        id: 2
        mac-address: 00:00:00:00:00:00
        max-tx-rate: 0
        min-tx-rate: 0
        spoof-check: false
        trust: false
      - _vf_iface_name: ens3f1v3
        id: 3
        mac-address: 00:00:00:00:00:00
        max-tx-rate: 0
        min-tx-rate: 0
        spoof-check: false
        trust: false
  ipv4:
    enabled: true
    address:
    - ip: 192.168.199.2
      prefix-length: 24
    dhcp: false
  ipv6:
    enabled: true
    address:
    - ip: 192:168:199::2
      prefix-length: 64
    - ip: fe80::e42:a1ff:fe5f:5c59
      prefix-length: 64
    autoconf: false
    dhcp: false
  lldp:
    enabled: false
  mac-address: 0C:42:A1:5F:5C:59
  mtu: 2000
ovs-db:
  external_ids:
    hostname: dell-per740-68.rhts.eng.pek2.redhat.com
    rundir: /var/run/openvswitch
    system-id: 624c230d-2b6f-43c4-817a-351676ddcb14
  other_config: {}
[19:14:17@dell-per740-68 ~]0# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e4 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e5 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e6 brd ff:ff:ff:ff:ff:ff
5: ens1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:a0:94:b6 brd ff:ff:ff:ff:ff:ff
6: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e7 brd ff:ff:ff:ff:ff:ff
7: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0c:42:a1:5f:5c:58 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
8: ens3f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0c:42:a1:5f:5c:59 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
9: ens3f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether ae:df:ad:1b:d2:ab brd ff:ff:ff:ff:ff:ff
10: ens3f0v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether c2:fd:bb:f2:c4:31 brd ff:ff:ff:ff:ff:ff
11: ens3f0v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 36:06:14:69:78:c8 brd ff:ff:ff:ff:ff:ff
12: ens3f0v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 96:6f:6e:69:c8:0d brd ff:ff:ff:ff:ff:ff
13: ens3f1v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 52:73:22:d7:90:10 brd ff:ff:ff:ff:ff:ff
14: ens3f1v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0a:a4:5a:8b:85:63 brd ff:ff:ff:ff:ff:ff
15: ens3f1v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether da:8c:8d:b0:ca:e1 brd ff:ff:ff:ff:ff:ff
16: ens3f1v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 6a:26:07:99:5d:35 brd ff:ff:ff:ff:ff:ff
[19:14:27@dell-per740-68 ~]0# rmiface ens3f[01]
ens3f[01]
/tmp/nmstatelog/2022-11-16-19:14:36-800176828.log
Desired state applied: 
---
interfaces:
- name: ens3f0
  type: ethernet
  state: absent
- name: ens3f1
  type: ethernet
  state: absent
/tmp/nmstatelog/2022-11-16-19:14:36-800176828.0.log nmstatectl set /tmp/rmiface_temp.yaml return 0
[19:14:39@dell-per740-68 ~]0# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e4 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e5 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e6 brd ff:ff:ff:ff:ff:ff
5: ens1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1b:21:a0:94:b6 brd ff:ff:ff:ff:ff:ff
6: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 34:48:ed:f8:ad:e7 brd ff:ff:ff:ff:ff:ff
7: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0c:42:a1:5f:5c:58 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
8: ens3f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0c:42:a1:5f:5c:59 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
9: ens3f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether ae:df:ad:1b:d2:ab brd ff:ff:ff:ff:ff:ff
10: ens3f0v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether c2:fd:bb:f2:c4:31 brd ff:ff:ff:ff:ff:ff
11: ens3f0v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 36:06:14:69:78:c8 brd ff:ff:ff:ff:ff:ff
12: ens3f0v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 96:6f:6e:69:c8:0d brd ff:ff:ff:ff:ff:ff
13: ens3f1v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 52:73:22:d7:90:10 brd ff:ff:ff:ff:ff:ff
14: ens3f1v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0a:a4:5a:8b:85:63 brd ff:ff:ff:ff:ff:ff
15: ens3f1v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether da:8c:8d:b0:ca:e1 brd ff:ff:ff:ff:ff:ff
16: ens3f1v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 6a:26:07:99:5d:35 brd ff:ff:ff:ff:ff:ff

Comment 9 errata-xmlrpc 2022-11-29 14:12:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8683