RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2019457 - nmstate changes names of existing ovs-bridge ports when attaching a veth
Summary: nmstate changes names of existing ovs-bridge ports when attaching a veth
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: nmstate
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Radim Hrazdil
QA Contact: Mingyu Shi
URL:
Whiteboard:
Depends On:
Blocks: 2034138 2034139
TreeView+ depends on / blocked
 
Reported: 2021-11-02 15:28 UTC by Radim Hrazdil
Modified: 2022-05-10 13:54 UTC (History)
7 users (show)

Fixed In Version: nmstate-1.2.0-0.1.alpha2.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2034138 2034139 (view as bug list)
Environment:
Last Closed: 2022-05-10 13:34:48 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github nmstate nmstate pull 1724 0 None open ovs: remove ovs-port- prefix for OVS bridge LAG ports 2021-11-18 10:49:47 UTC
Red Hat Issue Tracker RHELPLAN-101499 0 None None None 2021-11-02 15:31:39 UTC
Red Hat Product Errata RHEA-2022:1772 0 None None None 2022-05-10 13:35:11 UTC

Description Radim Hrazdil 2021-11-02 15:28:29 UTC
Description of problem:

When applying nmstate desiredState on OCP with OVNKubernetes that connects a veth interface to br-ex ovs-bridge, the names of ports of the ovs-bridge are prepended with 'ovs-port-'.

State br-ex state before applying desiredState:
$ ovs-vsctl show
Bridge  br-ex
    Port patch-br-ex_xyz.redhat.com-to-br-int
        Interface patch-br-ex_xyz.redhat.com-to-br-int
            type: patch
            options: {peer=patch-br-int-to-br-ex_xyz.redhat.com}
    Port br-ex
        Interface br-ex
            type: internal
    Port eno1
        Interface eno1
            type: system



After applying desiredState
$ ovs-vsctl show
Bridge  br-ex
    Port patch-br-ex_xyz.redhat.com-to-br-int
        Interface patch-br-ex_xyz.redhat.com-to-br-int
            type: patch
            options: {peer=patch-br-int-to-br-ex_xyz.redhat.com}
    Port ovs-port-ovn-veth
        Interface ovn-veth
            type: system
    Port ovs-port-br-ex
        Interface br-ex
            type: internal
    Port ovs-port-eno1
        Interface eno1
            type: system


the desiredState used:
interfaces:
- name: br-ex
  type: ovs-interface
  state: up
- name: eno1
  type: ethernet
  state: up
- name: br-ex
  type: ovs-bridge
  state: up
  bridge:
    options:
      fail-mode: ''
      mcast-snooping-enable: false
      rstp: false
      stp: false
    port:
    - name: br-ex
    - name: eno1
    - name: ovn-veth
  lldp:
    enabled: false
- name: cnv-veth
  type: veth
  state: up
  ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  veth:
    peer: ovn-veth
- name: ovn-veth
  type: veth
  state: up
  ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  veth:
    peer: cnv-veth



As a consequence, this seems to break ovnkube-node pods, as these pods are failing with the following error after applying the desiredState:
I1101 14:47:29.362491 2671168 ovs.go:213] exec(16): stderr: "ovs-vsctl: no row \"ovs-port-eno1\" in table Interface\n"
I1101 14:47:29.362495 2671168 ovs.go:215] exec(16): err: exit status 1
F1101 14:47:29.362522 2671168 ovnkube.go:130] Bridge for interface failed for br-ex: Failed to find intfName for br-ex: failed to get ofport of ovs-port-eno1, stderr: "ovs-vsctl: no row \"ovs-port-eno1\" in table Interface\n", error: exit status 1




Version-Release number of selected component (if applicable):
nmstate 1.0.4

$ oc version
Client Version: 4.9.0
Server Version: 4.9.5
Kubernetes Version: v1.22.0-rc.0+a44d0f0


How reproducible:
100

Steps to Reproduce:
1. Deploy OCP with OVNKubernetes
2. Apply desiredState given above
3. ovnkube-node pods are crashing (and other pods as well as a result)



Expected results:

nmstate shouldn't be changing names of existing ovs-bridge ports.

Comment 1 Fernando F. Mancera 2021-11-18 10:49:47 UTC
Upstream PR: https://github.com/nmstate/nmstate/pull/1724

Comment 2 Gris Ge 2021-11-19 10:29:27 UTC
Hi Mingyu,

Please do pretest using copr repo of these PRs:

 * RHEL 8.4: https://github.com/nmstate/nmstate/pull/1738
 * RHEL 8.5: https://github.com/nmstate/nmstate/pull/1737
 * RHEL 8.6: https://github.com/nmstate/nmstate/pull/1724

This bug is requesting 8.4 zstream (RHEL 8.5 is also required so we don't have regression comparing to 8.4.z).

Comment 3 Mingyu Shi 2021-11-22 06:14:43 UTC
Working on it

Comment 4 Mingyu Shi 2021-11-22 08:44:55 UTC
Hi Radim,

Could you please tell me how to reach the state *BEFORE* you applied desiredState? Or before you applied the new state, please show me something like:

ip link
nmstatectl show br-ex,(and other ovs stuff)
nmcli con

If I understand correctly, in the *BEFORE* stage, all the ovs devices were uncontrolled by NetworkManager, right? Then it appears that there's no chance for nmstate to configure it, as nmstate now cannot deal with a runtime ovs bridge.
So I got confused that how you can successfully apply the new state...

Thank you!

Comment 5 Radim Hrazdil 2021-11-22 10:10:33 UTC
Created attachment 1842977 [details]
nmstatectl files

Hello Mingyu Shi,

The state before applying the desiredState is a clean OCP installation after the process of migration to OVN Kubernetes:
https://docs.openshift.com/container-platform/4.9/networking/ovn_kubernetes_network_provider/migrate-from-openshift-sdn.html#nw-ovn-kubernetes-migration_migrate-from-openshift-sdn

The OVS components are created during this process and also are managed by NetworkManager

Comment 6 Mingyu Shi 2021-11-23 10:59:45 UTC
Tested on 8.6/8.5/8.4, take 8.4 as an example:

Tested with versions:
nmstate-1.0.5-0.20211119102720219917.pr1738.2.g8590e6d.el8.noarch
nispor-1.0.1-4.el8.x86_64
NetworkManager-1.30.0-13.el8_4.x86_64
openvswitch2.13-2.13.0-124.el8fdp.x86_64


nmcli con add type ovs-bridge con.interface-name br-ex
nmcli con add type ovs-port conn.interface-name br-ex master br-ex
nmcli con add type ovs-interface slave-type ovs-port conn.interface-name br-ex master br-ex ipv4.method manual ipv4.addresses 192.168.199.2/24 
nmcli con add type ovs-port conn.interface-name eno1 master br-ex
nmcli conn add type ethernet conn.interface eno1 master eno1
echo "interfaces:
- name: br-ex
  type: ovs-interface
  state: up
- name: eno1
  type: ethernet
  state: up
- name: br-ex
  type: ovs-bridge
  state: up
  bridge:
    options:
      fail-mode: ''
      mcast-snooping-enable: false
      rstp: false
      stp: false
    port:
    - name: br-ex
    - name: eno1
    - name: ovn-veth
  lldp:
    enabled: false
- name: cnv-veth
  type: veth
  state: up
  ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  veth:
    peer: ovn-veth
- name: ovn-veth
  type: veth
  state: up
  ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  veth:
    peer: cnv-veth" | nmstatectl set -

With older version of nmstate, the result:
[17:28:42@kvm-04-guest29 ~]0# ovs-vsctl show
4bb50d34-88d1-4dc8-86cb-b5f97d8d18c0
    Bridge br-ex
        Port ovs-port-ovn-veth
            Interface ovn-veth
                type: system
        Port ovs-port-eno1
            Interface eno1
                type: system
        Port ovs-port-br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.13.5"

The result if using fixed version:
[18:58:10@kvm-04-guest29 ~]0# ovs-vsctl show
4bb50d34-88d1-4dc8-86cb-b5f97d8d18c0
    Bridge br-ex
        Port ovn-veth
            Interface ovn-veth
                type: system
        Port eno1
            Interface eno1
                type: system
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.13.5"

Comment 13 Mingyu Shi 2022-01-11 09:14:41 UTC
Hi Radim,

It seems that the current fix only removes "PREFIX-" in the port name, it meets your scenario, but the root cause "changing names of existing ovs-bridge ports" still exists.
Does it OK for you?

Here's an example:
For testing, I manually prefixed "ovsport_" in port name:
[17:03:47@mshi860 ~]0# ovs-vsctl show
95ae970e-5943-488f-a63b-f9a07cf7a963
    Bridge br-ex
        Port ovsport_ovspatch0
            Interface ovspatch0
                type: patch
                options: {peer=ovspatch0_p}
        Port ovsport_veth0
            Interface veth0
                type: system
        Port ovsport_ovspatch0_p
            Interface ovspatch0_p
                type: patch
                options: {peer=ovspatch0}
        Port ovsport_br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.13.5"

After applied the new state, the port name changed(it removed the prefix part "ovsport_"):
[17:04:02@mshi860 ~]0# ovs-vsctl show
95ae970e-5943-488f-a63b-f9a07cf7a963
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port veth0
            Interface veth0
                type: system
        Port ovspatch0_p
            Interface ovspatch0_p
                type: patch
                options: {peer=ovspatch0}
        Port ovspatch0
            Interface ovspatch0
                type: patch
                options: {peer=ovspatch0_p}
        Port ovn-veth
            Interface ovn-veth
                type: system
    ovs_version: "2.13.5"


nmstate-1.2.0-1.el8.x86_64
nispor-1.2.2-1.el8.x86_64
NetworkManager-1.36.0-0.3.el8.x86_64
openvswitch2.13-2.13.0-142.el8fdp.x86_64

Comment 15 Radim Hrazdil 2022-01-11 09:45:18 UTC
Hello Mingyu,

Thanks for the heads up.
In our use-case, we're not using custom prefixes, so the current behaviour works for us.

But I see your point that the BZ desicription is not being completely honest.
Maybe we can change the title to 
"nmstate prefixes names of existing ovs-bridge ports with 'ovs-port-'" ? What do you think?


In the future, a new BZ may be opened to achieve name consistency for manually prefixed interfaces, if the need arises.

Comment 16 Mingyu Shi 2022-01-12 10:58:57 UTC
As https://bugzilla.redhat.com/show_bug.cgi?id=2019457#c13 shows, nmstate now uses same name on ovs bridge, port and interface.

Comment 18 errata-xmlrpc 2022-05-10 13:34:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:1772


Note You need to log in before you can comment on or make changes to this bug.