Bug 2073135 - After upgrade from OSP16.2.1 to 16.2.2 some ports missing OVS tags
Summary: After upgrade from OSP16.2.1 to 16.2.2 some ports missing OVS tags
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 16.2 (Train)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z3
: 16.2 (Train on RHEL 8.4)
Assignee: Rodolfo Alonso
QA Contact: Sanjay Upadhyay
URL:
Whiteboard:
Depends On:
Blocks: 2128250
TreeView+ depends on / blocked
 
Reported: 2022-04-07 17:26 UTC by Jamie Fargen
Modified: 2022-09-20 11:37 UTC (History)
21 users (show)

Fixed In Version: openstack-neutron-15.3.5-2.20220413175559.3810152.el8ost
Doc Type: No Doc Update
Doc Text:
If this bug requires documentation, please select an appropriate Doc Type value.
Clone Of:
: 2128250 (view as bug list)
Environment:
Last Closed: 2022-06-22 16:06:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker NFV-2506 0 None None None 2022-05-13 04:22:16 UTC
Red Hat Issue Tracker OSP-14564 0 None None None 2022-04-07 17:35:13 UTC
Red Hat Product Errata RHBA-2022:4793 0 None None None 2022-06-22 16:07:00 UTC

Description Jamie Fargen 2022-04-07 17:26:50 UTC
Description of problem:
VirtIO ports and VF representator ports are missing OVS tags.


Version-Release number of selected component (if applicable):
16.2.1 > 16.2.2


How reproducible:


Steps to Reproduce:
1. Upgrade OSP16.2.1 to OSP16.2.2

Actual results:
VMs with ports that have missing tags lose connectivity.

Expected results:
VMs to continue operating normally without configuraton issues.


Additional info:

Comment 13 Rodolfo Alonso 2022-04-19 16:18:04 UTC
Hello:

I've identified ISSUE 3, from c#8. The problem is solved in [1][2]. Again, this change was not intentionally pushed in this patch to fix an error found but to make this method more robust. It turns out that in the customer it is an issue.

In a nutshell:
- ISSUE 1: solved in [2] (already merged)
- ISSUE 2: the customer won't migrate to native, so we won't consider this anymore. I found out that is actually NOT causing any problem in the QoS driver.
- ISSUE 3: solved in [2].


openstack-neutron-15.3.5-2.20220304145337.34b1360.el8osttrunk is the first RPM with this code.

There is another issue detected while investigating the customer environment. The customer is using trunk bridges, that is another configuration never tested when the feature was implemented because that was never supported [3]. The presence of a port with the name "tpi-xxxxxxxx-xx" and VNIC type "direct", breaks the QoS implementation for HW offloaded devices. I know, for sure, the customer will request this as a must, but this should be consider as a RFE.

I'll push a patch, for now, to skip the "tpi-xxxxxxxx-xx" ports and avoid the QoS extension problem.

Regards.


[1]https://code.engineering.redhat.com/gerrit/c/neutron/+/288951/10/neutron/privileged/agent/linux/utils.py#27
[2]https://code.engineering.redhat.com/gerrit/c/neutron/+/288951
[3]https://bugs.launchpad.net/neutron/+bug/1639186

Comment 15 john.choo 2022-04-20 14:14:10 UTC
This is the error message we see on the hw-offload host after upgrading from OSP 16.2.2 to OSP 16.2.3
Error from openvswitch-agent.log
2022-04-20 14:07:32.215 29328 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.extension_drivers.qos_driver [req-0b52458c-fb02-4392-8ba0-3bd2852fae06 - - - - -] Port representor ens3f0_63 does not have a PF reference
2022-04-20 14:07:32.216 29328 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-0b52458c-fb02-4392-8ba0-3bd2852fae06 - - - - -] Error while processing VIF ports: TypeError: 'NoneType' object is not iterable

Port status showing missing OVS "tag"
[root@tpa-vim-p-computecl-2 neutron]# ovs-vsctl --columns=name,tag,other_config --format table list port
name           tag other_config
-------------- --- ----------------------------------------------------------------------------------------------------------------------------------
vxlan-ac141007 []  {}
qvo2e786ea9-17 []  {}
vxlan-ac14100a []  {}
ens3f0_63      []  {net_uuid="ed95fb06-bb4d-40e4-8141-aa68cee1b6b8", network_type=vlan, physical_network=datacentre, segmentation_id="609"}
vxlan-ac14100c []  {}
qvo585d2373-ad 3   {net_uuid="ed95fb06-bb4d-40e4-8141-aa68cee1b6b8", network_type=vlan, physical_network=datacentre, segmentation_id="609", tag="3"}
qvo37dbc03d-bb 2   {net_uuid="8aadf5fe-d9ca-474f-a119-b7959d567f91", network_type=vlan, physical_network=datacentre, segmentation_id="2151", tag="2"}
br-ex          []  {}
qvo617e38e3-85 5   {net_uuid="98e66b32-94fc-4741-8753-9b1a49140b27", network_type=vlan, physical_network=datacentre, segmentation_id="631", tag="5"}
qvofd65f87a-83 []  {net_uuid="ed95fb06-bb4d-40e4-8141-aa68cee1b6b8", network_type=vlan, physical_network=datacentre, segmentation_id="609"}
patch-tun      []  {mcast-snooping-flood=False, mcast-snooping-flood-reports=False}
int-br-ex      []  {mcast-snooping-flood=False, mcast-snooping-flood-reports=False}
patch-int      []  {}
qvo75d3a389-93 []  {net_uuid="8aadf5fe-d9ca-474f-a119-b7959d567f91", network_type=vlan, physical_network=datacentre, segmentation_id="2151"}
qvo543db827-df 2   {net_uuid="8aadf5fe-d9ca-474f-a119-b7959d567f91", network_type=vlan, physical_network=datacentre, segmentation_id="2151", tag="2"}
qvo0fae6d22-9e []  {}
qvoe1b81552-93 []  {net_uuid="b8ac47c3-f171-4c55-897f-87b46a0cfaeb", network_type=vlan, physical_network=datacentre, segmentation_id="610"}
vxlan-ac14100d []  {}
ens3f0_62      []  {}
vxlan-ac141006 []  {}
qvof369a9c5-8e 4   {net_uuid="6d5d2d27-d5af-4efc-8a7d-e3e843528a4a", network_type=vlan, physical_network=datacentre, segmentation_id="740", tag="4"}
vxlan-ac14100b []  {}
vxlan-ac141009 []  {}
qvoc7c88da9-5d []  {}
br-tun         []  {}
br-int         []  {}
phy-br-ex      []  {}
qvo109a3d4f-65 1   {net_uuid="b8ac47c3-f171-4c55-897f-87b46a0cfaeb", network_type=vlan, physical_network=datacentre, segmentation_id="610", tag="1"}
bond1          []  {}
[root@tpa-vim-p-computecl-2 neutron]# ovs-vsctl show | grep Bridge
    Bridge br-ex
    Bridge br-tun
    Bridge br-int
[root@tpa-vim-p-computecl-2 neutron]#

Comment 18 Raj Varadarajan 2022-04-25 22:26:22 UTC
Case 03191469

Comment 36 Sanjay Upadhyay 2022-05-18 11:47:16 UTC
As discussed with Haresh, this is already verified by Verizon and the specific scenario is not possible in QE lab.
Marking this as otherQE

Comment 42 errata-xmlrpc 2022-06-22 16:06:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4793


Note You need to log in before you can comment on or make changes to this bug.