Bug 2073135

Summary: After upgrade from OSP16.2.1 to 16.2.2 some ports missing OVS tags
Product: Red Hat OpenStack Reporter: Jamie Fargen <jfargen>
Component: openstack-neutronAssignee: Rodolfo Alonso <ralonsoh>
Status: CLOSED ERRATA QA Contact: Sanjay Upadhyay <supadhya>
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: averdagu, broose, chrisw, coldford, dalvarez, guilherme.giacon, hakhande, john.choo, marjones, mbracho, mburns, mlaniel, pgrist, ralonsoh, rvaradar, scohen, shaujohn, slinaber, spower, tschaibl, twilson
Target Milestone: z3Keywords: OtherQA, Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-15.3.5-2.20220413175559.3810152.el8ost Doc Type: No Doc Update
Doc Text:
If this bug requires documentation, please select an appropriate Doc Type value.
Story Points: ---
Clone Of:
: 2128250 (view as bug list) Environment:
Last Closed: 2022-06-22 16:06:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2128250    

Description Jamie Fargen 2022-04-07 17:26:50 UTC
Description of problem:
VirtIO ports and VF representator ports are missing OVS tags.


Version-Release number of selected component (if applicable):
16.2.1 > 16.2.2


How reproducible:


Steps to Reproduce:
1. Upgrade OSP16.2.1 to OSP16.2.2

Actual results:
VMs with ports that have missing tags lose connectivity.

Expected results:
VMs to continue operating normally without configuraton issues.


Additional info:

Comment 13 Rodolfo Alonso 2022-04-19 16:18:04 UTC
Hello:

I've identified ISSUE 3, from c#8. The problem is solved in [1][2]. Again, this change was not intentionally pushed in this patch to fix an error found but to make this method more robust. It turns out that in the customer it is an issue.

In a nutshell:
- ISSUE 1: solved in [2] (already merged)
- ISSUE 2: the customer won't migrate to native, so we won't consider this anymore. I found out that is actually NOT causing any problem in the QoS driver.
- ISSUE 3: solved in [2].


openstack-neutron-15.3.5-2.20220304145337.34b1360.el8osttrunk is the first RPM with this code.

There is another issue detected while investigating the customer environment. The customer is using trunk bridges, that is another configuration never tested when the feature was implemented because that was never supported [3]. The presence of a port with the name "tpi-xxxxxxxx-xx" and VNIC type "direct", breaks the QoS implementation for HW offloaded devices. I know, for sure, the customer will request this as a must, but this should be consider as a RFE.

I'll push a patch, for now, to skip the "tpi-xxxxxxxx-xx" ports and avoid the QoS extension problem.

Regards.


[1]https://code.engineering.redhat.com/gerrit/c/neutron/+/288951/10/neutron/privileged/agent/linux/utils.py#27
[2]https://code.engineering.redhat.com/gerrit/c/neutron/+/288951
[3]https://bugs.launchpad.net/neutron/+bug/1639186

Comment 15 john.choo 2022-04-20 14:14:10 UTC
This is the error message we see on the hw-offload host after upgrading from OSP 16.2.2 to OSP 16.2.3
Error from openvswitch-agent.log
2022-04-20 14:07:32.215 29328 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.extension_drivers.qos_driver [req-0b52458c-fb02-4392-8ba0-3bd2852fae06 - - - - -] Port representor ens3f0_63 does not have a PF reference
2022-04-20 14:07:32.216 29328 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-0b52458c-fb02-4392-8ba0-3bd2852fae06 - - - - -] Error while processing VIF ports: TypeError: 'NoneType' object is not iterable

Port status showing missing OVS "tag"
[root@tpa-vim-p-computecl-2 neutron]# ovs-vsctl --columns=name,tag,other_config --format table list port
name           tag other_config
-------------- --- ----------------------------------------------------------------------------------------------------------------------------------
vxlan-ac141007 []  {}
qvo2e786ea9-17 []  {}
vxlan-ac14100a []  {}
ens3f0_63      []  {net_uuid="ed95fb06-bb4d-40e4-8141-aa68cee1b6b8", network_type=vlan, physical_network=datacentre, segmentation_id="609"}
vxlan-ac14100c []  {}
qvo585d2373-ad 3   {net_uuid="ed95fb06-bb4d-40e4-8141-aa68cee1b6b8", network_type=vlan, physical_network=datacentre, segmentation_id="609", tag="3"}
qvo37dbc03d-bb 2   {net_uuid="8aadf5fe-d9ca-474f-a119-b7959d567f91", network_type=vlan, physical_network=datacentre, segmentation_id="2151", tag="2"}
br-ex          []  {}
qvo617e38e3-85 5   {net_uuid="98e66b32-94fc-4741-8753-9b1a49140b27", network_type=vlan, physical_network=datacentre, segmentation_id="631", tag="5"}
qvofd65f87a-83 []  {net_uuid="ed95fb06-bb4d-40e4-8141-aa68cee1b6b8", network_type=vlan, physical_network=datacentre, segmentation_id="609"}
patch-tun      []  {mcast-snooping-flood=False, mcast-snooping-flood-reports=False}
int-br-ex      []  {mcast-snooping-flood=False, mcast-snooping-flood-reports=False}
patch-int      []  {}
qvo75d3a389-93 []  {net_uuid="8aadf5fe-d9ca-474f-a119-b7959d567f91", network_type=vlan, physical_network=datacentre, segmentation_id="2151"}
qvo543db827-df 2   {net_uuid="8aadf5fe-d9ca-474f-a119-b7959d567f91", network_type=vlan, physical_network=datacentre, segmentation_id="2151", tag="2"}
qvo0fae6d22-9e []  {}
qvoe1b81552-93 []  {net_uuid="b8ac47c3-f171-4c55-897f-87b46a0cfaeb", network_type=vlan, physical_network=datacentre, segmentation_id="610"}
vxlan-ac14100d []  {}
ens3f0_62      []  {}
vxlan-ac141006 []  {}
qvof369a9c5-8e 4   {net_uuid="6d5d2d27-d5af-4efc-8a7d-e3e843528a4a", network_type=vlan, physical_network=datacentre, segmentation_id="740", tag="4"}
vxlan-ac14100b []  {}
vxlan-ac141009 []  {}
qvoc7c88da9-5d []  {}
br-tun         []  {}
br-int         []  {}
phy-br-ex      []  {}
qvo109a3d4f-65 1   {net_uuid="b8ac47c3-f171-4c55-897f-87b46a0cfaeb", network_type=vlan, physical_network=datacentre, segmentation_id="610", tag="1"}
bond1          []  {}
[root@tpa-vim-p-computecl-2 neutron]# ovs-vsctl show | grep Bridge
    Bridge br-ex
    Bridge br-tun
    Bridge br-int
[root@tpa-vim-p-computecl-2 neutron]#

Comment 18 Raj Varadarajan 2022-04-25 22:26:22 UTC
Case 03191469

Comment 36 Sanjay Upadhyay 2022-05-18 11:47:16 UTC
As discussed with Haresh, this is already verified by Verizon and the specific scenario is not possible in QE lab.
Marking this as otherQE

Comment 42 errata-xmlrpc 2022-06-22 16:06:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4793