Bug 2162756

Summary: Centralized traffic due to unexpected behavior of reside-on-redirect-chassis and ovn-chassis-mac-mappings
Product: Red Hat OpenStack Reporter: Luis Tomas Bolivar <ltomasbo>
Component: openstack-neutronAssignee: Luis Tomas Bolivar <ltomasbo>
Status: CLOSED ERRATA QA Contact: Eduardo Olivares <eolivare>
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: amusil, chrisw, ctrautma, eolivare, ihrachys, imatza, jiji, mmichels, ralonsoh, scohen
Target Milestone: betaKeywords: Triaged
Target Release: 17.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-18.6.1-1.20230412171020.541d969.el9ost Doc Type: Bug Fix
Doc Text:
Before this update, VLAN network traffic was centralized over the Controller nodes. With this update, if all the tenant provider networks that are connected to a router are of the VLAN/Flat type, that traffic is now distributed. The node that contains the instance sends the traffic directly.
Story Points: ---
Clone Of:
: 2177155 (view as bug list) Environment:
Last Closed: 2023-08-16 01:13:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2007120    
Bug Blocks: 2177155    
Attachments:
Description Flags
OVN NB DB none

Description Luis Tomas Bolivar 2023-01-20 18:43:01 UTC
Created attachment 1939483 [details]
OVN NB DB

Created attachment 1939483 [details]
OVN NB DB

In the next setup of openstack, the reside-on-redirect-chassis and ovn-chassis-mac-mappings are not behaving as expected, and is wrongly using geneve tunnels for VLAN networks, leading to MTU issues

We have a vlan provider network (provisioning) connected to a router (bm_router) that is connected (gateway) to a external flat provider network (nova)
$ openstack network show provisioning
+---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                            |
+---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                               |
| availability_zone_hints   |                                                                                                                                                  |
| availability_zones        | nova                                                                                                                                             |
| created_at                | 2023-01-19T17:57:32Z                                                                                                                             |
| description               |                                                                                                                                                  |
| dns_domain                |                                                                                                                                                  |
| id                        | e3804743-5366-45a0-8326-59190fd93ee9                                                                                                             |
| ipv4_address_scope        | None                                                                                                                                             |
| ipv6_address_scope        | None                                                                                                                                             |
| is_default                | False                                                                                                                                            |
| is_vlan_transparent       | None                                                                                                                                             |
| location                  | cloud='', project.domain_id=, project.domain_name=, project.id='68766b5c07fc4221bfc538823bf4676a', project.name=, region_name='regionOne', zone= |
| mtu                       | 1500                                                                                                                                             |
| name                      | provisioning                                                                                                                                     |
| port_security_enabled     | True                                                                                                                                             |
| project_id                | 68766b5c07fc4221bfc538823bf4676a                                                                                                                 |
| provider:network_type     | vlan                                                                                                                                             |
| provider:physical_network | tenant                                                                                                                                           |
| provider:segmentation_id  | 246                                                                                                                                              |
| qos_policy_id             | None                                                                                                                                             |
| revision_number           | 3                                                                                                                                                |
| router:external           | Internal                                                                                                                                         |
| segments                  | None                                                                                                                                             |
| shared                    | True                                                                                                                                             |
| status                    | ACTIVE                                                                                                                                           |
| subnets                   | a7186fe6-f5c6-449e-9232-1f62c7ed8309                                                                                                             |
| tags                      |                                                                                                                                                  |
| updated_at                | 2023-01-20T14:52:16Z                                                                                                                             |
+---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+

$ openstack router show bm_router
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                                   |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                                      |
| availability_zone_hints |                                                                                                                                                                                         |
| availability_zones      |                                                                                                                                                                                         |
| created_at              | 2023-01-19T17:57:45Z                                                                                                                                                                    |
| description             |                                                                                                                                                                                         |
| external_gateway_info   | {"network_id": "cb84816a-3a02-41f4-b7ec-8a5b0638da5b", "external_fixed_ips": [{"subnet_id": "fbba8651-a043-488e-9273-2733611aefe1", "ip_address": "10.46.44.97"}], "enable_snat": true} |
| flavor_id               | None                                                                                                                                                                                    |
| id                      | e9719af0-bfc0-4218-9787-fd9a553c530b                                                                                                                                                    |
| interfaces_info         | [{"port_id": "6b0624c2-f317-408e-ac3f-65cc250b3566", "ip_address": "172.27.7.1", "subnet_id": "a7186fe6-f5c6-449e-9232-1f62c7ed8309"}]                                                  |
| location                | cloud='', project.domain_id=, project.domain_name=, project.id='68766b5c07fc4221bfc538823bf4676a', project.name=, region_name='regionOne', zone=                                        |
| name                    | bm_router                                                                                                                                                                               |
| project_id              | 68766b5c07fc4221bfc538823bf4676a                                                                                                                                                        |
| revision_number         | 4                                                                                                                                                                                       |
| routes                  |                                                                                                                                                                                         |
| status                  | ACTIVE                                                                                                                                                                                  |
| tags                    |                                                                                                                                                                                         |
| updated_at              | 2023-01-19T17:57:49Z                                                                                                                                                                    |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

 openstack network show nova
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                                            |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                                               |
| availability_zone_hints   |                                                                                                                                                                  |
| availability_zones        | nova                                                                                                                                                             |
| created_at                | 2023-01-19T17:31:24Z                                                                                                                                             |
| description               |                                                                                                                                                                  |
| dns_domain                |                                                                                                                                                                  |
| id                        | cb84816a-3a02-41f4-b7ec-8a5b0638da5b                                                                                                                             |
| ipv4_address_scope        | None                                                                                                                                                             |
| ipv6_address_scope        | None                                                                                                                                                             |
| is_default                | False                                                                                                                                                            |
| is_vlan_transparent       | None                                                                                                                                                             |
| location                  | cloud='', project.domain_id=, project.domain_name='Default', project.id='381cffb54c8e402b8b7f7515aae5375f', project.name='admin', region_name='regionOne', zone= |
| mtu                       | 1500                                                                                                                                                             |
| name                      | nova                                                                                                                                                             |
| port_security_enabled     | True                                                                                                                                                             |
| project_id                | 381cffb54c8e402b8b7f7515aae5375f                                                                                                                                 |
| provider:network_type     | flat                                                                                                                                                             |
| provider:physical_network | datacentre                                                                                                                                                       |
| provider:segmentation_id  | None                                                                                                                                                             |
| qos_policy_id             | None                                                                                                                                                             |
| revision_number           | 2                                                                                                                                                                |
| router:external           | External                                                                                                                                                         |
| segments                  | None                                                                                                                                                             |
| shared                    | False                                                                                                                                                            |
| status                    | ACTIVE                                                                                                                                                           |
| subnets                   | fbba8651-a043-488e-9273-2733611aefe1                                                                                                                             |
| tags                      |                                                                                                                                                                  |
| updated_at                | 2023-01-19T17:31:30Z                                                                                                                                             |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Then we create a VM on the provisioning network, on compute-0:
$ openstack server list --all --long -c Name -c Networks -c Host
+-----------------------------+---------------------------+---------------------------+
| Name                        | Networks                  | Host                      |
+-----------------------------+---------------------------+---------------------------+
| ostest-hlksb-master-0       | provisioning=172.27.7.193 | compute-0.redhat.local    |
+-----------------------------+---------------------------+---------------------------+

This VM has MTU problems as the traffic is tunneled to the gateway port on controller-1, via geneve (even though none of the networks are of geneve type), and
despite reside-on-redirect-chassis-flag being set to False and ovn-chassis-mac-mappings being configured:
$ sudo ovs-vsctl list open . | grep mac-mappings
external_ids        : {hostname=compute-0.redhat.local, ovn-bridge=br-int, ovn-bridge-mappings="datacentre:br-ex,tenant:br-isolated", ovn-chassis-mac-mappings="datacentre:fa:16:3e:08:1b:64,tenant:fa:16:3e:6f:c6:30", ovn-encap-ip="172.17.2.79", ovn-encap-type=geneve, ovn-match-northd-version="true", ovn-monitor-all="true", ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:172.17.1.122:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="1b3edb7b-ca70-49bf-9c08-1de2790ef2e4"}

The expectation based on documentation here [1] are that:
- If reside-on-redirect-chassis is set to false (as in this case), the traffic is tunneled to the node with the gateway port. However, it is not expected to be tunneled with geneve since the network is a vlan network, right?
- In addition, as ovn-chassis-mac-mappings is configured, expectation is that traffic is not tunneled to the gateway port but directly out


In addition, I tested changing the value of reside-on-redirect-chassis flag to True. In that case the traffic is not tunneled anymore (thus no MTU issue), but, as expected, it is centralized through the gateway port node. 


Version-Release number of selected component (if applicable):
Tested with ovn-2021-21.12.0-94 and ovn-2021-21.12.0-104

[1] https://github.com/ovn-org/ovn/blob/main/ovn-nb.xml#L2922

Comment 1 Mark Michelson 2023-02-20 16:34:20 UTC
Hi, I had a look at the configuration and documentation, and based on what I'm seeing, it appears there's a bit of configuration missing that should make this work. Since you do not have options:reside-on-redirect-chassis set to true on the gateway port, you have to set options:redirect-type=bridged in order for traffic to be redirected over the localnet port instead of being tunneled. According to the NB DB uploaded here, that option is not set on the gateway router port.

Also, since the southbound database has not been uploaded, we can only assume the ovn-chassis-mac-mappings have been set correctly.

Please do the following:

1) Try setting options:redirect-type=bridged on the gateway router port and retry your scenario. If the traffic is sent over the localnet port, then all is well and we can close this issue.
2) If (1) doesn't work, then please upload your southbound database so that we can attempt to reproduce with the same ovn-chassis-mac-mappings configured.

Thanks!

Comment 2 Luis Tomas Bolivar 2023-02-21 15:54:29 UTC
(In reply to Mark Michelson from comment #1)
> Hi, I had a look at the configuration and documentation, and based on what
> I'm seeing, it appears there's a bit of configuration missing that should
> make this work. Since you do not have options:reside-on-redirect-chassis set
> to true on the gateway port, you have to set options:redirect-type=bridged
> in order for traffic to be redirected over the localnet port instead of
> being tunneled. According to the NB DB uploaded here, that option is not set
> on the gateway router port.
> 
> Also, since the southbound database has not been uploaded, we can only
> assume the ovn-chassis-mac-mappings have been set correctly.
> 
> Please do the following:
> 
> 1) Try setting options:redirect-type=bridged on the gateway router port and
> retry your scenario. If the traffic is sent over the localnet port, then all
> is well and we can close this issue.
> 2) If (1) doesn't work, then please upload your southbound database so that
> we can attempt to reproduce with the same ovn-chassis-mac-mappings
> configured.
> 
> Thanks!

Either I'm setting it wrong or it still does not work. If I set the option redirect-type=bridged [1] it does nothing and the traffic is still going tunneled to the controller. I'm attaching the new DBs for the new setup 

[1] ovn-nbctl set logical_router_port 099c0427-d457-4a4d-b4fc-58c0a20db544 options:redirect-type=bridged

Comment 5 Ales Musil 2023-02-28 08:23:38 UTC
Hi,

to me it still seems like misconfiguration. Looking through the SB DB you did set the redirect-type to bridged,
but for the other LRP. The DGP is "lrp-9dfdf3ed-9ace-4baf-a611-934cd9f0ec44".

This is also mentioned in documentation that it works only for DGP:

"If set to bridged on a distributed  gateway  port,  this  option
causes  OVN  to  redirect  packets to the gateway chassis over a
localnet port instead of a tunnel.  The  relevant  chassis  must
share a localnet port."

Can you please try to set it on the DGP instead and check if there is still an issue? 
We have a test for this scenario that is working fine, but there might be something wrong
nevertheless. 

Thanks,
Ales

Comment 6 Luis Tomas Bolivar 2023-02-28 08:36:06 UTC
Sorry, my bad, I'll try to reprovision and check again by setting it on the lrp-XXX asociated to the OVN router gateway port (the one that has the cr-lrp associated to it)

Comment 7 Luis Tomas Bolivar 2023-03-01 10:09:55 UTC
Testing it on the different port does not seems to work. Ales pointing out it may be due to https://bugzilla.redhat.com/show_bug.cgi?id=2007120

Comment 8 Luis Tomas Bolivar 2023-03-01 11:49:09 UTC
And yet another update, the "options:redirect-type=bridged" flag works as expected, the issue was that neutron is not aware of it and it is removing it automatically. So moving this bug to neutron so that this flag is managed properly

Comment 30 errata-xmlrpc 2023-08-16 01:13:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577