Bug 1673650

Summary: OSP14 Hackfest: OVS>OVN migration does not update neutron provider:network_type field, prevents any further manipulation of existing tenant networks
Product: Red Hat OpenStack Reporter: Joe Antkowiak <joea>
Component: python-networking-ovnAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED ERRATA QA Contact: Roman Safronov <rsafrono>
Severity: high Docs Contact:
Priority: high    
Version: 16.1 (Train)CC: apevec, dalvarez, jamsmith, jlibosva, lhh, lmartins, majopela, nlevinki, tkajinam
Target Milestone: z7Keywords: Reopened, Triaged, ZStream
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: python-networking-ovn-7.3.1-1.20210713093309.4e24f4c.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-09 20:17:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joe Antkowiak 2019-02-07 16:58:38 UTC
Description of problem:
After executing the OVS>OVN migration process, tenant networks that existed pre-migration (as VXLAN network types) do not get their neutron provider:network_type updated to geneve from vxlan.  This prevents any further manipulation/change to the tenant network settings.

Version-Release number of selected component (if applicable):
OSP14

How reproducible:
Easily

Steps to Reproduce:
1. Deploy OSP14 overcloud with OVS, create vxlan tenant networks
2. Run OVS>OVN migration process
3. View and try to modify tenant networks that were originally vxlan and were converted to geneve

Actual results:
This network is actually geneve, but still shows vxlan in output:

(overcloud) [stack@joea-director14 ~]$ openstack network show tenant                                                                                                                                             
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2019-02-06T15:21:41Z                 |
| description               |                                      |
| dns_domain                |                                      |
| id                        | d87ec876-85ed-43cb-8535-d0c115e423bc |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_transparent       | None                                 |
| mtu                       | 1442                                 |
| name                      | tenant                               |
| port_security_enabled     | True                                 |
| project_id                | 093a314be2d24707a21fcb75a4bc65a6     |
| provider:network_type     | vxlan                                |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 4                                    |
| qos_policy_id             | None                                 |
| revision_number           | 7                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | c2909ab7-4617-4f44-845d-0d58bbea8c1a |
| tags                      | adapted_mtu, test                    |
| updated_at                | 2019-02-07T16:50:42Z                 |
+---------------------------+--------------------------------------+
(overcloud) [stack@joea-director14 ~]$ openstack network set --disable-port-security tenant
BadRequestException: 400: Client Error for url: http://10.9.66.103:9696/v2.0/networks/d87ec876-85ed-43cb-8535-d0c115e423bc, {"NeutronError": {"message": "Invalid input for operation: Network type vxlan is not supported.", "type": "InvalidInput", "detail": ""}}


Expected results:

Neutron fields reporting network_type should be geneve, and we should be able to modify network configuration as per usual

Additional info:

OSP14 Hackfest

Comment 1 Joe Antkowiak 2019-02-07 17:18:57 UTC
Creating new tenant networks after migration results in the correct network_type value and this network can be manipulated without issue:

(overcloud) [stack@joea-director14 ~]$ openstack network show tenant2
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2019-02-07T17:16:19Z                 |
| description               |                                      |
| dns_domain                |                                      |
| id                        | df1f839c-958e-4984-8bbf-52cdfcfde6be |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_transparent       | None                                 |
| mtu                       | 1442                                 |
| name                      | tenant2                              |
| port_security_enabled     | True                                 |
| project_id                | 093a314be2d24707a21fcb75a4bc65a6     |
| provider:network_type     | geneve                               |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 32                                   |
| qos_policy_id             | None                                 |
| revision_number           | 3                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 79957257-6aaa-4814-b36d-97b4098697f5 |
| tags                      |                                      |
| updated_at                | 2019-02-07T17:16:20Z                 |
+---------------------------+--------------------------------------+
(overcloud) [stack@joea-director14 ~]$ openstack network set --disable-port-security tenant2
(overcloud) [stack@joea-director14 ~]$

Comment 2 Daniel Alvarez Sanchez 2019-02-08 16:46:44 UTC
Good catch. We are updating the MTU but not the type of the existing networks.
This is tricky as we're failing the precommit action of the network update due to an unsupported type [0]. I can think of a possible solution which would be:

1. After migration, add 'vxlan' as a temporary valid value for tenant network types [1]. Restart neutron server containers.
2. Update all existing networks to change the type to 'geneve'
3. Revert changes made in step 1.

[0] https://github.com/openstack/networking-ovn/blob/fe8b9c75e7b8ede8a6e8c774fdbdd9448b0ef232/networking_ovn/ml2/mech_driver.py#L280
[1] https://github.com/openstack/networking-ovn/blob/85c227e1fff98fcbed3da057a1e02efbe176440f/migration/migrate-to-ovn.yml#L108

Comment 3 Mike Burns 2020-08-26 21:24:45 UTC
This Release is retired.  If this bug is still relevant, please reopen and retarget to an open release.

Comment 5 Daniel Alvarez Sanchez 2021-02-02 13:43:07 UTC
Now ML2/OVN supports VXLAN overlay so we need to account for the case when we migrate to ML2/OVN but we want to keep VXLAN (eg. to not change MTU)

Comment 6 Takashi Kajinami 2021-03-11 05:56:32 UTC
One customer now observes the same issue after they have completed migration from ovs to ovn
in their RHOSP16.1 deployment which was upgraded from RHOSP13.

I confirmed that their existing networks still have network_type: vxlan instead of geneve,
and they can't update these vxlan networks because of the same error.

I don't have any handy deployment to test this now but I don't think that neutron supports updating 
network_type of existing networks. There is a validation logic implemented in ml2 plugin layer
which blocks updating provider network attribute except for segmentation id.
 https://github.com/openstack/neutron/blob/stable/train/neutron/plugins/ml2/plugin.py#L866-L884

So I'm afraid that only option is to allow vxlan for tenant networks.


@Daniel
Could you check above points ?

Also, I'd like to ask you to confirm that using vxlan with ml2-ovn is currently supported
in RHOSP16.1.

As per upstream change to enable vxlan support, I see vxlan is supported with ovn>20.09
and we do have the version meeting that requirements.
Also I noticed that we do enable vxlan during ovs->ovn migration.
So I believe usage of vxlan is supported, but I could not find any clear documentation
which explains it.
 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/753453

Comment 8 Takashi Kajinami 2021-03-15 23:50:14 UTC
@ Jakub

Could you help me get some feedback here ?

I've checked the networking-ovn package in RHOSP, but I'm afraid that it doesn't include the following change
which added support for vxlan network to ovn drivers. So I'm afraid that it is not yet supported to use vxlan
in ml2+ovn deployment...

 https://review.opendev.org/c/openstack/neutron/+/734889

Comment 9 Jakub Libosvar 2021-03-23 10:19:10 UTC
(In reply to Takashi Kajinami from comment #8)
> @ Jakub
> 
> Could you help me get some feedback here ?
> 
> I've checked the networking-ovn package in RHOSP, but I'm afraid that it
> doesn't include the following change
> which added support for vxlan network to ovn drivers. So I'm afraid that it
> is not yet supported to use vxlan
> in ml2+ovn deployment...
> 
>  https://review.opendev.org/c/openstack/neutron/+/734889

That's correct. The problem here is we should change from vxlan to Geneve during the migration. I just sent patches and will build an RPM soon.

Comment 10 Takashi Kajinami 2021-03-23 13:28:49 UTC
Thanks Jakub for submitting the fix.

If I understand the proposed fix correctly, the change makes migration process
trigger sql command to update network_type column in the networksegments table
and replaces 'vxlan' with 'geneve'.

Can we apply the same approach to the delployment where ovn migration has been completed ?

Side note:
If that approach can be applied in such deployment, I'm thinking of stopping all neutron processes
on controller nodes to avoid any possible conflict and then run sql query to apply the same change.
Maybe it would be more safe to apply the change to only one network first then apply the same
to the remaining networks if we don't see any problems.

Comment 11 Takashi Kajinami 2021-04-06 00:45:38 UTC
Hi. May I ask any feedback about my previous comment ?

Comment 26 Roman Safronov 2021-08-01 08:52:17 UTC
Verified on RHOS-16.1-RHEL-8-20210727.n.1 with python3-networking-ovn-7.3.1-1.20210714143305.4e24f4c.el8ost.noarch
Created several vxlan networks before migration to OVN, migrated to OVN, then created several new networks and verified that after migration all geneve networks (old and newly created) have unique segmentation ids. Additionally verified that there are no issues when modifying a network that was created before the migration to OVN.

Comment 37 errata-xmlrpc 2021-12-09 20:17:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3762