Bug 1987249

Summary: [OVN] Neutron sets the type of LSP with ha_router_replicated_interface device owner to empty
Product: Red Hat OpenStack Reporter: Kamil Sambor <ksambor>
Component: python-networking-ovnAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED ERRATA QA Contact: Roman Safronov <rsafrono>
Severity: urgent Docs Contact:
Priority: high    
Version: 16.1 (Train)CC: apevec, atragler, jlibosva, lhh, lmartins, majopela, pmannidi, rsafrono, scohen, spower
Target Milestone: z7Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-networking-ovn-7.3.1-1.20210714143308.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1998579 (view as bug list) Environment:
Last Closed: 2021-12-09 20:20:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kamil Sambor 2021-07-29 10:24:35 UTC 
Description of problem:

During OVN migration after the neutron agents are deleted workload VMs are unreacheable. This causes migration procedure to fail.
The issue is a regression.

See end of ovn migration log below [1]



Version:
RHOS-16.1-RHEL-8-20210727.n.1
python3-networking-ovn-7.3.1-1.20210714143305.4e24f4c.el8ost.noarch
python3-networking-ovn-migration-tool-7.3.1-1.20210714143305.4e24f4c.el8ost.noarch


Reproduced mainly on environments without dvr enabled. Happened several times already on downstream CI jobs.


How to reproduce:
1. Deploy ML2/OVS environment with DVR disabled.
2. Try to perform ML2/OVS -> ML2/OVN migration procedure according to official documentation
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/networking_with_open_virtual_network/migrating-ml2ovs-to-ovn

Actual results:
After deleting neutron resources workload VMs are unreachable. OVN migration fails and stops.

Expected results:
After deleting neutron resources workload VMs remain reachable. OVN migration proceeds and ends successfully.




Additional details:

[1]
From OVN migration log:

TASK [delete-neutron-resources : Deleting the neutron agents] ******************
task path: /home/stack/ovn_migration/playbooks/roles/delete-neutron-resources/tasks/main.yml:18
Thursday 29 July 2021  05:47:39 +0000 (0:00:00.758)       0:46:12.572 *********
changed: [localhost] => {"changed": true, "cmd": "/home/stack/ovn_migration/delete_neutron_resources/delete-neutron-resources.sh 2>&1 > /home/stack/ovn_migration/delete_neutron_resources/delete-neutron-resources.sh.log\n", "delta": "0:01:48.992275", "end": "2021-07-29 05:49:28.976546", "rc": 0, "start": "2021-07-29 05:47:39.984271", "stderr": "", "stderr_lines": [], "stdout": "+ source /home/stack/overcloudrc\n+++ set\n+++ awk '{FS=\"=\"}  /^OS_/ {print $1}'\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_AUTH_TYPE\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_AUTH_URL\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_CLOUDNAME\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_IDENTITY_API_VERSION\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_NO_CACHE\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_PASSWORD\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_PROJECT_DOMAIN_NAME\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_PROJECT_NAME\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_USERNAME\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_USER_DOMAIN_NAME\n++ export NOVA_VERSION=1.1\n++ NOVA_VERSION=1.1\n++ export COMPUTE_API_VERSION=1.1\n++ COMPUTE_API_VERSION=1.1\n++ export OS_USERNAME=admin\n++ OS_USERNAME=admin\n++ export OS_PROJECT_NAME=admin\n++ OS_PROJECT_NAME=admin\n++ export OS_USER_DOMAIN_NAME=Default\n++ OS_USER_DOMAIN_NAME=Default\n++ export OS_PROJECT_DOMAIN_NAME=Default\n++ OS_PROJECT_DOMAIN_NAME=Default\n++ export OS_NO_CACHE=True\n++ OS_NO_CACHE=True\n++ export OS_CLOUDNAME=overcloud\n++ OS_CLOUDNAME=overcloud\n++ export no_proxy=,10.0.0.133,192.168.24.31\n++ no_proxy=,10.0.0.133,192.168.24.31\n++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ export OS_AUTH_TYPE=password\n++ OS_AUTH_TYPE=password\n++ export OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao\n++ OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao\n++ export OS_AUTH_URL=http://10.0.0.133:5000\n++ OS_AUTH_URL=http://10.0.0.133:5000\n++ export OS_IDENTITY_API_VERSION=3\n++ OS_IDENTITY_API_VERSION=3\n++ export OS_COMPUTE_API_VERSION=2.latest\n++ OS_COMPUTE_API_VERSION=2.latest\n++ export OS_IMAGE_API_VERSION=2\n++ OS_IMAGE_API_VERSION=2\n++ export OS_VOLUME_API_VERSION=3\n++ OS_VOLUME_API_VERSION=3\n++ export OS_REGION_NAME=regionOne\n++ OS_REGION_NAME=regionOne\n++ '[' -z 1 ']'\n++ openstack network agent list\n++ grep neutron-\n++ grep -v ':-)'\n++ awk '{print $2}'\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 30e4fc8b-1547-4bec-9c63-688a8c53cf33\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 42ce1c84-79f7-4237-b5d7-c6235905e566\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 49c42175-539b-4352-80e9-3ab934512d4d\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 563a0753-320c-4cba-bdcc-a46f8cc7fc86\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 6c160998-a2fd-4f71-8259-a27e197ae0b5\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 77c59d4e-b6c9-41e3-ac73-b35393fa28e7\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 7dedbba7-10a8-46db-8a3d-2540179b95be\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 7fe25e79-c492-4373-96ea-e3dd6ac3f1ac\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 83406eb5-8a2a-4b8c-9185-10f0bd3c3759\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 8c4be5dd-ae3e-4009-8a0e-aff51b4fb60f\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 924bc911-3632-4d94-a19f-3e6c98852371\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete a787a75e-d612-4dd0-aa62-847bb39bba53\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete b576c9e7-2509-4c00-81a1-5be95e2c0810\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete fd520501-161f-4197-a6fe-ab3b42367bfc\n++ openstack network list\n++ grep 'HA network tenant'\n++ awk '{print $2}'\n+ for i in `openstack network list | grep \"HA network tenant\" | awk '{print $2}'`\n+ delete_network_ports 7147f79d-9eda-423c-9bad-938583ba2f70\n+ net_id=7147f79d-9eda-423c-9bad-938583ba2f70\n++ openstack port list --network 7147f79d-9eda-423c-9bad-938583ba2f70\n++ grep -v ID\n++ awk '{print $2}'\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 030d3fc5-17a0-4509-b9fc-1dffacae60d5\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 08abc9fa-0797-461b-a705-4f252b412493\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 3067c203-9432-496a-9245-b215ba7417da\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 4b78e570-0851-4d2a-a4a2-73df82ecae8f\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 6740e6d9-7cdf-4e41-b88e-8923058728f0\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 8db98596-c2a1-4b57-ae57-a4faa3132ca6\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete d31681b7-7506-4093-9fea-1788a265b4ec\n+ openstack network delete 7147f79d-9eda-423c-9bad-938583ba2f70\n+ exit 0", "stdout_lines": ["+ source /home/stack/overcloudrc", "+++ set", "+++ awk '{FS=\"=\"}  /^OS_/ {print $1}'", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_AUTH_TYPE", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_AUTH_URL", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_CLOUDNAME", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_IDENTITY_API_VERSION", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_NO_CACHE", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_PASSWORD", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_PROJECT_DOMAIN_NAME", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_PROJECT_NAME", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_USERNAME", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_USER_DOMAIN_NAME", "++ export NOVA_VERSION=1.1", "++ NOVA_VERSION=1.1", "++ export COMPUTE_API_VERSION=1.1", "++ COMPUTE_API_VERSION=1.1", "++ export OS_USERNAME=admin", "++ OS_USERNAME=admin", "++ export OS_PROJECT_NAME=admin", "++ OS_PROJECT_NAME=admin", "++ export OS_USER_DOMAIN_NAME=Default", "++ OS_USER_DOMAIN_NAME=Default", "++ export OS_PROJECT_DOMAIN_NAME=Default", "++ OS_PROJECT_DOMAIN_NAME=Default", "++ export OS_NO_CACHE=True", "++ OS_NO_CACHE=True", "++ export OS_CLOUDNAME=overcloud", "++ OS_CLOUDNAME=overcloud", "++ export no_proxy=,10.0.0.133,192.168.24.31", "++ no_proxy=,10.0.0.133,192.168.24.31", "++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ export OS_AUTH_TYPE=password", "++ OS_AUTH_TYPE=password", "++ export OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao", "++ OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao", "++ export OS_AUTH_URL=http://10.0.0.133:5000", "++ OS_AUTH_URL=http://10.0.0.133:5000", "++ export OS_IDENTITY_API_VERSION=3", "++ OS_IDENTITY_API_VERSION=3", "++ export OS_COMPUTE_API_VERSION=2.latest", "++ OS_COMPUTE_API_VERSION=2.latest", "++ export OS_IMAGE_API_VERSION=2", "++ OS_IMAGE_API_VERSION=2", "++ export OS_VOLUME_API_VERSION=3", "++ OS_VOLUME_API_VERSION=3", "++ export OS_REGION_NAME=regionOne", "++ OS_REGION_NAME=regionOne", "++ '[' -z 1 ']'", "++ openstack network agent list", "++ grep neutron-", "++ grep -v ':-)'", "++ awk '{print $2}'", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 30e4fc8b-1547-4bec-9c63-688a8c53cf33", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 42ce1c84-79f7-4237-b5d7-c6235905e566", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 49c42175-539b-4352-80e9-3ab934512d4d", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 563a0753-320c-4cba-bdcc-a46f8cc7fc86", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 6c160998-a2fd-4f71-8259-a27e197ae0b5", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 77c59d4e-b6c9-41e3-ac73-b35393fa28e7", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 7dedbba7-10a8-46db-8a3d-2540179b95be", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 7fe25e79-c492-4373-96ea-e3dd6ac3f1ac", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 83406eb5-8a2a-4b8c-9185-10f0bd3c3759", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 8c4be5dd-ae3e-4009-8a0e-aff51b4fb60f", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 924bc911-3632-4d94-a19f-3e6c98852371", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete a787a75e-d612-4dd0-aa62-847bb39bba53", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete b576c9e7-2509-4c00-81a1-5be95e2c0810", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete fd520501-161f-4197-a6fe-ab3b42367bfc", "++ openstack network list", "++ grep 'HA network tenant'", "++ awk '{print $2}'", "+ for i in `openstack network list | grep \"HA network tenant\" | awk '{print $2}'`", "+ delete_network_ports 7147f79d-9eda-423c-9bad-938583ba2f70", "+ net_id=7147f79d-9eda-423c-9bad-938583ba2f70", "++ openstack port list --network 7147f79d-9eda-423c-9bad-938583ba2f70", "++ grep -v ID", "++ awk '{print $2}'", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 030d3fc5-17a0-4509-b9fc-1dffacae60d5", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 08abc9fa-0797-461b-a705-4f252b412493", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 3067c203-9432-496a-9245-b215ba7417da", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 4b78e570-0851-4d2a-a4a2-73df82ecae8f", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 6740e6d9-7cdf-4e41-b88e-8923058728f0", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 8db98596-c2a1-4b57-ae57-a4faa3132ca6", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete d31681b7-7506-4093-9fea-1788a265b4ec", "+ openstack network delete 7147f79d-9eda-423c-9bad-938583ba2f70", "+ exit 0"]}
 
TASK [Validate pre migration resources after migration] ************************
task path: /home/stack/ovn_migration/playbooks/roles/post-migration/tasks/main.yml:7
Thursday 29 July 2021  05:49:29 +0000 (0:01:49.374)       0:48:01.946 *********
 
TASK [resources/validate : Generate resource validation script] ****************
task path: /home/stack/ovn_migration/playbooks/roles/resources/validate/tasks/main.yml:1
Thursday 29 July 2021  05:49:29 +0000 (0:00:00.093)       0:48:02.040 *********
ok: [localhost] => {"changed": false, "checksum": "fb774ecdea1401e40e315c2d3d1ca93a6bfd5f86", "dest": "/home/stack/ovn_migration/pre_migration_resources/validate-resources.sh", "gid": 1001, "group": "stack", "mode": "0744", "owner": "stack", "path": "/home/stack/ovn_migration/pre_migration_resources/validate-resources.sh", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 926, "state": "file", "uid": 1001}
 
TASK [resources/validate : Run the validation script] **************************
task path: /home/stack/ovn_migration/playbooks/roles/resources/validate/tasks/main.yml:7
Thursday 29 July 2021  05:49:29 +0000 (0:00:00.619)       0:48:02.660 *********
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /home/stack/ovn_migration/pre_migration_resources/validate-resources.sh 2>&1 | tee /home/stack/ovn_migration/pre_migration_resources/validate-resources.sh.log\n", "delta": "0:00:42.525218", "end": "2021-07-29 05:50:12.554445", "msg": "non-zero return code", "rc": 1, "start": "2021-07-29 05:49:30.029227", "stderr": "", "stderr_lines": [], "stdout": "+ set -e\n+ source /home/stack/overcloudrc\n+++ set\n+++ awk '{FS=\"=\"}  /^OS_/ {print $1}'\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_AUTH_TYPE\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_AUTH_URL\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_CLOUDNAME\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_IDENTITY_API_VERSION\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_NO_CACHE\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_PASSWORD\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_PROJECT_DOMAIN_NAME\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_PROJECT_NAME\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_USERNAME\n++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )\n++ unset OS_USER_DOMAIN_NAME\n++ export NOVA_VERSION=1.1\n++ NOVA_VERSION=1.1\n++ export COMPUTE_API_VERSION=1.1\n++ COMPUTE_API_VERSION=1.1\n++ export OS_USERNAME=admin\n++ OS_USERNAME=admin\n++ export OS_PROJECT_NAME=admin\n++ OS_PROJECT_NAME=admin\n++ export OS_USER_DOMAIN_NAME=Default\n++ OS_USER_DOMAIN_NAME=Default\n++ export OS_PROJECT_DOMAIN_NAME=Default\n++ OS_PROJECT_DOMAIN_NAME=Default\n++ export OS_NO_CACHE=True\n++ OS_NO_CACHE=True\n++ export OS_CLOUDNAME=overcloud\n++ OS_CLOUDNAME=overcloud\n++ export no_proxy=,10.0.0.133,192.168.24.31\n++ no_proxy=,10.0.0.133,192.168.24.31\n++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ export OS_AUTH_TYPE=password\n++ OS_AUTH_TYPE=password\n++ export OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao\n++ OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao\n++ export OS_AUTH_URL=http://10.0.0.133:5000\n++ OS_AUTH_URL=http://10.0.0.133:5000\n++ export OS_IDENTITY_API_VERSION=3\n++ OS_IDENTITY_API_VERSION=3\n++ export OS_COMPUTE_API_VERSION=2.latest\n++ OS_COMPUTE_API_VERSION=2.latest\n++ export OS_IMAGE_API_VERSION=2\n++ OS_IMAGE_API_VERSION=2\n++ export OS_VOLUME_API_VERSION=3\n++ OS_VOLUME_API_VERSION=3\n++ export OS_REGION_NAME=regionOne\n++ OS_REGION_NAME=regionOne\n++ '[' -z 1 ']'\n++ cat /home/stack/ovn_migration/pre_migration_resources/server_public_ip\n+ server_ip=10.0.0.187\n+ set +e\n+ ping -c 30 10.0.0.187\nPING 10.0.0.187 (10.0.0.187) 56(84) bytes of data.\nFrom 10.0.0.83 icmp_seq=9 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=10 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=11 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=12 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=13 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=14 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=15 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=16 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=17 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=18 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=19 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=20 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=21 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=22 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=23 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=24 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=25 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=26 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=27 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=28 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=29 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=30 Destination Host Unreachable\n\n--- 10.0.0.187 ping statistics ---\n30 packets transmitted, 0 received, +22 errors, 100% packet loss, time 480ms\npipe 4\n+ set -e\n+ echo 'Running ping test with -c 3 to the server ip - 10.0.0.187'\nRunning ping test with -c 3 to the server ip - 10.0.0.187\n+ ping -c 3 10.0.0.187\nPING 10.0.0.187 (10.0.0.187) 56(84) bytes of data.\nFrom 10.0.0.83 icmp_seq=1 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=2 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=3 Destination Host Unreachable\n\n--- 10.0.0.187 ping statistics ---\n3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 6ms\npipe 3", "stdout_lines": ["+ set -e", "+ source /home/stack/overcloudrc", "+++ set", "+++ awk '{FS=\"=\"}  /^OS_/ {print $1}'", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_AUTH_TYPE", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_AUTH_URL", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_CLOUDNAME", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_IDENTITY_API_VERSION", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_NO_CACHE", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_PASSWORD", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_PROJECT_DOMAIN_NAME", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_PROJECT_NAME", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_USERNAME", "++ for key in $( set | awk '{FS=\"=\"}  /^OS_/ {print $1}' )", "++ unset OS_USER_DOMAIN_NAME", "++ export NOVA_VERSION=1.1", "++ NOVA_VERSION=1.1", "++ export COMPUTE_API_VERSION=1.1", "++ COMPUTE_API_VERSION=1.1", "++ export OS_USERNAME=admin", "++ OS_USERNAME=admin", "++ export OS_PROJECT_NAME=admin", "++ OS_PROJECT_NAME=admin", "++ export OS_USER_DOMAIN_NAME=Default", "++ OS_USER_DOMAIN_NAME=Default", "++ export OS_PROJECT_DOMAIN_NAME=Default", "++ OS_PROJECT_DOMAIN_NAME=Default", "++ export OS_NO_CACHE=True", "++ OS_NO_CACHE=True", "++ export OS_CLOUDNAME=overcloud", "++ OS_CLOUDNAME=overcloud", "++ export no_proxy=,10.0.0.133,192.168.24.31", "++ no_proxy=,10.0.0.133,192.168.24.31", "++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ export OS_AUTH_TYPE=password", "++ OS_AUTH_TYPE=password", "++ export OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao", "++ OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao", "++ export OS_AUTH_URL=http://10.0.0.133:5000", "++ OS_AUTH_URL=http://10.0.0.133:5000", "++ export OS_IDENTITY_API_VERSION=3", "++ OS_IDENTITY_API_VERSION=3", "++ export OS_COMPUTE_API_VERSION=2.latest", "++ OS_COMPUTE_API_VERSION=2.latest", "++ export OS_IMAGE_API_VERSION=2", "++ OS_IMAGE_API_VERSION=2", "++ export OS_VOLUME_API_VERSION=3", "++ OS_VOLUME_API_VERSION=3", "++ export OS_REGION_NAME=regionOne", "++ OS_REGION_NAME=regionOne", "++ '[' -z 1 ']'", "++ cat /home/stack/ovn_migration/pre_migration_resources/server_public_ip", "+ server_ip=10.0.0.187", "+ set +e", "+ ping -c 30 10.0.0.187", "PING 10.0.0.187 (10.0.0.187) 56(84) bytes of data.", "From 10.0.0.83 icmp_seq=9 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=10 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=11 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=12 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=13 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=14 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=15 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=16 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=17 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=18 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=19 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=20 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=21 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=22 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=23 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=24 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=25 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=26 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=27 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=28 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=29 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=30 Destination Host Unreachable", "", "--- 10.0.0.187 ping statistics ---", "30 packets transmitted, 0 received, +22 errors, 100% packet loss, time 480ms", "pipe 4", "+ set -e", "+ echo 'Running ping test with -c 3 to the server ip - 10.0.0.187'", "Running ping test with -c 3 to the server ip - 10.0.0.187", "+ ping -c 3 10.0.0.187", "PING 10.0.0.187 (10.0.0.187) 56(84) bytes of data.", "From 10.0.0.83 icmp_seq=1 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=2 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=3 Destination Host Unreachable", "", "--- 10.0.0.187 ping statistics ---", "3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 6ms", "pipe 3"]}
 
PLAY RECAP *********************************************************************
compute-0                  : ok=20   changed=14   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0  
compute-1                  : ok=20   changed=14   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0  
controller-0               : ok=33   changed=23   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0  
controller-1               : ok=28   changed=20   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0  
controller-2               : ok=28   changed=20   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0  
localhost                  : ok=19   changed=15   unreachable=0    failed=1    skipped=1    rescued=0    ignored=0


[2]
Errors in neutron server log

2021-08-01 15:54:47.654 32 ERROR neutron.plugins.ml2.managers sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'ml2_port_bindings' expected to update 1 row(s); 0 were matched.




2021-08-01 16:33:52.577 32 ERROR neutron.plugins.ml2.managers RuntimeError: neutron-09f6ad84-23f6-4f40-bf26-c669d8a08d30 does not exist in name of Logical_Switch
2021-08-01 16:33:52.577 32 ERROR neutron.plugins.ml2.managers 
2021-08-01 16:33:52.582 32 ERROR neutron.plugins.ml2.plugin [req-bf8571ba-97dc-4437-a09b-c8c0ebd0e254 - - - - -] mechanism_manager.create_port_postcommit failed, deleting port '9d1af412-769b-4d77-8971-d916df5c6abc': neutron.plugins.ml2.common.exceptions.MechanismDriverError
Comment 2 Jakub Libosvar 2021-08-05 17:54:45 UTC 
I looked at the live env - thanks Roman for providing it. There seems to be 2 things I found:

1) The metadata port was created on br-migration which was later deleted, leaving the metadata port unplugged or the port was not re-plugged to br-int.

2) The LSP connected to router is not set as router port:
_uuid               : 4e93ef48-2c78-4bc8-9bc9-74234929c4b8
addresses           : [unknown]
dhcpv4_options      : []
dhcpv6_options      : []
dynamic_addresses   : []
enabled             : true
external_ids        : {"neutron:cidrs"="192.168.168.1/24", "neutron:device_id"="8ad182d0-a6e9-4cee-b2b9-ba083b76c75f", "neutron:device_owner"="network:ha_router_replicated_interface", "neutron:network_name"=neutron-7b1de520-d7b6-4140-b178-74d4122faed7, "neutron:port_name"="", "neutron:project_id"="4d05a9e4a662427d9c0d619fc5ef0ffd", "neutron:revision_number"="2341", "neutron:security_group_ids"=""}
ha_chassis_group    : []
name                : "247d4fe4-8721-41ce-98e8-299fcb62257c"
options             : {mcast_flood_reports="true", requested-chassis=controller-1.redhat.local}
parent_name         : []
port_security       : []
tag                 : []
tag_request         : []
type                : ""
up                  : false


The db-sync script sets the port to correct state:
2021-08-01 16:37:04.868 48 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterPortInLSwitchPortCommand(lswitch_port=247d4fe4-8721-41ce-98e8-299fcb62257c, lrouter_port=lrp-247d4fe4-8721-41ce-98e8-299fcb62257c, is_gw_port=False, if_exists=True, lsp_address=router) do_commit /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84

but somebody then misconfigures the LSP:
record 324: 2021-08-05 17:10:18.778
  table Logical_Switch_Port row "247d4fe4-8721-41ce-98e8-299fcb62257c" (4e93ef48):
    addresses=[router]
    options={router-port=lrp-247d4fe4-8721-41ce-98e8-299fcb62257c}
    type=router
  table Logical_Router_Port row "lrp-247d4fe4-8721-41ce-98e8-299fcb62257c" (d1e2e77b):
    external_ids={"neutron:network_name"=neutron-7b1de520-d7b6-4140-b178-74d4122faed7, "neutron:revision_number"="2337", "neutron:router_name"="8ad182d0-a6e9-4cee-b2b9-ba083b76c75f", "neutron:subnet_ids"="affdbfe7-5478-4088-912b-3c549b0650be"}

record 325: 2021-08-05 17:10:18.785
  table Logical_Switch_Port row "247d4fe4-8721-41ce-98e8-299fcb62257c" (4e93ef48):
    up=true

record 326: 2021-08-05 17:10:19.299
  table Logical_Switch_Port row "4ea7881d-bbe7-4415-926d-f49d0dc4adfb" (f97d8fc0):
    addresses=[router]
    options={router-port=lrp-4ea7881d-bbe7-4415-926d-f49d0dc4adfb}
    type=router
  table Logical_Router_Port row "lrp-4ea7881d-bbe7-4415-926d-f49d0dc4adfb" (8d52cebd):
    external_ids={"neutron:network_name"=neutron-a5e872d7-320a-4368-8972-023fdcf6687a, "neutron:revision_number"="2336", "neutron:router_name"="d2ca7c41-ba1e-47ad-834e-fd3c56a0c297", "neutron:subnet_ids"="8966895c-75db-4d03-8e5e-501ee1c201dd"}

record 327: 2021-08-05 17:10:19.305
  table Logical_Switch_Port row "4ea7881d-bbe7-4415-926d-f49d0dc4adfb" (f97d8fc0):
    up=true

record 328: 2021-08-05 17:10:19.391
  table Logical_Switch_Port row "247d4fe4-8721-41ce-98e8-299fcb62257c" (4e93ef48):
    addresses=[unknown]
    options={mcast_flood_reports="true", requested-chassis=controller-1.redhat.local}
    external_ids={"neutron:cidrs"="192.168.168.1/24", "neutron:device_id"="8ad182d0-a6e9-4cee-b2b9-ba083b76c75f", "neutron:device_owner"="network:ha_router_replicated_interface", "neutron:network_name"=neutron-7b1de520-d7b6-4140-b178-74d4122faed7, "neutron:port_name"="", "neutron:project_id"="4d05a9e4a662427d9c0d619fc5ef0ffd", "neutron:revision_number"="2338", "neutron:security_group_ids"=""}
    type=""

It seems to be some of the events because one of the neutron server contain transaction responsible for doing it:
2021-08-05 12:03:39.797 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLSwitchPortCommand(lport=247d4fe4-8721-41ce-98e8-299fcb62257c, columns={'external_ids': {'neutron:port_name': '', 'neutron:device_id': '8ad182d0-a6e9-4cee-b2b9-ba083b76c75f', 'neutron:project_id': '4d05a9e4a662427d9c0d619fc5ef0ffd', 'neutron:cidrs': '192.168.168.1/24', 'neutron:device_owner': 'network:ha_router_replicated_interface', 'neutron:network_name': 'neutron-7b1de520-d7b6-4140-b178-74d4122faed7', 'neutron:security_group_ids': '', 'neutron:revision_number': '2212'}, 'parent_name': [], 'tag': [], 'options': {'requested-chassis': 'controller-1.redhat.local', 'mcast_flood_reports': 'true'}, 'enabled': True, 'port_security': [], 'dhcpv4_options': [], 'dhcpv6_options': [], 'type': '', 'addresses': ['unknown'], 'ha_chassis_group': []}, if_exists=False) do_commit /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84

The culprit might be the device owner of the router port: network:ha_router_replicated_interface. I'm wondering if there was any change to the tests to use L3 HA router.
Comment 3 spower 2021-08-09 08:05:32 UTC 
This has no dev ack or pm ack and the email for the blocker porcess has not been sent. please review and if it is a blocker, follow the process to get it approved.
Comment 4 Jakub Libosvar 2021-08-10 12:50:43 UTC 
I installed previous puddle and I don't see the behavior described in comment 2. This is a regression in between the puddles.
Comment 5 Jakub Libosvar 2021-08-11 14:31:18 UTC 
I think the culprit might be this patch - https://code.engineering.redhat.com/gerrit/c/networking-ovn/+/253716/2/networking_ovn/ml2/db_migration.py lines 58-60

When I used older environment and changed the vif_details to what OVN uses, the issue re-appeared. Still investigating.
Comment 6 Jakub Libosvar 2021-08-11 15:35:12 UTC 
Disregard the previous comment. I found out that this happens when the router port is in DOWN state and db-sync/maintenance task fixes it.
Comment 7 Jakub Libosvar 2021-08-11 15:40:13 UTC 
This also happens on previous composes and it's not a regression.
Comment 8 Eran Kuris 2021-08-12 07:00:30 UTC 
(In reply to Jakub Libosvar from comment #7)
> This also happens on previous composes and it's not a regression.

Hi Kuba ,

Did you success to reproduce the issue on this puddle :  RHOS-16.1-RHEL-8-20210727.n.1 ?

can you share what is the puddle id of "previous composes" that  you mentioned?
Comment 9 Jakub Libosvar 2021-08-12 08:39:53 UTC 
(In reply to Eran Kuris from comment #8)
> (In reply to Jakub Libosvar from comment #7)
> > This also happens on previous composes and it's not a regression.
> 
> Hi Kuba ,
> 
> Did you success to reproduce the issue on this puddle : 
> RHOS-16.1-RHEL-8-20210727.n.1 ?
> 
> can you share what is the puddle id of "previous composes" that  you
> mentioned?

I reproduced on RHOS-16.1-RHEL-8-20210604.n.0 and it's been reported for puddle RHOS-16.1-RHEL-8-20210727.n.1
Comment 19 Roman Safronov 2021-10-06 11:36:04 UTC 
Was tested on RHOS-16.1-RHEL-8-20210928.n.1 
OVN migration workload VMs remain accessible after deleting neutron resources. Tested nodvr2dvr, nodvr2nodvr and dvr2dvr ovs2ovn migration scenarios.
Comment 28 errata-xmlrpc 2021-12-09 20:20:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3762