Description of problem: During OVN migration after the neutron agents are deleted workload VMs are unreacheable. This causes migration procedure to fail. The issue is a regression. See end of ovn migration log below [1] Version: RHOS-16.1-RHEL-8-20210727.n.1 python3-networking-ovn-7.3.1-1.20210714143305.4e24f4c.el8ost.noarch python3-networking-ovn-migration-tool-7.3.1-1.20210714143305.4e24f4c.el8ost.noarch Reproduced mainly on environments without dvr enabled. Happened several times already on downstream CI jobs. How to reproduce: 1. Deploy ML2/OVS environment with DVR disabled. 2. Try to perform ML2/OVS -> ML2/OVN migration procedure according to official documentation https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/networking_with_open_virtual_network/migrating-ml2ovs-to-ovn Actual results: After deleting neutron resources workload VMs are unreachable. OVN migration fails and stops. Expected results: After deleting neutron resources workload VMs remain reachable. OVN migration proceeds and ends successfully. Additional details: [1] From OVN migration log: TASK [delete-neutron-resources : Deleting the neutron agents] ****************** task path: /home/stack/ovn_migration/playbooks/roles/delete-neutron-resources/tasks/main.yml:18 Thursday 29 July 2021 05:47:39 +0000 (0:00:00.758) 0:46:12.572 ********* changed: [localhost] => {"changed": true, "cmd": "/home/stack/ovn_migration/delete_neutron_resources/delete-neutron-resources.sh 2>&1 > /home/stack/ovn_migration/delete_neutron_resources/delete-neutron-resources.sh.log\n", "delta": "0:01:48.992275", "end": "2021-07-29 05:49:28.976546", "rc": 0, "start": "2021-07-29 05:47:39.984271", "stderr": "", "stderr_lines": [], "stdout": "+ source /home/stack/overcloudrc\n+++ set\n+++ awk '{FS=\"=\"} /^OS_/ {print $1}'\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_AUTH_TYPE\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_AUTH_URL\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_CLOUDNAME\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_IDENTITY_API_VERSION\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_NO_CACHE\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_PASSWORD\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_PROJECT_DOMAIN_NAME\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_PROJECT_NAME\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_USERNAME\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_USER_DOMAIN_NAME\n++ export NOVA_VERSION=1.1\n++ NOVA_VERSION=1.1\n++ export COMPUTE_API_VERSION=1.1\n++ COMPUTE_API_VERSION=1.1\n++ export OS_USERNAME=admin\n++ OS_USERNAME=admin\n++ export OS_PROJECT_NAME=admin\n++ OS_PROJECT_NAME=admin\n++ export OS_USER_DOMAIN_NAME=Default\n++ OS_USER_DOMAIN_NAME=Default\n++ export OS_PROJECT_DOMAIN_NAME=Default\n++ OS_PROJECT_DOMAIN_NAME=Default\n++ export OS_NO_CACHE=True\n++ OS_NO_CACHE=True\n++ export OS_CLOUDNAME=overcloud\n++ OS_CLOUDNAME=overcloud\n++ export no_proxy=,10.0.0.133,192.168.24.31\n++ no_proxy=,10.0.0.133,192.168.24.31\n++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ export OS_AUTH_TYPE=password\n++ OS_AUTH_TYPE=password\n++ export OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao\n++ OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao\n++ export OS_AUTH_URL=http://10.0.0.133:5000\n++ OS_AUTH_URL=http://10.0.0.133:5000\n++ export OS_IDENTITY_API_VERSION=3\n++ OS_IDENTITY_API_VERSION=3\n++ export OS_COMPUTE_API_VERSION=2.latest\n++ OS_COMPUTE_API_VERSION=2.latest\n++ export OS_IMAGE_API_VERSION=2\n++ OS_IMAGE_API_VERSION=2\n++ export OS_VOLUME_API_VERSION=3\n++ OS_VOLUME_API_VERSION=3\n++ export OS_REGION_NAME=regionOne\n++ OS_REGION_NAME=regionOne\n++ '[' -z 1 ']'\n++ openstack network agent list\n++ grep neutron-\n++ grep -v ':-)'\n++ awk '{print $2}'\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 30e4fc8b-1547-4bec-9c63-688a8c53cf33\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 42ce1c84-79f7-4237-b5d7-c6235905e566\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 49c42175-539b-4352-80e9-3ab934512d4d\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 563a0753-320c-4cba-bdcc-a46f8cc7fc86\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 6c160998-a2fd-4f71-8259-a27e197ae0b5\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 77c59d4e-b6c9-41e3-ac73-b35393fa28e7\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 7dedbba7-10a8-46db-8a3d-2540179b95be\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 7fe25e79-c492-4373-96ea-e3dd6ac3f1ac\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 83406eb5-8a2a-4b8c-9185-10f0bd3c3759\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 8c4be5dd-ae3e-4009-8a0e-aff51b4fb60f\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete 924bc911-3632-4d94-a19f-3e6c98852371\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete a787a75e-d612-4dd0-aa62-847bb39bba53\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete b576c9e7-2509-4c00-81a1-5be95e2c0810\n+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`\n+ openstack network agent delete fd520501-161f-4197-a6fe-ab3b42367bfc\n++ openstack network list\n++ grep 'HA network tenant'\n++ awk '{print $2}'\n+ for i in `openstack network list | grep \"HA network tenant\" | awk '{print $2}'`\n+ delete_network_ports 7147f79d-9eda-423c-9bad-938583ba2f70\n+ net_id=7147f79d-9eda-423c-9bad-938583ba2f70\n++ openstack port list --network 7147f79d-9eda-423c-9bad-938583ba2f70\n++ grep -v ID\n++ awk '{print $2}'\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 030d3fc5-17a0-4509-b9fc-1dffacae60d5\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 08abc9fa-0797-461b-a705-4f252b412493\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 3067c203-9432-496a-9245-b215ba7417da\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 4b78e570-0851-4d2a-a4a2-73df82ecae8f\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 6740e6d9-7cdf-4e41-b88e-8923058728f0\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete 8db98596-c2a1-4b57-ae57-a4faa3132ca6\n+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`\n+ openstack port delete d31681b7-7506-4093-9fea-1788a265b4ec\n+ openstack network delete 7147f79d-9eda-423c-9bad-938583ba2f70\n+ exit 0", "stdout_lines": ["+ source /home/stack/overcloudrc", "+++ set", "+++ awk '{FS=\"=\"} /^OS_/ {print $1}'", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_AUTH_TYPE", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_AUTH_URL", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_CLOUDNAME", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_IDENTITY_API_VERSION", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_NO_CACHE", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_PASSWORD", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_PROJECT_DOMAIN_NAME", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_PROJECT_NAME", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_USERNAME", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_USER_DOMAIN_NAME", "++ export NOVA_VERSION=1.1", "++ NOVA_VERSION=1.1", "++ export COMPUTE_API_VERSION=1.1", "++ COMPUTE_API_VERSION=1.1", "++ export OS_USERNAME=admin", "++ OS_USERNAME=admin", "++ export OS_PROJECT_NAME=admin", "++ OS_PROJECT_NAME=admin", "++ export OS_USER_DOMAIN_NAME=Default", "++ OS_USER_DOMAIN_NAME=Default", "++ export OS_PROJECT_DOMAIN_NAME=Default", "++ OS_PROJECT_DOMAIN_NAME=Default", "++ export OS_NO_CACHE=True", "++ OS_NO_CACHE=True", "++ export OS_CLOUDNAME=overcloud", "++ OS_CLOUDNAME=overcloud", "++ export no_proxy=,10.0.0.133,192.168.24.31", "++ no_proxy=,10.0.0.133,192.168.24.31", "++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ export OS_AUTH_TYPE=password", "++ OS_AUTH_TYPE=password", "++ export OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao", "++ OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao", "++ export OS_AUTH_URL=http://10.0.0.133:5000", "++ OS_AUTH_URL=http://10.0.0.133:5000", "++ export OS_IDENTITY_API_VERSION=3", "++ OS_IDENTITY_API_VERSION=3", "++ export OS_COMPUTE_API_VERSION=2.latest", "++ OS_COMPUTE_API_VERSION=2.latest", "++ export OS_IMAGE_API_VERSION=2", "++ OS_IMAGE_API_VERSION=2", "++ export OS_VOLUME_API_VERSION=3", "++ OS_VOLUME_API_VERSION=3", "++ export OS_REGION_NAME=regionOne", "++ OS_REGION_NAME=regionOne", "++ '[' -z 1 ']'", "++ openstack network agent list", "++ grep neutron-", "++ grep -v ':-)'", "++ awk '{print $2}'", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 30e4fc8b-1547-4bec-9c63-688a8c53cf33", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 42ce1c84-79f7-4237-b5d7-c6235905e566", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 49c42175-539b-4352-80e9-3ab934512d4d", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 563a0753-320c-4cba-bdcc-a46f8cc7fc86", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 6c160998-a2fd-4f71-8259-a27e197ae0b5", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 77c59d4e-b6c9-41e3-ac73-b35393fa28e7", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 7dedbba7-10a8-46db-8a3d-2540179b95be", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 7fe25e79-c492-4373-96ea-e3dd6ac3f1ac", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 83406eb5-8a2a-4b8c-9185-10f0bd3c3759", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 8c4be5dd-ae3e-4009-8a0e-aff51b4fb60f", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete 924bc911-3632-4d94-a19f-3e6c98852371", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete a787a75e-d612-4dd0-aa62-847bb39bba53", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete b576c9e7-2509-4c00-81a1-5be95e2c0810", "+ for i in `openstack network agent list | grep neutron- | grep -v ':-)' | awk {'print $2'}`", "+ openstack network agent delete fd520501-161f-4197-a6fe-ab3b42367bfc", "++ openstack network list", "++ grep 'HA network tenant'", "++ awk '{print $2}'", "+ for i in `openstack network list | grep \"HA network tenant\" | awk '{print $2}'`", "+ delete_network_ports 7147f79d-9eda-423c-9bad-938583ba2f70", "+ net_id=7147f79d-9eda-423c-9bad-938583ba2f70", "++ openstack port list --network 7147f79d-9eda-423c-9bad-938583ba2f70", "++ grep -v ID", "++ awk '{print $2}'", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 030d3fc5-17a0-4509-b9fc-1dffacae60d5", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 08abc9fa-0797-461b-a705-4f252b412493", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 3067c203-9432-496a-9245-b215ba7417da", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 4b78e570-0851-4d2a-a4a2-73df82ecae8f", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 6740e6d9-7cdf-4e41-b88e-8923058728f0", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete 8db98596-c2a1-4b57-ae57-a4faa3132ca6", "+ for p in `openstack port list --network $net_id | grep -v ID | awk '{print $2}'`", "+ openstack port delete d31681b7-7506-4093-9fea-1788a265b4ec", "+ openstack network delete 7147f79d-9eda-423c-9bad-938583ba2f70", "+ exit 0"]} TASK [Validate pre migration resources after migration] ************************ task path: /home/stack/ovn_migration/playbooks/roles/post-migration/tasks/main.yml:7 Thursday 29 July 2021 05:49:29 +0000 (0:01:49.374) 0:48:01.946 ********* TASK [resources/validate : Generate resource validation script] **************** task path: /home/stack/ovn_migration/playbooks/roles/resources/validate/tasks/main.yml:1 Thursday 29 July 2021 05:49:29 +0000 (0:00:00.093) 0:48:02.040 ********* ok: [localhost] => {"changed": false, "checksum": "fb774ecdea1401e40e315c2d3d1ca93a6bfd5f86", "dest": "/home/stack/ovn_migration/pre_migration_resources/validate-resources.sh", "gid": 1001, "group": "stack", "mode": "0744", "owner": "stack", "path": "/home/stack/ovn_migration/pre_migration_resources/validate-resources.sh", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 926, "state": "file", "uid": 1001} TASK [resources/validate : Run the validation script] ************************** task path: /home/stack/ovn_migration/playbooks/roles/resources/validate/tasks/main.yml:7 Thursday 29 July 2021 05:49:29 +0000 (0:00:00.619) 0:48:02.660 ********* fatal: [localhost]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /home/stack/ovn_migration/pre_migration_resources/validate-resources.sh 2>&1 | tee /home/stack/ovn_migration/pre_migration_resources/validate-resources.sh.log\n", "delta": "0:00:42.525218", "end": "2021-07-29 05:50:12.554445", "msg": "non-zero return code", "rc": 1, "start": "2021-07-29 05:49:30.029227", "stderr": "", "stderr_lines": [], "stdout": "+ set -e\n+ source /home/stack/overcloudrc\n+++ set\n+++ awk '{FS=\"=\"} /^OS_/ {print $1}'\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_AUTH_TYPE\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_AUTH_URL\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_CLOUDNAME\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_IDENTITY_API_VERSION\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_NO_CACHE\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_PASSWORD\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_PROJECT_DOMAIN_NAME\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_PROJECT_NAME\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_USERNAME\n++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )\n++ unset OS_USER_DOMAIN_NAME\n++ export NOVA_VERSION=1.1\n++ NOVA_VERSION=1.1\n++ export COMPUTE_API_VERSION=1.1\n++ COMPUTE_API_VERSION=1.1\n++ export OS_USERNAME=admin\n++ OS_USERNAME=admin\n++ export OS_PROJECT_NAME=admin\n++ OS_PROJECT_NAME=admin\n++ export OS_USER_DOMAIN_NAME=Default\n++ OS_USER_DOMAIN_NAME=Default\n++ export OS_PROJECT_DOMAIN_NAME=Default\n++ OS_PROJECT_DOMAIN_NAME=Default\n++ export OS_NO_CACHE=True\n++ OS_NO_CACHE=True\n++ export OS_CLOUDNAME=overcloud\n++ OS_CLOUDNAME=overcloud\n++ export no_proxy=,10.0.0.133,192.168.24.31\n++ no_proxy=,10.0.0.133,192.168.24.31\n++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ export OS_AUTH_TYPE=password\n++ OS_AUTH_TYPE=password\n++ export OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao\n++ OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao\n++ export OS_AUTH_URL=http://10.0.0.133:5000\n++ OS_AUTH_URL=http://10.0.0.133:5000\n++ export OS_IDENTITY_API_VERSION=3\n++ OS_IDENTITY_API_VERSION=3\n++ export OS_COMPUTE_API_VERSION=2.latest\n++ OS_COMPUTE_API_VERSION=2.latest\n++ export OS_IMAGE_API_VERSION=2\n++ OS_IMAGE_API_VERSION=2\n++ export OS_VOLUME_API_VERSION=3\n++ OS_VOLUME_API_VERSION=3\n++ export OS_REGION_NAME=regionOne\n++ OS_REGION_NAME=regionOne\n++ '[' -z 1 ']'\n++ cat /home/stack/ovn_migration/pre_migration_resources/server_public_ip\n+ server_ip=10.0.0.187\n+ set +e\n+ ping -c 30 10.0.0.187\nPING 10.0.0.187 (10.0.0.187) 56(84) bytes of data.\nFrom 10.0.0.83 icmp_seq=9 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=10 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=11 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=12 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=13 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=14 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=15 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=16 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=17 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=18 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=19 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=20 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=21 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=22 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=23 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=24 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=25 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=26 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=27 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=28 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=29 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=30 Destination Host Unreachable\n\n--- 10.0.0.187 ping statistics ---\n30 packets transmitted, 0 received, +22 errors, 100% packet loss, time 480ms\npipe 4\n+ set -e\n+ echo 'Running ping test with -c 3 to the server ip - 10.0.0.187'\nRunning ping test with -c 3 to the server ip - 10.0.0.187\n+ ping -c 3 10.0.0.187\nPING 10.0.0.187 (10.0.0.187) 56(84) bytes of data.\nFrom 10.0.0.83 icmp_seq=1 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=2 Destination Host Unreachable\nFrom 10.0.0.83 icmp_seq=3 Destination Host Unreachable\n\n--- 10.0.0.187 ping statistics ---\n3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 6ms\npipe 3", "stdout_lines": ["+ set -e", "+ source /home/stack/overcloudrc", "+++ set", "+++ awk '{FS=\"=\"} /^OS_/ {print $1}'", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_AUTH_TYPE", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_AUTH_URL", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_CLOUDNAME", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_IDENTITY_API_VERSION", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_NO_CACHE", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_PASSWORD", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_PROJECT_DOMAIN_NAME", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_PROJECT_NAME", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_USERNAME", "++ for key in $( set | awk '{FS=\"=\"} /^OS_/ {print $1}' )", "++ unset OS_USER_DOMAIN_NAME", "++ export NOVA_VERSION=1.1", "++ NOVA_VERSION=1.1", "++ export COMPUTE_API_VERSION=1.1", "++ COMPUTE_API_VERSION=1.1", "++ export OS_USERNAME=admin", "++ OS_USERNAME=admin", "++ export OS_PROJECT_NAME=admin", "++ OS_PROJECT_NAME=admin", "++ export OS_USER_DOMAIN_NAME=Default", "++ OS_USER_DOMAIN_NAME=Default", "++ export OS_PROJECT_DOMAIN_NAME=Default", "++ OS_PROJECT_DOMAIN_NAME=Default", "++ export OS_NO_CACHE=True", "++ OS_NO_CACHE=True", "++ export OS_CLOUDNAME=overcloud", "++ OS_CLOUDNAME=overcloud", "++ export no_proxy=,10.0.0.133,192.168.24.31", "++ no_proxy=,10.0.0.133,192.168.24.31", "++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ export OS_AUTH_TYPE=password", "++ OS_AUTH_TYPE=password", "++ export OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao", "++ OS_PASSWORD=84GHLIaS07Rq9xighEV2Vs7Ao", "++ export OS_AUTH_URL=http://10.0.0.133:5000", "++ OS_AUTH_URL=http://10.0.0.133:5000", "++ export OS_IDENTITY_API_VERSION=3", "++ OS_IDENTITY_API_VERSION=3", "++ export OS_COMPUTE_API_VERSION=2.latest", "++ OS_COMPUTE_API_VERSION=2.latest", "++ export OS_IMAGE_API_VERSION=2", "++ OS_IMAGE_API_VERSION=2", "++ export OS_VOLUME_API_VERSION=3", "++ OS_VOLUME_API_VERSION=3", "++ export OS_REGION_NAME=regionOne", "++ OS_REGION_NAME=regionOne", "++ '[' -z 1 ']'", "++ cat /home/stack/ovn_migration/pre_migration_resources/server_public_ip", "+ server_ip=10.0.0.187", "+ set +e", "+ ping -c 30 10.0.0.187", "PING 10.0.0.187 (10.0.0.187) 56(84) bytes of data.", "From 10.0.0.83 icmp_seq=9 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=10 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=11 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=12 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=13 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=14 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=15 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=16 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=17 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=18 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=19 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=20 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=21 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=22 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=23 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=24 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=25 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=26 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=27 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=28 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=29 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=30 Destination Host Unreachable", "", "--- 10.0.0.187 ping statistics ---", "30 packets transmitted, 0 received, +22 errors, 100% packet loss, time 480ms", "pipe 4", "+ set -e", "+ echo 'Running ping test with -c 3 to the server ip - 10.0.0.187'", "Running ping test with -c 3 to the server ip - 10.0.0.187", "+ ping -c 3 10.0.0.187", "PING 10.0.0.187 (10.0.0.187) 56(84) bytes of data.", "From 10.0.0.83 icmp_seq=1 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=2 Destination Host Unreachable", "From 10.0.0.83 icmp_seq=3 Destination Host Unreachable", "", "--- 10.0.0.187 ping statistics ---", "3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 6ms", "pipe 3"]} PLAY RECAP ********************************************************************* compute-0 : ok=20 changed=14 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 compute-1 : ok=20 changed=14 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 controller-0 : ok=33 changed=23 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 controller-1 : ok=28 changed=20 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 controller-2 : ok=28 changed=20 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 localhost : ok=19 changed=15 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 [2] Errors in neutron server log 2021-08-01 15:54:47.654 32 ERROR neutron.plugins.ml2.managers sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'ml2_port_bindings' expected to update 1 row(s); 0 were matched. 2021-08-01 16:33:52.577 32 ERROR neutron.plugins.ml2.managers RuntimeError: neutron-09f6ad84-23f6-4f40-bf26-c669d8a08d30 does not exist in name of Logical_Switch 2021-08-01 16:33:52.577 32 ERROR neutron.plugins.ml2.managers 2021-08-01 16:33:52.582 32 ERROR neutron.plugins.ml2.plugin [req-bf8571ba-97dc-4437-a09b-c8c0ebd0e254 - - - - -] mechanism_manager.create_port_postcommit failed, deleting port '9d1af412-769b-4d77-8971-d916df5c6abc': neutron.plugins.ml2.common.exceptions.MechanismDriverError
I looked at the live env - thanks Roman for providing it. There seems to be 2 things I found: 1) The metadata port was created on br-migration which was later deleted, leaving the metadata port unplugged or the port was not re-plugged to br-int. 2) The LSP connected to router is not set as router port: _uuid : 4e93ef48-2c78-4bc8-9bc9-74234929c4b8 addresses : [unknown] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"="192.168.168.1/24", "neutron:device_id"="8ad182d0-a6e9-4cee-b2b9-ba083b76c75f", "neutron:device_owner"="network:ha_router_replicated_interface", "neutron:network_name"=neutron-7b1de520-d7b6-4140-b178-74d4122faed7, "neutron:port_name"="", "neutron:project_id"="4d05a9e4a662427d9c0d619fc5ef0ffd", "neutron:revision_number"="2341", "neutron:security_group_ids"=""} ha_chassis_group : [] name : "247d4fe4-8721-41ce-98e8-299fcb62257c" options : {mcast_flood_reports="true", requested-chassis=controller-1.redhat.local} parent_name : [] port_security : [] tag : [] tag_request : [] type : "" up : false The db-sync script sets the port to correct state: 2021-08-01 16:37:04.868 48 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterPortInLSwitchPortCommand(lswitch_port=247d4fe4-8721-41ce-98e8-299fcb62257c, lrouter_port=lrp-247d4fe4-8721-41ce-98e8-299fcb62257c, is_gw_port=False, if_exists=True, lsp_address=router) do_commit /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 but somebody then misconfigures the LSP: record 324: 2021-08-05 17:10:18.778 table Logical_Switch_Port row "247d4fe4-8721-41ce-98e8-299fcb62257c" (4e93ef48): addresses=[router] options={router-port=lrp-247d4fe4-8721-41ce-98e8-299fcb62257c} type=router table Logical_Router_Port row "lrp-247d4fe4-8721-41ce-98e8-299fcb62257c" (d1e2e77b): external_ids={"neutron:network_name"=neutron-7b1de520-d7b6-4140-b178-74d4122faed7, "neutron:revision_number"="2337", "neutron:router_name"="8ad182d0-a6e9-4cee-b2b9-ba083b76c75f", "neutron:subnet_ids"="affdbfe7-5478-4088-912b-3c549b0650be"} record 325: 2021-08-05 17:10:18.785 table Logical_Switch_Port row "247d4fe4-8721-41ce-98e8-299fcb62257c" (4e93ef48): up=true record 326: 2021-08-05 17:10:19.299 table Logical_Switch_Port row "4ea7881d-bbe7-4415-926d-f49d0dc4adfb" (f97d8fc0): addresses=[router] options={router-port=lrp-4ea7881d-bbe7-4415-926d-f49d0dc4adfb} type=router table Logical_Router_Port row "lrp-4ea7881d-bbe7-4415-926d-f49d0dc4adfb" (8d52cebd): external_ids={"neutron:network_name"=neutron-a5e872d7-320a-4368-8972-023fdcf6687a, "neutron:revision_number"="2336", "neutron:router_name"="d2ca7c41-ba1e-47ad-834e-fd3c56a0c297", "neutron:subnet_ids"="8966895c-75db-4d03-8e5e-501ee1c201dd"} record 327: 2021-08-05 17:10:19.305 table Logical_Switch_Port row "4ea7881d-bbe7-4415-926d-f49d0dc4adfb" (f97d8fc0): up=true record 328: 2021-08-05 17:10:19.391 table Logical_Switch_Port row "247d4fe4-8721-41ce-98e8-299fcb62257c" (4e93ef48): addresses=[unknown] options={mcast_flood_reports="true", requested-chassis=controller-1.redhat.local} external_ids={"neutron:cidrs"="192.168.168.1/24", "neutron:device_id"="8ad182d0-a6e9-4cee-b2b9-ba083b76c75f", "neutron:device_owner"="network:ha_router_replicated_interface", "neutron:network_name"=neutron-7b1de520-d7b6-4140-b178-74d4122faed7, "neutron:port_name"="", "neutron:project_id"="4d05a9e4a662427d9c0d619fc5ef0ffd", "neutron:revision_number"="2338", "neutron:security_group_ids"=""} type="" It seems to be some of the events because one of the neutron server contain transaction responsible for doing it: 2021-08-05 12:03:39.797 27 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLSwitchPortCommand(lport=247d4fe4-8721-41ce-98e8-299fcb62257c, columns={'external_ids': {'neutron:port_name': '', 'neutron:device_id': '8ad182d0-a6e9-4cee-b2b9-ba083b76c75f', 'neutron:project_id': '4d05a9e4a662427d9c0d619fc5ef0ffd', 'neutron:cidrs': '192.168.168.1/24', 'neutron:device_owner': 'network:ha_router_replicated_interface', 'neutron:network_name': 'neutron-7b1de520-d7b6-4140-b178-74d4122faed7', 'neutron:security_group_ids': '', 'neutron:revision_number': '2212'}, 'parent_name': [], 'tag': [], 'options': {'requested-chassis': 'controller-1.redhat.local', 'mcast_flood_reports': 'true'}, 'enabled': True, 'port_security': [], 'dhcpv4_options': [], 'dhcpv6_options': [], 'type': '', 'addresses': ['unknown'], 'ha_chassis_group': []}, if_exists=False) do_commit /usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 The culprit might be the device owner of the router port: network:ha_router_replicated_interface. I'm wondering if there was any change to the tests to use L3 HA router.
This has no dev ack or pm ack and the email for the blocker porcess has not been sent. please review and if it is a blocker, follow the process to get it approved.
I installed previous puddle and I don't see the behavior described in comment 2. This is a regression in between the puddles.
I think the culprit might be this patch - https://code.engineering.redhat.com/gerrit/c/networking-ovn/+/253716/2/networking_ovn/ml2/db_migration.py lines 58-60 When I used older environment and changed the vif_details to what OVN uses, the issue re-appeared. Still investigating.
Disregard the previous comment. I found out that this happens when the router port is in DOWN state and db-sync/maintenance task fixes it.
This also happens on previous composes and it's not a regression.
(In reply to Jakub Libosvar from comment #7) > This also happens on previous composes and it's not a regression. Hi Kuba , Did you success to reproduce the issue on this puddle : RHOS-16.1-RHEL-8-20210727.n.1 ? can you share what is the puddle id of "previous composes" that you mentioned?
(In reply to Eran Kuris from comment #8) > (In reply to Jakub Libosvar from comment #7) > > This also happens on previous composes and it's not a regression. > > Hi Kuba , > > Did you success to reproduce the issue on this puddle : > RHOS-16.1-RHEL-8-20210727.n.1 ? > > can you share what is the puddle id of "previous composes" that you > mentioned? I reproduced on RHOS-16.1-RHEL-8-20210604.n.0 and it's been reported for puddle RHOS-16.1-RHEL-8-20210727.n.1
Was tested on RHOS-16.1-RHEL-8-20210928.n.1 OVN migration workload VMs remain accessible after deleting neutron resources. Tested nodvr2dvr, nodvr2nodvr and dvr2dvr ovs2ovn migration scenarios.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3762