Description of problem:
OVS-specific interfaces still present on compute nodes after migrating to OVN

As can be see in the logs there were conflicts during ports deletion, see the attached log below [2]

Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20240329.n.1

How reproducible:
100%

Steps to Reproduce:
1. Deploy HA environment (in my case it had 3 controllers and 4 compute nodes)
2. Create router, networks, security groups, VMs
3. Perform migration from OVS to OVN in batches, see batches configs below [1]
i.e. first run ovn-migration script with install-ovn, then activate-ovn for each batch
4. After activating ovn on all batches run ovn_migration script with cleanup-ovs parameter

Actual results:
OVS interfaces were not removed since there were conflicts because some ports were in use, see log below [2]

Expected results:
There are no conflicts, OVS interfaces removed correctly

Additional info:

[1] batches config 
[batch1]
compute-0
compute-1

[batch2]
compute-2
compute-3

[batch3]
controller-0
controller-1
controller-2


[2] from cleanup-ovs.sh.log

PLAY [cleanup Neutron ml2/OVS resources] ***************************************

TASK [delete-neutron-resources : Perform cleanup] ******************************
task path: /usr/share/ansible/neutron-ovn-migration/playbooks/roles/delete-neutron-resources/tasks/main.yml:2
Thursday 04 April 2024  09:34:27 +0000 (0:00:02.241)       0:00:40.025 ******** 
changed: [localhost] => {"changed": true, "cmd": "set -x\n\nsource /home/stack/overcloudrc\n\ntmpfile=$(mktemp)\n\n# Wait for Neutron API\nwhile ! openstack network agent list > $tmpfile; do\n    sleep 1\ndone\n\n# Delete non alive neutron agents in a batch of 300\ngrep neutron- $tmpfile | grep -v 'ovn' | awk {'print $2'} | xargs -n300 openstack network agent delete $agent\n\n# Dump all HA networks to a file\nopenstack network list -c ID -c Name -c Subnets | grep \"HA network tenant\" | tr -d '|' > $tmpfile\n\n# Dump all ports to a file\nports_tmpfile=$(mktemp)\nopenstack port list > $ports_tmpfile\n\n# HA network has always only one subnet so we can use that\nports_to_delete=\"\"\nfor subnet_id in $(awk '{ print $3 }' $tmpfile); do\n        ports_to_delete=\"$ports_to_delete $(awk \"/subnet_id='$subnet_id'/{ print \\$2 }\" $ports_tmpfile)\"\ndone\necho $ports_to_delete | xargs -n300 openstack port delete\n\n# Delete all HA networks\nawk '{ print $1 }' $tmpfile | xargs -n300 openstack network delete\n\n# Delete DVR gateway ports\nawk '/network:floatingip_agent_gateway/{ print $2 }' $ports_tmpfile | xargs -n300 openstack port delete || :\n", "delta": "0:00:30.335600", "end": "2024-04-04 09:34:58.080027", "msg": "", "rc": 0, "start": "2024-04-04 09:34:27.744427", "stderr": "+ source /home/stack/overcloudrc\n+++ set\n+++ awk '{FS=\"=\"}  /^OS_/ {print $1}'\n++ export OS_USERNAME=admin\n++ OS_USERNAME=admin\n++ export OS_PROJECT_NAME=admin\n++ OS_PROJECT_NAME=admin\n++ export OS_USER_DOMAIN_NAME=Default\n++ OS_USER_DOMAIN_NAME=Default\n++ export OS_PROJECT_DOMAIN_NAME=Default\n++ OS_PROJECT_DOMAIN_NAME=Default\n++ export OS_NO_CACHE=True\n++ OS_NO_CACHE=True\n++ export OS_CLOUD=overcloud\n++ OS_CLOUD=overcloud\n++ export no_proxy=,192.168.24.33,overcloud.redhat.local\n++ no_proxy=,192.168.24.33,overcloud.redhat.local\n++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'\n++ export OS_AUTH_TYPE=password\n++ OS_AUTH_TYPE=password\n++ export OS_PASSWORD=0HpSfT9fLCH9MiBteYgMYgwZ0\n++ OS_PASSWORD=0HpSfT9fLCH9MiBteYgMYgwZ0\n++ export OS_AUTH_URL=https://overcloud.redhat.local:13000\n++ OS_AUTH_URL=https://overcloud.redhat.local:13000\n++ export OS_IDENTITY_API_VERSION=3\n++ OS_IDENTITY_API_VERSION=3\n++ export OS_COMPUTE_API_VERSION=2.1\n++ OS_COMPUTE_API_VERSION=2.1\n++ export OS_IMAGE_API_VERSION=2\n++ OS_IMAGE_API_VERSION=2\n++ export OS_VOLUME_API_VERSION=3\n++ OS_VOLUME_API_VERSION=3\n++ export OS_REGION_NAME=regionOne\n++ OS_REGION_NAME=regionOne\n++ '[' -z '' ']'\n++ export PS1=\n++ PS1=\n++ export 'PS1=${OS_CLOUD:+($OS_CLOUD)} '\n++ PS1='${OS_CLOUD:+($OS_CLOUD)} '\n++ export CLOUDPROMPT_ENABLED=1\n++ CLOUDPROMPT_ENABLED=1\n++ mktemp\n+ tmpfile=/tmp/tmp.313fnmlHOh\n+ openstack network agent list\n/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!\n  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\n+ grep neutron- /tmp/tmp.313fnmlHOh\n+ grep -v ovn\n+ awk '{print $2}'\n+ xargs -n300 openstack network agent delete\n/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!\n  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\n+ openstack network list -c ID -c Name -c Subnets\n+ grep 'HA network tenant'\n+ tr -d '|'\n/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!\n  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\n++ mktemp\n+ ports_tmpfile=/tmp/tmp.O0tkyYtbL2\n+ openstack port list\n/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!\n  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\n+ ports_to_delete=\n++ awk '{ print $3 }' /tmp/tmp.313fnmlHOh\n+ for subnet_id in $(awk '{ print $3 }' $tmpfile)\n++ awk '/subnet_id='\\''network'\\''/{ print $2 }' /tmp/tmp.O0tkyYtbL2\n+ ports_to_delete=' '\n+ echo\n+ xargs -n300 openstack port delete\n/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!\n  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\nusage: openstack port delete [-h] <port> [<port> ...]\nopenstack port delete: error: the following arguments are required: <port>\n\n+ awk '{ print $1 }' /tmp/tmp.313fnmlHOh\n+ xargs -n300 openstack network delete\n/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!\n  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\nFailed to delete network with name or ID '107a0ed2-58b7-4bd4-ad6c-7c1555b7accd': ConflictException: 409: Client Error for url: https://overcloud.redhat.local:13696/v2.0/networks/107a0ed2-58b7-4bd4-ad6c-7c1555b7accd, Unable to complete operation on network 107a0ed2-58b7-4bd4-ad6c-7c1555b7accd. There are one or more ports still in use on the network.\n1 of 1 networks failed to delete.\n+ awk '/network:floatingip_agent_gateway/{ print $2 }' /tmp/tmp.O0tkyYtbL2\n+ xargs -n300 openstack port delete\n/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!\n  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\nusage: openstack port delete [-h] <port> [<port> ...]\nopenstack port delete: error: the following arguments are required: <port>\n\n+ :", "stderr_lines": ["+ source /home/stack/overcloudrc", "+++ set", "+++ awk '{FS=\"=\"}  /^OS_/ {print $1}'", "++ export OS_USERNAME=admin", "++ OS_USERNAME=admin", "++ export OS_PROJECT_NAME=admin", "++ OS_PROJECT_NAME=admin", "++ export OS_USER_DOMAIN_NAME=Default", "++ OS_USER_DOMAIN_NAME=Default", "++ export OS_PROJECT_DOMAIN_NAME=Default", "++ OS_PROJECT_DOMAIN_NAME=Default", "++ export OS_NO_CACHE=True", "++ OS_NO_CACHE=True", "++ export OS_CLOUD=overcloud", "++ OS_CLOUD=overcloud", "++ export no_proxy=,192.168.24.33,overcloud.redhat.local", "++ no_proxy=,192.168.24.33,overcloud.redhat.local", "++ export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'", "++ export OS_AUTH_TYPE=password", "++ OS_AUTH_TYPE=password", "++ export OS_PASSWORD=0HpSfT9fLCH9MiBteYgMYgwZ0", "++ OS_PASSWORD=0HpSfT9fLCH9MiBteYgMYgwZ0", "++ export OS_AUTH_URL=https://overcloud.redhat.local:13000", "++ OS_AUTH_URL=https://overcloud.redhat.local:13000", "++ export OS_IDENTITY_API_VERSION=3", "++ OS_IDENTITY_API_VERSION=3", "++ export OS_COMPUTE_API_VERSION=2.1", "++ OS_COMPUTE_API_VERSION=2.1", "++ export OS_IMAGE_API_VERSION=2", "++ OS_IMAGE_API_VERSION=2", "++ export OS_VOLUME_API_VERSION=3", "++ OS_VOLUME_API_VERSION=3", "++ export OS_REGION_NAME=regionOne", "++ OS_REGION_NAME=regionOne", "++ '[' -z '' ']'", "++ export PS1=", "++ PS1=", "++ export 'PS1=${OS_CLOUD:+($OS_CLOUD)} '", "++ PS1='${OS_CLOUD:+($OS_CLOUD)} '", "++ export CLOUDPROMPT_ENABLED=1", "++ CLOUDPROMPT_ENABLED=1", "++ mktemp", "+ tmpfile=/tmp/tmp.313fnmlHOh", "+ openstack network agent list", "/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!", "  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"", "+ grep neutron- /tmp/tmp.313fnmlHOh", "+ grep -v ovn", "+ awk '{print $2}'", "+ xargs -n300 openstack network agent delete", "/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!", "  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"", "+ openstack network list -c ID -c Name -c Subnets", "+ grep 'HA network tenant'", "+ tr -d '|'", "/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!", "  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"", "++ mktemp", "+ ports_tmpfile=/tmp/tmp.O0tkyYtbL2", "+ openstack port list", "/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!", "  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"", "+ ports_to_delete=", "++ awk '{ print $3 }' /tmp/tmp.313fnmlHOh", "+ for subnet_id in $(awk '{ print $3 }' $tmpfile)", "++ awk '/subnet_id='\\''network'\\''/{ print $2 }' /tmp/tmp.O0tkyYtbL2", "+ ports_to_delete=' '", "+ echo", "+ xargs -n300 openstack port delete", "/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!", "  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"", "usage: openstack port delete [-h] <port> [<port> ...]", "openstack port delete: error: the following arguments are required: <port>", "", "+ awk '{ print $1 }' /tmp/tmp.313fnmlHOh", "+ xargs -n300 openstack network delete", "/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!", "  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"", "Failed to delete network with name or ID '107a0ed2-58b7-4bd4-ad6c-7c1555b7accd': ConflictException: 409: Client Error for url: https://overcloud.redhat.local:13696/v2.0/networks/107a0ed2-58b7-4bd4-ad6c-7c1555b7accd, Unable to complete operation on network 107a0ed2-58b7-4bd4-ad6c-7c1555b7accd. There are one or more ports still in use on the network.", "1 of 1 networks failed to delete.", "+ awk '/network:floatingip_agent_gateway/{ print $2 }' /tmp/tmp.O0tkyYtbL2", "+ xargs -n300 openstack port delete", "/usr/lib/python3.9/site-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (5.2.0) doesn't match a supported version!", "  warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"", "usage: openstack port delete [-h] <port> [<port> ...]", "openstack port delete: error: the following arguments are required: <port>", "", "+ :"], "stdout": "", "stdout_lines": []}