Bug 1967142

Description of problem:
During execution of ML2OVS -> ML2OVN migration procedure validation of migration resources failed. 

64 bytes from 10.0.0.254: icmp_seq=2699 ttl=62 time=0.639 ms
64 bytes from 10.0.0.254: icmp_seq=2700 ttl=62 time=1.61 ms
64 bytes from 10.0.0.254: icmp_seq=2701 ttl=62 time=0.610 ms
64 bytes from 10.0.0.254: icmp_seq=2702 ttl=62 time=0.605 ms
From 10.0.0.84 icmp_seq=2712 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2713 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2714 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2715 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2716 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2717 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2718 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2719 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2720 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2721 Destination Host Unreachable
From 10.0.0.84 icmp_seq=2722 Destination Host Unreachable

(overcloud) [stack@undercloud-0 ovn_migration]$ openstack network agent list
Failed to discover available identity versions when contacting http://10.0.0.149:5000. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://10.0.0.149:5000: HTTPConnectionPool(host='10.0.0.149', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f62ff100b38>: Failed to establish a new connection: [Errno 113] No route to host',))

Seems like keystone not responding.

Pacemaker cluster stopped. 

[heat-admin@controller-0 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: controller-0 (version 2.0.5-9.el8-ba59be7122) - partition WITHOUT quorum
  * Last updated: Wed Jun  2 13:42:20 2021
  * Last change:  Wed Jun  2 12:11:19 2021 by root via cibadmin on controller-0
  * 15 nodes configured
  * 47 resource instances configured

Node List:
  * Online: [ controller-0 ]
  * OFFLINE: [ controller-1 controller-2 ]

Full List of Resources:
  * ip-192.168.24.20	(ocf::heartbeat:IPaddr2):	 Stopped
  * ip-10.0.0.149	(ocf::heartbeat:IPaddr2):	 Stopped
  * ip-172.17.1.140	(ocf::heartbeat:IPaddr2):	 Stopped
  * ip-172.17.1.72	(ocf::heartbeat:IPaddr2):	 Stopped
  * ip-172.17.3.74	(ocf::heartbeat:IPaddr2):	 Stopped
  * ip-172.17.4.27	(ocf::heartbeat:IPaddr2):	 Stopped
  * Container bundle set: haproxy-bundle [cluster.common.tag/rhosp16-openstack-haproxy:pcmklatest]:
    * haproxy-bundle-podman-0	(ocf::heartbeat:podman):	 Stopped
    * haproxy-bundle-podman-1	(ocf::heartbeat:podman):	 Stopped
    * haproxy-bundle-podman-2	(ocf::heartbeat:podman):	 Stopped
  * Container bundle set: galera-bundle [cluster.common.tag/rhosp16-openstack-mariadb:pcmklatest]:
    * galera-bundle-0	(ocf::heartbeat:galera):	 Stopped
    * galera-bundle-1	(ocf::heartbeat:galera):	 Stopped
    * galera-bundle-2	(ocf::heartbeat:galera):	 Stopped
  * Container bundle set: rabbitmq-bundle [cluster.common.tag/rhosp16-openstack-rabbitmq:pcmklatest]:
    * rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	 Stopped
    * rabbitmq-bundle-1	(ocf::heartbeat:rabbitmq-cluster):	 Stopped
    * rabbitmq-bundle-2	(ocf::heartbeat:rabbitmq-cluster):	 Stopped
  * Container bundle set: redis-bundle [cluster.common.tag/rhosp16-openstack-redis:pcmklatest]:
    * redis-bundle-0	(ocf::heartbeat:redis):	 Stopped
    * redis-bundle-1	(ocf::heartbeat:redis):	 Stopped
    * redis-bundle-2	(ocf::heartbeat:redis):	 Stopped
  * Container bundle: openstack-cinder-volume [cluster.common.tag/rhosp16-openstack-cinder-volume:pcmklatest]:
    * openstack-cinder-volume-podman-0	(ocf::heartbeat:podman):	 Stopped
  * Container bundle set: ovn-dbs-bundle [cluster.common.tag/rhosp16-openstack-ovn-northd:pcmklatest]:
    * ovn-dbs-bundle-0	(ocf::ovn:ovndb-servers):	 Stopped
    * ovn-dbs-bundle-1	(ocf::ovn:ovndb-servers):	 Stopped
    * ovn-dbs-bundle-2	(ocf::ovn:ovndb-servers):	 Stopped
  * ip-172.17.1.39	(ocf::heartbeat:IPaddr2):	 Stopped

Failed Resource Actions:
  * redis-bundle-0_monitor_30000 on controller-0 'error' (1): call=15, status='Error', exitreason='', last-rc-change='2021-06-02 12:26:37Z', queued=0ms, exec=0ms
  * rabbitmq-bundle-0_monitor_30000 on controller-0 'error' (1): call=10, status='Error', exitreason='', last-rc-change='2021-06-02 12:26:27Z', queued=0ms, exec=0ms
  * galera-bundle-0_monitor_30000 on controller-0 'error' (1): call=5, status='Error', exitreason='', last-rc-change='2021-06-02 12:26:17Z', queued=0ms, exec=0ms
  * ip-172.17.1.72_monitor_10000 on controller-0 'error' (1): call=21, status='complete', exitreason='[findif] failed', last-rc-change='2021-06-02 12:25:58Z', queued=0ms, exec=0ms

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled


Feel free to move the bug under more relevant component/DFG.


 
Version-Release number of selected component (if applicable):
RHOS-16.2-RHEL-8-20210525.n.0

How reproducible:
100%


1. Deploy a HA environment (3 controllers + 2 compute)
2. Create some resources (internal network, router between external and internal networks, server with fip connected to the internal network) before starting migration and start a script that records ping results to the server's fip. 
3. Perform OVN migration steps from the official documentation https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/networking_with_open_virtual_network/migrating-ml2ovs-to-ovn


Actual results:
Migration to OVN fails. Migration resources are not reachable. Openstack services are not responding.

Expected results:
Migration succeeds

Additional info: