Description of problem:
Following a controlplane outage as a result of a traffic flood, multiple compute nodes were down in 'openstack compute service list'. The controlplane had been restored but these compute nodes were still down. Looking in the output of 'podman logs nova_compute' on the affected compute nodes, we saw this repeatedly:
Running command: '/var/lib/nova/instanceha/check-run-nova-compute '
+ exec /var/lib/nova/instanceha/check-run-nova-compute
Checking 169 migrations
Waiting for evacuations to complete or fail
Checking 169 migrations
Waiting for evacuations to complete or fail
...truncated...
Logging in /var/log/containers/nova/nova-compute.log had stopped.
There were only about 30 instances on this node, nowhere near 169. The evacuations did not appear to be in progress, only 1 instance was in "MIGRATING" and it was on a different compute node than the above messages were captured from (though it was also on an affected node). It does not look like the nodes successfully fenced.
We were able to recover from this with a manual code change to the script in /var/lib/nova/instanceha/check-run-nova-compute but this should not be necessary. Is there a better way to recover from this or even better prevent this state in the first place?
Version-Release number of selected component (if applicable):
16.1.8
How reproducible:
In one environment following a network outage.
Steps to Reproduce:
1. Experience controlplane network outage
2. Stop outage
3. Attempt to restore overcloud services