Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2127285

Summary:	Instance HA compute nodes not coming up after outage
Product:	Red Hat OpenStack	Reporter:	nalmond
Component:	openstack-tripleo-heat-templates	Assignee:	Luca Miccini <lmiccini>
Status:	CLOSED ERRATA	QA Contact:	Joe H. Rahme <jhakimra>
Severity:	high	Docs Contact:
Priority:	high
Version:	17.1 (Wallaby)	CC:	dasmith, dhill, eglynn, jhakimra, kchamart, lmiccini, mariel, mburns, ratailor, sbauza, sgordon, smooney, vromanso
Target Milestone:	z2	Keywords:	Triaged
Target Release:	17.1
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-14.3.1-17.1.20231103010821.e7c7ce3.el9ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-01-16 14:31:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description nalmond 2022-09-15 20:21:27 UTC

Description of problem:
Following a controlplane outage as a result of a traffic flood, multiple compute nodes were down in 'openstack compute service list'. The controlplane had been restored but these compute nodes were still down. Looking in the output of 'podman logs nova_compute' on the affected compute nodes, we saw this repeatedly:

Running command: '/var/lib/nova/instanceha/check-run-nova-compute '
+ exec /var/lib/nova/instanceha/check-run-nova-compute
Checking 169 migrations
Waiting for evacuations to complete or fail
Checking 169 migrations
Waiting for evacuations to complete or fail
...truncated...

Logging in /var/log/containers/nova/nova-compute.log had stopped.

There were only about 30 instances on this node, nowhere near 169. The evacuations did not appear to be in progress, only 1 instance was in "MIGRATING" and it was on a different compute node than the above messages were captured from (though it was also on an affected node). It does not look like the nodes successfully fenced.

We were able to recover from this with a manual code change to the script in /var/lib/nova/instanceha/check-run-nova-compute but this should not be necessary. Is there a better way to recover from this or even better prevent this state in the first place?

Version-Release number of selected component (if applicable):
16.1.8

How reproducible:
In one environment following a network outage.

Steps to Reproduce:
1. Experience controlplane network outage
2. Stop outage
3. Attempt to restore overcloud services

Comment 15 Luca Miccini 2023-08-10 11:03:46 UTC

let's use this bz to track the backport of https://review.opendev.org/c/openstack/tripleo-heat-templates/+/564024

Comment 17 Luca Miccini 2023-08-10 11:06:24 UTC

(In reply to Luca Miccini from comment #15)
> let's use this bz to track the backport of
> https://review.opendev.org/c/openstack/tripleo-heat-templates/+/564024

obviously meant revert instead of backport: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/889698

Comment 22 Luca Miccini 2023-11-14 06:57:15 UTC

[stack@undercloud-0 ~]$ cat core_puddle_version 
RHOS-17.1-RHEL-9-20231110.n.1

[root@compute-0 ~]# podman exec -it -u root nova_compute bash
[root@compute-0 /]# cat /run_command 
/usr/bin/nova-compute

[root@compute-0 ~]# ls /var/lib/nova/instanceha/check-run-nova-compute
ls: cannot access '/var/lib/nova/instanceha/check-run-nova-compute': No such file or directory

Comment 31 errata-xmlrpc 2024-01-16 14:31:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 17.1.2 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:0209