Bug 2237659

Description of problem:

I'm testing FFU from 16.2 to 17.1 in my test environment where Instance-HA is enabled.
When I tried to upgrade Instance-HA compute nodes using the following command, it failed with the following error.

~~~
(undercloud) [stack@undercloud ~]$ openstack overcloud upgrade run --yes --tags system_upgrade --stack overcloud --limit overcloud-novacompute-0,overcloud-novacompute-1

2023-09-06 16:43:23.729721 | 525400dd-aae4-848d-14b9-000000000029 |     TIMING | include_tasks | overcloud-novacompute-0 | 0:00:05.533928 | 0.04s
2023-09-06 16:43:23.768307 | 41fbf775-5dda-4845-b877-303ffdc51a76 |   INCLUDED | /home/stack/overcloud-deploy/overcloud/config-download/overcloud/Compute/upgrade_tasks_step2.yaml | overcloud-novacompute-0
2023-09-06 16:43:23.797551 | 525400dd-aae4-848d-14b9-0000000002c3 |       TASK | Check if pcs is present
2023-09-06 16:43:23.804193 | 525400dd-aae4-848d-14b9-000000000029 |     TIMING | include_tasks | overcloud-novacompute-1 | 0:00:05.608387 | 0.09s
2023-09-06 16:43:23.841599 | da10ce9b-11e5-41b9-ad1c-afb4fbdfd52c |   INCLUDED | /home/stack/overcloud-deploy/overcloud/config-download/overcloud/Compute/upgrade_tasks_step2.yaml | overcloud-novacompute-1
2023-09-06 16:43:23.870207 | 525400dd-aae4-848d-14b9-0000000002e7 |       TASK | Check if pcs is present
2023-09-06 16:43:24.165912 | 525400dd-aae4-848d-14b9-0000000002c3 |         OK | Check if pcs is present | overcloud-novacompute-0
2023-09-06 16:43:24.167516 | 525400dd-aae4-848d-14b9-0000000002c3 |     TIMING | Check if pcs is present | overcloud-novacompute-0 | 0:00:05.971731 | 0.37s
2023-09-06 16:43:24.189624 | 525400dd-aae4-848d-14b9-0000000002c4 |       TASK | Stop pacemaker cluster before stopping all docker containers
2023-09-06 16:43:24.196430 | 525400dd-aae4-848d-14b9-0000000002e7 |         OK | Check if pcs is present | overcloud-novacompute-1
2023-09-06 16:43:24.198298 | 525400dd-aae4-848d-14b9-0000000002e7 |     TIMING | Check if pcs is present | overcloud-novacompute-1 | 0:00:06.002502 | 0.33s
2023-09-06 16:43:24.221563 | 525400dd-aae4-848d-14b9-0000000002e8 |       TASK | Stop pacemaker cluster before stopping all docker containers
2023-09-06 16:48:27.335198 | 525400dd-aae4-848d-14b9-0000000002c4 |      FATAL | Stop pacemaker cluster before stopping all docker containers | overcloud-novacompute-0 | error={"changed": false, "msg": "Failed to set the state `offline` on the cluster\n"}
2023-09-06 16:48:27.338251 | 525400dd-aae4-848d-14b9-0000000002c4 |     TIMING | Stop pacemaker cluster before stopping all docker containers | overcloud-novacompute-0 | 0:05:09.142451 | 303.15s
2023-09-06 16:48:27.641226 | 525400dd-aae4-848d-14b9-0000000002e8 |      FATAL | Stop pacemaker cluster before stopping all docker containers | overcloud-novacompute-1 | error={"changed": false, "msg": "Failed to set the state `offline` on the cluster\n"}
2023-09-06 16:48:27.642011 | 525400dd-aae4-848d-14b9-0000000002e8 |     TIMING | Stop pacemaker cluster before stopping all docker containers | overcloud-novacompute-1 | 0:05:09.446240 | 303.42s

PLAY RECAP *********************************************************************
overcloud-novacompute-0    : ok=13   changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=1   
overcloud-novacompute-1    : ok=12   changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=1 
~~~

The reason why the above ansible task fails is that remote nodes won't become "offline" even if "pcs cluster stop" runs.
The following is the task causing the issue.

~~~
https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/deployment/podman/podman-baremetal-ansible.yaml#L215-L224

            - name: Check if pcs is present
              stat:
                path: /usr/sbin/pcs
              register: pcs_stat
            - name: Stop pacemaker cluster before stopping all docker containers
              # NOTE: We are intentionally not using the community version of
              # pacemaker_cluster here due to variances between the two:
              # https://bugs.launchpad.net/tripleo/+bug/1938967
              pacemaker_cluster: state=offline
              when: pcs_stat.stat.exists
~~~

"pacemaker_cluster" module internally calls "pcs cluster stop" command and ensures that the node become offline using "crm_node -q" and "pcs cluster status".

~~~
https://github.com/redhat-openstack/ansible-pacemaker/blob/master/modules/pacemaker_cluster.py#L120-L121

    if state == 'offline':
        cmd = "pcs cluster stop"

         :

    while time.time() < t+timeout:
        cluster_state = get_cluster_status(module)
        if cluster_state == state:
            ready = True
            break
    if not ready:
        module.fail_json(msg="Failed to set the state `%s` on the cluster\n" % (state))


https://github.com/redhat-openstack/ansible-pacemaker/blob/master/modules/pacemaker_cluster.py#L86-L96

def get_cluster_status(module):
    cmd_partition = "crm_node -q"
    partition_rc, partition_out, partition_err = module.run_command(cmd_partition)
    if partition_out.strip() != "1": # we're not in a quorate partition or cluster is down
        return 'offline'
    cmd = "pcs cluster status"
    rc, out, err = module.run_command(cmd)
    if rc != 0:
        return 'offline'
    else:
        return 'online'
~~~

However, on remote nodes (on Instance-HA compute nodes), "crm_node -q" and "pcs cluster status" won't show "offline" even if we run "pcs cluster stop".

~~~
[root@overcloud-novacompute-0 ~]# pcs cluster stop
Stopping Cluster (pacemaker)...
Stopping Cluster (corosync)...

[root@overcloud-novacompute-0 ~]# crm_node -q
1

  ==> "1" means the node is online

[root@overcloud-novacompute-0 ~]# pcs cluster status > /dev/null
[root@overcloud-novacompute-0 ~]# echo $?
0

  ==> "0" means the node is online
~~~

I suppose we can skip the "Stop pacemaker cluster before stopping all docker containers" task on Instance-HA compute nodes.
Because fencing is disabled in the previous steps, I suppose we can safely upgrade/reboot the Instance-HA compute nodes without any other operations.
What do you think about adding a ansible task to check if the node is a cluster node or a remote node, and then skipping the "Stop pacemaker cluster before stopping all docker containers" task when the node is a remote node?


Version-Release number of selected component (if applicable):
RHOSP 17.1

How reproducible:

Steps to Reproduce:
1. Deploy RHOSP 16.2 with Instance HA
2. Do FFU according to our document https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.1/html-single/framework_for_upgrades_16.2_to_17.1/index


Actual results:
Upgrading compute nodes fails

Expected results:
Upgrading compute nodes succeeds