Bug 2225205 - ffu3 with iha broken because pacemaker authkey is overridden
Summary: ffu3 with iha broken because pacemaker authkey is overridden
Keywords:
Status: MODIFIED
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 17.1 (Wallaby)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z1
: 17.1
Assignee: Luca Miccini
QA Contact: Joe H. Rahme
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-07-24 14:43 UTC by Luca Miccini
Modified: 2023-08-08 13:22 UTC (History)
3 users (show)

Fixed In Version: openstack-tripleo-heat-templates-14.3.1-17.1.20230801070949.d134056.el9osttrunk
Doc Type: Known Issue
Doc Text:
Outdated upgrade orchestration logic overrides the existing pacemaker authkey during the Fast Forward Upgrade (FFU) procedure, preventing Pacemaker from connecting to `pacemaker_remote` running on Compute nodes when Instance HA is enabled. As a result, the upgrade fails and `pacemaker_remote` running on Compute nodes is unreachable from the central cluster. Contact Red Hat support to receive instructions on how to perform FFU if Instance HA is configured.
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 889306 0 None MERGED Revert "pcmk_remote FFU support for Instance HA" 2023-07-31 16:35:52 UTC
Red Hat Issue Tracker OSP-26862 0 None None None 2023-07-24 14:45:33 UTC

Description Luca Miccini 2023-07-24 14:43:23 UTC 
Description of problem:

ffu3 of iha-enabled compute fails because of

[root@compute-0 ~]# pcs status
Error: unable to get cib


[root@compute-0 ~]# systemctl status pacemaker_remote
● pacemaker_remote.service - Pacemaker Remote executor daemon
     Loaded: loaded (/usr/lib/systemd/system/pacemaker_remote.service; enabled; preset: disabled)
     Active: active (running) since Sun 2023-07-23 19:03:47 UTC; 12h ago
       Docs: man:pacemaker-remoted
             https://clusterlabs.org/pacemaker/doc/
   Main PID: 1805 (pacemaker-remot)
      Tasks: 1
     Memory: 4.1M
        CPU: 6.400s
     CGroup: /system.slice/pacemaker_remote.service
             └─1805 /usr/sbin/pacemaker-remoted

Jul 24 07:30:14 compute-0 pacemaker-remoted[1805]:  warning: Cannot proxy IPC connection from uid 0 gid 0 to stonith-ng because not connected to cluster
Jul 24 07:30:14 compute-0 pacemaker-remoted[1805]:  error: Error in connection setup (/dev/shm/qb-1805-426744-15-hXLZnC/qb): Remote I/O error (121)
Jul 24 07:30:14 compute-0 pacemaker-remoted[1805]:  warning: Cannot proxy IPC connection from uid 0 gid 0 to cib_ro because not connected to cluster
Jul 24 07:30:14 compute-0 pacemaker-remoted[1805]:  error: Error in connection setup (/dev/shm/qb-1805-426744-15-mu5pgV/qb): Remote I/O error (121)
Jul 24 07:30:14 compute-0 pacemaker-remoted[1805]:  warning: Cannot proxy IPC connection from uid 0 gid 0 to cib_rw because not connected to cluster
Jul 24 07:30:14 compute-0 pacemaker-remoted[1805]:  error: Error in connection setup (/dev/shm/qb-1805-426745-15-NbzTcs/qb): Remote I/O error (121)
Jul 24 07:30:22 compute-0 pacemaker-remoted[1805]:  error: TLS handshake with remote client failed: An illegal parameter has been received.
Jul 24 07:30:22 compute-0 pacemaker-remoted[1805]:  notice: Cleaning up after remote client 4337f0b1-17fd-4801-b796-3042f907e351 disconnected
Jul 24 07:32:19 compute-0 pacemaker-remoted[1805]:  error: TLS handshake with remote client failed: An illegal parameter has been received.
Jul 24 07:32:19 compute-0 pacemaker-remoted[1805]:  notice: Cleaning up after remote client 6c198dd7-bb62-47ed-bf2f-f35cc886f456 disconnected



[root@compute-0 ~]# md5sum /etc/pacemaker/authkey
33e5d2975e2b9b808427e4d7f42dc66a  /etc/pacemaker/authkey
[root@compute-0 ~]#
logout
[tripleo-admin@compute-0 ~]$
logout
Connection to compute-0.ctlplane closed.
[stack@undercloud-0 ~]$ ssh controller-0.ctlplane -l tripleo-admin
Warning: Permanently added 'controller-0.ctlplane' (ED25519) to the list of known hosts.
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
Last login: Mon Jul 24 07:31:07 2023 from 192.168.24.1
[tripleo-admin@controller-0 ~]$ sudo  md5sum /etc/pacemaker/authkey
5fc58f1a19cf646c13de195d7963356a  /etc/pacemaker/authkey
[tripleo-admin@controller-0 ~]$



[root@compute-0 ~]# ll /etc/pacemaker/authkey
-rw-r-----. 1 hacluster haclient 4096 Jul 23 16:00 /etc/pacemaker/authkey

[tripleo-admin@controller-0 ~]$ sudo ls -l /etc/pacemaker/authkey
-r--------. 1 hacluster haclient 256 Jul 23 14:07 /etc/pacemaker/authkey


https://review.opendev.org/c/openstack/tripleo-heat-templates/+/889306
Note You need to log in before you can comment on or make changes to this bug.