Bug 2037409

Summary: [16.2] backup-and-restore : openstack overcloud backup fails at "Galera desync the MySQL node"
Product: Red Hat OpenStack Reporter: Jean-Francois Beaudoin <jbeaudoi>
Component: tripleo-ansibleAssignee: Juan Larriba <jlarriba>
Status: CLOSED ERRATA QA Contact: myadla
Severity: medium Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: aschultz, ayefimov, camorris, dhill, jbadiapa, mciecier, renich
Target Milestone: z3Keywords: Reopened, Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tripleo-ansible-0.8.1-2.20220215090656.c0ddf19.el8osttrunk Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2039801 (view as bug list) Environment:
Last Closed: 2022-06-22 16:03:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2039801    

Description Jean-Francois Beaudoin 2022-01-05 15:15:02 UTC 
Description of problem:

Following this documentation:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/backing_up_and_restoring_the_undercloud_and_control_plane_nodes/assembly_backing-up-the-control-plane-nodes_br-undercloud-ctlplane

When running:
(undercloud) [stack@rhosp-inst ~]$ openstack overcloud backup --inventory /home/stack/tripleo-inventory.yaml


With 16.2:
~~~
TASK [backup-and-restore : Set the tripleo_backup_and_restore_mysql_container id] ***
ok: [rhosp16-1-ctl-0] => {"ansible_facts": {"tripleo_backup_and_restore_mysql_container": "xxxxxxxxxxxx"}, "changed": false}
ok: [rhosp16-1-ctl-1] => {"ansible_facts": {"tripleo_backup_and_restore_mysql_container": "xxxxxxxxxxxx"}, "changed": false}
ok: [rhosp16-1-ctl-2] => {"ansible_facts": {"tripleo_backup_and_restore_mysql_container": "xxxxxxxxxxxx"}, "changed": false}

TASK [backup-and-restore : Unpause mysql for backup if it is paused] ***********
skipping: [rhosp16-1-ctl-0] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [rhosp16-1-ctl-1] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [rhosp16-1-ctl-2] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [backup-and-restore : MySQL Grants backup] ********************************
changed: [rhosp16-1-ctl-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}
changed: [rhosp16-1-ctl-1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}
changed: [rhosp16-1-ctl-2] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}
~~~

With 16.2.1:
~~~
TASK [backup-and-restore : Set the tripleo_backup_and_restore_mysql_container id] ********************************************************************************************************************************
ok: [rhosp16-1-ctl-0] => {"ansible_facts": {"tripleo_backup_and_restore_mysql_container": "xxxxxxxxxxxx"}, "changed": false}
ok: [rhosp16-1-ctl-1] => {"ansible_facts": {"tripleo_backup_and_restore_mysql_container": "xxxxxxxxxxxx"}, "changed": false}
ok: [rhosp16-1-ctl-2] => {"ansible_facts": {"tripleo_backup_and_restore_mysql_container": "xxxxxxxxxxxx"}, "changed": false}

TASK [backup-and-restore : Unpause mysql for backup if it is paused] *********************************************************************************************************************************************
skipping: [rhosp16-1-ctl-0] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [rhosp16-1-ctl-1] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [rhosp16-1-ctl-2] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [backup-and-restore : Galera desync the MySQL node] *********************************************************************************************************************************************************
fatal: [rhosp16-1-ctl-0]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
fatal: [rhosp16-1-ctl-1]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
fatal: [rhosp16-1-ctl-2]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
~~~

Version-Release number of selected component (if applicable):
OpenStack 16.2.1

How reproducible:
Every time.

Steps to Reproduce:
1.
2.
3.

Actual results:
Overcloud backup failing.

Expected results:
Being able to run `openstack overcloud backup --inventory /home/stack/tripleo-inventory.yaml` with 16.2.1

Additional info:
~~~
  * Container bundle set: galera-bundle [cluster.common.tag/openstack-mariadb:pcmklatest]:
    * galera-bundle-0   (ocf::heartbeat:galera):         Master rhosp16-1-ctl-1
    * galera-bundle-1   (ocf::heartbeat:galera):         Master rhosp16-1-ctl-0
    * galera-bundle-2   (ocf::heartbeat:galera):         Master rhosp16-1-ctl-2
~~~
Comment 3 Juan Larriba 2022-01-10 15:27:03 UTC 
This issue is a race condition that was fixed after 16.2.1 release and, as such, will be included in 16.2.2.
Comment 8 Juan Larriba 2022-01-17 08:33:35 UTC 
I am closing this issue as it is included in current release 16.2.1
Comment 18 errata-xmlrpc 2022-06-22 16:03:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4793