Bug 2101765

Summary: [OSP17] openstack overcloud backup return error
Product: Red Hat OpenStack Reporter: Arnau Verdaguer <averdagu>
Component: tripleo-ansibleAssignee: Juan Larriba <jlarriba>
Status: CLOSED ERRATA QA Contact: Alex Yefimov <ayefimov>
Severity: medium Docs Contact:
Priority: high    
Version: 17.0 (Wallaby)CC: ayefimov, jlarriba, jslagle, skovili
Target Milestone: gaKeywords: Reopened, Triaged
Target Release: 17.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tripleo-ansible-3.3.1-0.20220709033429.f5b4f56.el9ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-21 12:23:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Arnau Verdaguer 2022-06-28 10:40:00 UTC
Description of problem:
It is not possible to run openstack overcloud backup on OSP 17.
The command return an error after using all retries on task
"Galera desync the MySQL node"

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. deploy a devnest node with OSP17
2. run openstack overcloud backup

Actual results:
openstack overcloud backup finish with error

Expected results:
openstack overcloud backup finish without error

Additional info:
The problem seem to be that podman returns a warning every time it is executed.
The warning is: 'WARN[0000]  binary not found, container dns will not be enabled'
- name: Galera desync the MySQL node
  shell: |
    set -o pipefail
    {{ tripleo_container_cli }} exec {{ tripleo_backup_and_restore_mysql_container }} bash -c "mysql -p -u root \
    -p{{ mysql_password.stdout }} --execute 'SET GLOBAL wsrep_desync = ON'"
  register: desync_output
  retries: 300
  delay: 5
  until: desync_output.stderr == ""
  when:
    - mysql_password.stderr is defined
    - enabled_galera
    - tripleo_backup_and_restore_enable_snapshots|bool or not tripleo_backup_and_restore_service_manager|bool
  tags:
    - bar_create_recover_image
  no_log: "{{ tripleo_backup_and_restore_hide_sensitive_logs | bool }}"


Given that this task has 'until: desync_output.stderr == ""' it will fail since on stderr will have the warn
even if it's not related to the actual task.

Comment 1 Juan Larriba 2022-06-30 08:17:34 UTC
Hi Arnau,

there is a workaround for this issue. Pass the parameter

tripleo_backup_and_restore_hide_sensitive_logs: true

to the execution of openstack overcloud backup and it will work.

However, a fix for this was introduced a couple of months ago so if you use the latest compose, that workaround should not be needed.

Comment 2 Juan Larriba 2022-07-01 11:35:20 UTC
Re-opening this issue as it has been reproduced now by our CI, and it is a different issue than I believed.

It seems that someone has removed the "mysql" binary from the mariadb container, a fix is on the works.

Comment 4 Alex Yefimov 2022-07-01 19:55:28 UTC
Workaround per JuanB and verified by alexy:

    There is not a workaround, unless you modify the  ansible role directly on your undercloud.

    If you want to do that, you have to edit /usr/share/ansible/roles/backup_and_restore/tasks/db_backup, line 115.

    Replace the current line:

    until: not desync_output.stderr

    with 

    until: desync_output.stderr == "" or '"WARN" in desync_output.stderr'

Comment 7 Alex Yefimov 2022-07-16 22:57:23 UTC
Bug fix verified via Jenkins job.

Comment 12 errata-xmlrpc 2022-09-21 12:23:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543