Bug 2101765 - [OSP17] openstack overcloud backup return error
Summary: [OSP17] openstack overcloud backup return error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: tripleo-ansible
Version: 17.0 (Wallaby)
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ga
: 17.0
Assignee: Juan Larriba
QA Contact: Alex Yefimov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-28 10:40 UTC by Arnau Verdaguer
Modified: 2022-09-21 12:23 UTC (History)
4 users (show)

Fixed In Version: tripleo-ansible-3.3.1-0.20220709033429.f5b4f56.el9ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-09-21 12:23:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 847738 0 None NEW [B&R] More robust check on Galera desync the MySQL node 2022-06-28 11:12:43 UTC
OpenStack gerrit 848442 0 None NEW The stderr of the command can have WARNs 2022-07-01 12:59:44 UTC
Red Hat Issue Tracker OSP-16088 0 None None None 2022-06-28 10:52:19 UTC
Red Hat Product Errata RHEA-2022:6543 0 None None None 2022-09-21 12:23:41 UTC

Description Arnau Verdaguer 2022-06-28 10:40:00 UTC
Description of problem:
It is not possible to run openstack overcloud backup on OSP 17.
The command return an error after using all retries on task
"Galera desync the MySQL node"

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. deploy a devnest node with OSP17
2. run openstack overcloud backup

Actual results:
openstack overcloud backup finish with error

Expected results:
openstack overcloud backup finish without error

Additional info:
The problem seem to be that podman returns a warning every time it is executed.
The warning is: 'WARN[0000]  binary not found, container dns will not be enabled'
- name: Galera desync the MySQL node
  shell: |
    set -o pipefail
    {{ tripleo_container_cli }} exec {{ tripleo_backup_and_restore_mysql_container }} bash -c "mysql -p -u root \
    -p{{ mysql_password.stdout }} --execute 'SET GLOBAL wsrep_desync = ON'"
  register: desync_output
  retries: 300
  delay: 5
  until: desync_output.stderr == ""
  when:
    - mysql_password.stderr is defined
    - enabled_galera
    - tripleo_backup_and_restore_enable_snapshots|bool or not tripleo_backup_and_restore_service_manager|bool
  tags:
    - bar_create_recover_image
  no_log: "{{ tripleo_backup_and_restore_hide_sensitive_logs | bool }}"


Given that this task has 'until: desync_output.stderr == ""' it will fail since on stderr will have the warn
even if it's not related to the actual task.

Comment 1 Juan Larriba 2022-06-30 08:17:34 UTC
Hi Arnau,

there is a workaround for this issue. Pass the parameter

tripleo_backup_and_restore_hide_sensitive_logs: true

to the execution of openstack overcloud backup and it will work.

However, a fix for this was introduced a couple of months ago so if you use the latest compose, that workaround should not be needed.

Comment 2 Juan Larriba 2022-07-01 11:35:20 UTC
Re-opening this issue as it has been reproduced now by our CI, and it is a different issue than I believed.

It seems that someone has removed the "mysql" binary from the mariadb container, a fix is on the works.

Comment 4 Alex Yefimov 2022-07-01 19:55:28 UTC
Workaround per JuanB and verified by alexy:

    There is not a workaround, unless you modify the  ansible role directly on your undercloud.

    If you want to do that, you have to edit /usr/share/ansible/roles/backup_and_restore/tasks/db_backup, line 115.

    Replace the current line:

    until: not desync_output.stderr

    with 

    until: desync_output.stderr == "" or '"WARN" in desync_output.stderr'

Comment 7 Alex Yefimov 2022-07-16 22:57:23 UTC
Bug fix verified via Jenkins job.

Comment 12 errata-xmlrpc 2022-09-21 12:23:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543


Note You need to log in before you can comment on or make changes to this bug.