Bug 1879914 - Overcloud deployment failed after mysql_bootstrap container failed to start properly.
Summary: Overcloud deployment failed after mysql_bootstrap container failed to start p...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.2 (Train)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Damien Ciabrini
QA Contact: pkomarov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-17 10:51 UTC by Sandeep Yadav
Modified: 2021-09-15 07:09 UTC (History)
7 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.3.2-2.20201011004954.680761b.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-15 07:09:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1896009 0 None None None 2020-09-17 14:14:17 UTC
OpenStack gerrit 752619 0 None MERGED Remove race during mysql database creation 2021-02-15 12:20:11 UTC
Red Hat Product Errata RHEA-2021:3483 0 None None None 2021-09-15 07:09:39 UTC

Description Sandeep Yadav 2020-09-17 10:51:01 UTC
Description of problem:

Overcloud deployment failed after mysql_bootstrap container failed to start properly.

Deployment failed with below error:-

~~~
This", "ensures that someone cannot guess at the root password from the network.", "", "Disallow root login remotely? [Y/n] n", " ... skipping.", "", "By default, MariaDB comes with a database named 'test' that anyone can", "access.  This is also intended only for testing, and should be removed", "before moving into a production environment.", "", "Remove test database and access to it? [Y/n] y", " - Dropping test database...", " ... Success!", " - Removing privileges on test database...", " ... Success!", "", "Reloading the privilege tables will ensure that all changes made so far", "will take effect immediately.", "", "Reload privilege tables now? [Y/n] y", " ... Success!", "", "Cleaning up...", "", "All done!  If you've completed all of the above steps, your MariaDB", "installation should now be secure.", "", "Thanks for using MariaDB!", "200917 07:24:13 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.", "200917 07:24:13 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql"]}
2020-09-17 03:25:14 | 
2020-09-17 03:25:14 | NO MORE HOSTS LEFT *************************************************************
2020-09-17 03:25:14 | 
2020-09-17 03:25:14 | PLAY RECAP *********************************************************************
2020-09-17 03:25:14 | overcloud-controller-0     : ok=216  changed=137  unreachable=0    failed=1    skipped=227  rescued=0    ignored=0   
2020-09-17 03:25:14 | overcloud-controller-1     : ok=210  changed=137  unreachable=0    failed=1    skipped=223  rescued=0    ignored=0   
2020-09-17 03:25:14 | overcloud-controller-2     : ok=211  changed=137  unreachable=0    failed=0    skipped=223  rescued=0    ignored=0   
2020-09-17 03:25:14 | overcloud-novacompute-0    : ok=174  changed=103  unreachable=0    failed=0    skipped=240  rescued=0    ignored=0   
2020-09-17 03:25:14 | undercloud                 : ok=22   changed=13   unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
2020-09-17 03:25:14 | 
2020-09-17 03:25:14 | Thursday 17 September 2020  03:25:14 -0400 (0:01:19.765)       0:24:43.415 **** 
2020-09-17 03:25:14 | =============================================================================== 
2020-09-17 03:25:17 | Overcloud configuration failed.
2020-09-17 03:25:17 | 
~~~

Looking at /var/log/extra/podman/containers/mysql_bootstrap/stdout.log on controller, We noticed "mysqld is alive" line is missing in logs:-

~~~
2020-09-17T07:24:12.174748349+00:00 stdout F Thanks for using MariaDB!
2020-09-17T07:24:13.448180470+00:00 stdout F 200917 07:24:13 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
2020-09-17T07:24:13.509833466+00:00 stdout F 200917 07:24:13 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
~~~

Expected:-

Normally we see below logs(/var/log/extra/podman/containers/mysql_bootstrap/stdout.log) in good environment 

~~~
Thanks for using MariaDB!
200917 09:02:24 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
200917 09:02:25 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
mysqld is alive
~~~


Version-Release number of selected component (if applicable):

16.2


How reproducible:
Very frequent in ci 

Steps to Reproduce:
1. Deploy tripleio featureset001 on real baremetal environment


Additional info:

Comment 5 Julia Marciano 2021-05-19 11:56:15 UTC
Verified:

The issue was fixed in openstack-tripleo-heat-templates-11.3.2-2.20201011004954.680761b.el8ost, which has been included in OSP 16.2 composes since RHOS-16.2-RHEL-8-20201014.n.3 (Please see [1]).

Here tested with core_puddle: RHOS-16.2-RHEL-8-20210514.n.0 on bare metal deployment:
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/enterprise/view/scenario/job/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/

Current tripleo-heat-templates version:
[stack@undercloud-0 ~]$ yum list installed|grep heat-templates
openstack-tripleo-heat-templates.noarch       11.5.1-2.20210506094805.28ae385.el8ost.1    @rhelosp-16.2  

In the corresponding log[2], we can see that mysql database bootstrap process finishes successfully:

Thanks for using MariaDB!
210517 04:54:50 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
210517 04:54:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
mysqld is alive

[1] http://download.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/16.2-RHEL-8/RHOS-16.2-RHEL-8-20201014.n.3/compose/OpenStack/source/tree/Packages/
[2] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/16/controller-1/var/log/extra/podman/containers/mysql_bootstrap/stdout.log.gz

Comment 7 Julia Marciano 2021-08-01 20:04:04 UTC
Is doc text for errata release required for the bug (comment 6)?

Comment 10 errata-xmlrpc 2021-09-15 07:09:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483


Note You need to log in before you can comment on or make changes to this bug.