Description of problem: Overcloud deployment failed after mysql_bootstrap container failed to start properly. Deployment failed with below error:- ~~~ This", "ensures that someone cannot guess at the root password from the network.", "", "Disallow root login remotely? [Y/n] n", " ... skipping.", "", "By default, MariaDB comes with a database named 'test' that anyone can", "access. This is also intended only for testing, and should be removed", "before moving into a production environment.", "", "Remove test database and access to it? [Y/n] y", " - Dropping test database...", " ... Success!", " - Removing privileges on test database...", " ... Success!", "", "Reloading the privilege tables will ensure that all changes made so far", "will take effect immediately.", "", "Reload privilege tables now? [Y/n] y", " ... Success!", "", "Cleaning up...", "", "All done! If you've completed all of the above steps, your MariaDB", "installation should now be secure.", "", "Thanks for using MariaDB!", "200917 07:24:13 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.", "200917 07:24:13 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql"]} 2020-09-17 03:25:14 | 2020-09-17 03:25:14 | NO MORE HOSTS LEFT ************************************************************* 2020-09-17 03:25:14 | 2020-09-17 03:25:14 | PLAY RECAP ********************************************************************* 2020-09-17 03:25:14 | overcloud-controller-0 : ok=216 changed=137 unreachable=0 failed=1 skipped=227 rescued=0 ignored=0 2020-09-17 03:25:14 | overcloud-controller-1 : ok=210 changed=137 unreachable=0 failed=1 skipped=223 rescued=0 ignored=0 2020-09-17 03:25:14 | overcloud-controller-2 : ok=211 changed=137 unreachable=0 failed=0 skipped=223 rescued=0 ignored=0 2020-09-17 03:25:14 | overcloud-novacompute-0 : ok=174 changed=103 unreachable=0 failed=0 skipped=240 rescued=0 ignored=0 2020-09-17 03:25:14 | undercloud : ok=22 changed=13 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 2020-09-17 03:25:14 | 2020-09-17 03:25:14 | Thursday 17 September 2020 03:25:14 -0400 (0:01:19.765) 0:24:43.415 **** 2020-09-17 03:25:14 | =============================================================================== 2020-09-17 03:25:17 | Overcloud configuration failed. 2020-09-17 03:25:17 | ~~~ Looking at /var/log/extra/podman/containers/mysql_bootstrap/stdout.log on controller, We noticed "mysqld is alive" line is missing in logs:- ~~~ 2020-09-17T07:24:12.174748349+00:00 stdout F Thanks for using MariaDB! 2020-09-17T07:24:13.448180470+00:00 stdout F 200917 07:24:13 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 2020-09-17T07:24:13.509833466+00:00 stdout F 200917 07:24:13 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql ~~~ Expected:- Normally we see below logs(/var/log/extra/podman/containers/mysql_bootstrap/stdout.log) in good environment ~~~ Thanks for using MariaDB! 200917 09:02:24 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 200917 09:02:25 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql mysqld is alive ~~~ Version-Release number of selected component (if applicable): 16.2 How reproducible: Very frequent in ci Steps to Reproduce: 1. Deploy tripleio featureset001 on real baremetal environment Additional info:
Verified: The issue was fixed in openstack-tripleo-heat-templates-11.3.2-2.20201011004954.680761b.el8ost, which has been included in OSP 16.2 composes since RHOS-16.2-RHEL-8-20201014.n.3 (Please see [1]). Here tested with core_puddle: RHOS-16.2-RHEL-8-20210514.n.0 on bare metal deployment: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/enterprise/view/scenario/job/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/ Current tripleo-heat-templates version: [stack@undercloud-0 ~]$ yum list installed|grep heat-templates openstack-tripleo-heat-templates.noarch 11.5.1-2.20210506094805.28ae385.el8ost.1 @rhelosp-16.2 In the corresponding log[2], we can see that mysql database bootstrap process finishes successfully: Thanks for using MariaDB! 210517 04:54:50 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 210517 04:54:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql mysqld is alive [1] http://download.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/16.2-RHEL-8/RHOS-16.2-RHEL-8-20201014.n.3/compose/OpenStack/source/tree/Packages/ [2] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-enterprise-baremetal-scenario-16.2-3control_2compute_1freeipavm_externalceph-anycluster_tls/16/controller-1/var/log/extra/podman/containers/mysql_bootstrap/stdout.log.gz
Is doc text for errata release required for the bug (comment 6)?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483