Bug 2019178

Summary:	FFU: upgrade failed during "Upgrade Mysql database from a temporary container" step
Product:	Red Hat OpenStack	Reporter:	Jesse Pretorius <jpretori>
Component:	openstack-tripleo-heat-templates	Assignee:	Damien Ciabrini <dciabrin>
Status:	CLOSED ERRATA	QA Contact:	Jason Grosso <jgrosso>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	16.2 (Train)	CC:	amcleod, dabarzil, gregraka, mburns, spower, tvignaud
Target Milestone:	z1	Keywords:	Triaged
Target Release:	16.2 (Train on RHEL 8.4)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-11.5.1-2.20210603174829.el8ost.10	Doc Type:	Bug Fix
Doc Text:	Before this update, an upgradable `mariadb-server` package in the RHEL repository caused the package manager to upgrade the `mariadb-server` package on the host, interfering with the containerized `mariadb-server` that pre-exists on the same host. With this update, the Red Hat OpenStack Platform (RHOSP) director removes the `mariadb-server` package from any hosts which also have the containerized MariaDB, and the RHOSP FFU process continues.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-12-09 20:41:59 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2015325

Description Jesse Pretorius 2021-11-01 18:58:32 UTC

This bug was initially created as a copy of Bug #2015325 (the same bug and fix applies)

Description of problem:

As part of FFU upgrade (13 to 16.2), OpenStack node upgrade failed. The OS has been upgraded from 7.7 to 8.4



TASK [Make sure Mysql upgrade temporary directory exists] **********************
Monday 18 October 2021  18:12:37 -0400 (0:00:00.070)       0:02:32.120 ********
changed: [overcloud-controller-0] => {"changed": true, "gid": 0, "group": "root", "mode": "0700", "owner": "root", "path": "/tmp/mariadb-upgrade", "secontext": "system_u:object_r:container_file_t:s0", "size": 25, "state": "directory", "uid": 0}


TASK [Upgrade Mysql database from a temporary container] ***********************
Monday 18 October 2021  18:12:37 -0400 (0:00:00.323)       0:02:32.443 ********
fatal: [overcloud-controller-0]: FAILED! => {"changed": true, "cmd": "podman run --rm --log-driver=k8s-file --log-opt path=/var/log/containers/mysql/db-upgrade.log -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ipa/ca.crt:/etc/ipa/ca.crt:ro -v /etc/puppet:/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json:rw,z -v /var/lib/config-data/puppet-generated/mysql:/var/lib/kolla/config_files/src:ro,z -v /var/lib/mysql:/var/lib/mysql:rw,z -v /tmp/mariadb-upgrade:/var/log/mariadb:rw,z \"cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb:pcmklatest\" /bin/bash -ecx \"kolla_set_configs; if mysqladmin ping --silent; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown\"", "delta": "0:01:01.933135", "end": "2021-10-18 22:13:41.815918", "msg": "non-zero return code", "rc": 124, "start": "2021-10-18 22:12:39.882783", "stderr": "+ kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets\nINFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets\nINFO:__main__:Deleting /etc/my.cnf.d/galera.cnf\nINFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf\nINFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck\nINFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf\nINFO:__main__:Writing out command to execute\n+ mysqladmin ping --silent\n+ chown -R mysql:mysql /var/lib/mysql\n+ chown -R mysql:mysql /var/log/mariadb\n+ timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'\n+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off\n/usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option\nulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]", "stderr_lines": ["+ kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets", "INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets", "INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck", "INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf", "INFO:__main__:Writing out command to execute", "+ mysqladmin ping --silent", "+ chown -R mysql:mysql /var/lib/mysql", "+ chown -R mysql:mysql /var/log/mariadb", "+ timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'", "+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off", "/usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option", "ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]"], "stdout": "211018 22:12:41 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.\n211018 22:12:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql", "stdout_lines": ["211018 22:12:41 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.", "211018 22:12:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql"]}
=====

db upgrade.log
===

2021-10-18T19:30:28.000842383+00:00 stderr F + kolla_set_configs
2021-10-18T19:30:28.057513303+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
2021-10-18T19:30:28.073106728+00:00 stderr F INFO:__main__:Validating config file
2021-10-18T19:30:28.073138945+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
2021-10-18T19:30:28.073249957+00:00 stderr F INFO:__main__:Copying service configuration files
2021-10-18T19:30:28.073426804+00:00 stderr F INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets
2021-10-18T19:30:28.074292634+00:00 stderr F INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets
2021-10-18T19:30:28.075036288+00:00 stderr F INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf
2021-10-18T19:30:28.075297933+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf
2021-10-18T19:30:28.113359491+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck
2021-10-18T19:30:28.114171971+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf
2021-10-18T19:30:28.133941457+00:00 stderr F INFO:__main__:Writing out command to execute
2021-10-18T19:30:28.139521748+00:00 stderr F + mysqladmin ping --silent
2021-10-18T19:30:28.146077371+00:00 stderr F + chown -R mysql:mysql /var/lib/mysql
2021-10-18T19:30:28.154708787+00:00 stderr F + chown -R mysql:mysql /var/log/mariadb
2021-10-18T19:30:28.157003629+00:00 stderr F + timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'
2021-10-18T19:30:28.157003629+00:00 stderr F + mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off
2021-10-18T19:30:28.362029466+00:00 stdout F 211018 19:30:28 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
2021-10-18T19:30:28.362223846+00:00 stderr F /usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option
2021-10-18T19:30:28.362223846+00:00 stderr F ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]
2021-10-18T19:30:28.415781999+00:00 stdout F 211018 19:30:28 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2021-10-18T19:47:22.298286332+00:00 stderr F + kolla_set_configs
2021-10-18T19:47:22.351423054+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
2021-10-18T19:47:22.351624672+00:00 stderr F INFO:__main__:Validating config file
2021-10-18T19:47:22.351661191+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
2021-10-18T19:47:22.351751122+00:00 stderr F INFO:__main__:Copying service configuration files
2021-10-18T19:47:22.351863919+00:00 stderr F INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets
2021-10-18T19:47:22.352658648+00:00 stderr F INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets
2021-10-18T19:47:22.353403800+00:00 stderr F INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf
2021-10-18T19:47:22.353586985+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf
2021-10-18T19:47:22.354200874+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck
2021-10-18T19:47:22.354776569+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf
2021-10-18T19:47:22.355289235+00:00 stderr F INFO:__main__:Writing out command to execute
2021-10-18T19:47:22.360242559+00:00 stderr F + mysqladmin ping --silent
2021-10-18T19:47:22.365549100+00:00 stderr F + chown -R mysql:mysql /var/lib/mysql
2021-10-18T19:47:22.374229759+00:00 stderr F + chown -R mysql:mysql /var/log/mariadb
2021-10-18T19:47:22.377126900+00:00 stderr F + timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'
2021-10-18T19:47:22.377126900+00:00 stderr F + mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off
2021-10-18T19:47:22.594935297+00:00 stdout F 211018 19:47:22 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
2021-10-18T19:47:22.595222682+00:00 stderr F /usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option
2021-10-18T19:47:22.595222682+00:00 stderr F ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]
2021-10-18T19:47:22.649664590+00:00 stdout F 211018 19:47:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
~    
=========================

[root@overcloud-controller-0 cinder]# sudo podman ps
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
[root@overcloud-controller-0 cinder]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.2 (Ootpa)
[root@overcloud-controller-0 cinder]#
[root@overcloud-controller-0 cinder]# sudo podman ps -a
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
[root@overcloud-controller-0 cinder]# podman images
REPOSITORY                                                                                                                   TAG          IMAGE ID       CREATED        SIZE
uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-cinder-volume   16.1.6       4cae7176b5aa   2 months ago   1.17 GB
cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-cinder-volume                   pcmklatest   4cae7176b5aa   2 months ago   1.17 GB
uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb         16.1.6       f6dbb15de16e   2 months ago   718 MB
cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb                         pcmklatest   f6dbb15de16e   2 months ago   718 MB
cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-haproxy                         pcmklatest   cd1788711919   2 months ago   523 MB
uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-haproxy         16.1.6       cd1788711919   2 months ago   523 MB
[root@overcloud-controller-0 cinder]# 
===============

Comment 10 dabarzil 2021-11-18 07:48:38 UTC

FFU succuded on:
https://rhos-ci-staging-jenkins.lab.eng.tlv2.redhat.com/job/DFG-upgrades-ffu-16.2-from-13-latest_cdn-3cont_2comp_2net-ipv4-ovs_vlan-provider-network/
[stack@undercloud-0 ~]$ rpm -qa| grep openstack-tripleo-heat-template
openstack-tripleo-heat-templates-11.5.1-2.20210603174829.el8ost.10.noarch
[stack@undercloud-0 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.4 (Ootpa)

Comment 12 Jesse Pretorius 2021-11-26 13:11:24 UTC

For OSP version 16.2.0, the following workaround may be used:

# remove the mariadb-server package from the controllers
$ ansible -i ~/inventory.yaml Controller -b -m yum -a "name=mariadb-server state=absent"

This can happen during the steps performed in Chapter 8: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/framework_for_upgrades_13_to_16.2/configuring-the-overcloud-for-a-leapp-upgrade_preparing-overcloud

Comment 13 Jesse Pretorius 2021-11-26 18:16:31 UTC

A better workaround is to include 'sudo yum -y remove mariadb-server*’ in LeappInitCommand in 8.1: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/framework_for_upgrades_13_to_16.1/configuring-the-overcloud-for-a-leapp-upgrade_preparing-overcloud

That would ensure that it doesn't add more steps - it just includes more configuration.

Comment 18 Jesse Pretorius 2021-11-30 12:17:51 UTC

(In reply to Jesse Pretorius from comment #13)
> A better workaround is to include 'sudo yum -y remove mariadb-server*’ in
> LeappInitCommand in 8.1:
> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.
> 1/html/framework_for_upgrades_13_to_16.1/configuring-the-overcloud-for-a-
> leapp-upgrade_preparing-overcloud
> 
> That would ensure that it doesn't add more steps - it just includes more
> configuration.

Revised version, which is necessary in the case where there may be a failure that requires the re-run of the LeappInitCommand. This ensures that if the mariadb-server* packages are not present, the yum command doesn't fail and cause the whole script to fail.

parameter_defaults:
  LeappInitCommand: |
    sudo yum -y remove mariadb-server* || true

Comment 28 errata-xmlrpc 2021-12-09 20:41:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.1 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:5067