This bug was initially created as a copy of Bug #2015325 (the same bug and fix applies) Description of problem: As part of FFU upgrade (13 to 16.2), OpenStack node upgrade failed. The OS has been upgraded from 7.7 to 8.4 TASK [Make sure Mysql upgrade temporary directory exists] ********************** Monday 18 October 2021 18:12:37 -0400 (0:00:00.070) 0:02:32.120 ******** changed: [overcloud-controller-0] => {"changed": true, "gid": 0, "group": "root", "mode": "0700", "owner": "root", "path": "/tmp/mariadb-upgrade", "secontext": "system_u:object_r:container_file_t:s0", "size": 25, "state": "directory", "uid": 0} TASK [Upgrade Mysql database from a temporary container] *********************** Monday 18 October 2021 18:12:37 -0400 (0:00:00.323) 0:02:32.443 ******** fatal: [overcloud-controller-0]: FAILED! => {"changed": true, "cmd": "podman run --rm --log-driver=k8s-file --log-opt path=/var/log/containers/mysql/db-upgrade.log -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ipa/ca.crt:/etc/ipa/ca.crt:ro -v /etc/puppet:/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json:rw,z -v /var/lib/config-data/puppet-generated/mysql:/var/lib/kolla/config_files/src:ro,z -v /var/lib/mysql:/var/lib/mysql:rw,z -v /tmp/mariadb-upgrade:/var/log/mariadb:rw,z \"cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb:pcmklatest\" /bin/bash -ecx \"kolla_set_configs; if mysqladmin ping --silent; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown\"", "delta": "0:01:01.933135", "end": "2021-10-18 22:13:41.815918", "msg": "non-zero return code", "rc": 124, "start": "2021-10-18 22:12:39.882783", "stderr": "+ kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets\nINFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets\nINFO:__main__:Deleting /etc/my.cnf.d/galera.cnf\nINFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf\nINFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck\nINFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf\nINFO:__main__:Writing out command to execute\n+ mysqladmin ping --silent\n+ chown -R mysql:mysql /var/lib/mysql\n+ chown -R mysql:mysql /var/log/mariadb\n+ timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'\n+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off\n/usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option\nulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]", "stderr_lines": ["+ kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets", "INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets", "INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck", "INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf", "INFO:__main__:Writing out command to execute", "+ mysqladmin ping --silent", "+ chown -R mysql:mysql /var/lib/mysql", "+ chown -R mysql:mysql /var/log/mariadb", "+ timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'", "+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off", "/usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option", "ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]"], "stdout": "211018 22:12:41 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.\n211018 22:12:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql", "stdout_lines": ["211018 22:12:41 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.", "211018 22:12:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql"]} ===== db upgrade.log === 2021-10-18T19:30:28.000842383+00:00 stderr F + kolla_set_configs 2021-10-18T19:30:28.057513303+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json 2021-10-18T19:30:28.073106728+00:00 stderr F INFO:__main__:Validating config file 2021-10-18T19:30:28.073138945+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS 2021-10-18T19:30:28.073249957+00:00 stderr F INFO:__main__:Copying service configuration files 2021-10-18T19:30:28.073426804+00:00 stderr F INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets 2021-10-18T19:30:28.074292634+00:00 stderr F INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets 2021-10-18T19:30:28.075036288+00:00 stderr F INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf 2021-10-18T19:30:28.075297933+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf 2021-10-18T19:30:28.113359491+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck 2021-10-18T19:30:28.114171971+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf 2021-10-18T19:30:28.133941457+00:00 stderr F INFO:__main__:Writing out command to execute 2021-10-18T19:30:28.139521748+00:00 stderr F + mysqladmin ping --silent 2021-10-18T19:30:28.146077371+00:00 stderr F + chown -R mysql:mysql /var/lib/mysql 2021-10-18T19:30:28.154708787+00:00 stderr F + chown -R mysql:mysql /var/log/mariadb 2021-10-18T19:30:28.157003629+00:00 stderr F + timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done' 2021-10-18T19:30:28.157003629+00:00 stderr F + mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off 2021-10-18T19:30:28.362029466+00:00 stdout F 211018 19:30:28 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 2021-10-18T19:30:28.362223846+00:00 stderr F /usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option 2021-10-18T19:30:28.362223846+00:00 stderr F ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit] 2021-10-18T19:30:28.415781999+00:00 stdout F 211018 19:30:28 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 2021-10-18T19:47:22.298286332+00:00 stderr F + kolla_set_configs 2021-10-18T19:47:22.351423054+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json 2021-10-18T19:47:22.351624672+00:00 stderr F INFO:__main__:Validating config file 2021-10-18T19:47:22.351661191+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS 2021-10-18T19:47:22.351751122+00:00 stderr F INFO:__main__:Copying service configuration files 2021-10-18T19:47:22.351863919+00:00 stderr F INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets 2021-10-18T19:47:22.352658648+00:00 stderr F INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets 2021-10-18T19:47:22.353403800+00:00 stderr F INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf 2021-10-18T19:47:22.353586985+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf 2021-10-18T19:47:22.354200874+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck 2021-10-18T19:47:22.354776569+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf 2021-10-18T19:47:22.355289235+00:00 stderr F INFO:__main__:Writing out command to execute 2021-10-18T19:47:22.360242559+00:00 stderr F + mysqladmin ping --silent 2021-10-18T19:47:22.365549100+00:00 stderr F + chown -R mysql:mysql /var/lib/mysql 2021-10-18T19:47:22.374229759+00:00 stderr F + chown -R mysql:mysql /var/log/mariadb 2021-10-18T19:47:22.377126900+00:00 stderr F + timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done' 2021-10-18T19:47:22.377126900+00:00 stderr F + mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off 2021-10-18T19:47:22.594935297+00:00 stdout F 211018 19:47:22 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 2021-10-18T19:47:22.595222682+00:00 stderr F /usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option 2021-10-18T19:47:22.595222682+00:00 stderr F ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit] 2021-10-18T19:47:22.649664590+00:00 stdout F 211018 19:47:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql ~ ========================= [root@overcloud-controller-0 cinder]# sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@overcloud-controller-0 cinder]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.2 (Ootpa) [root@overcloud-controller-0 cinder]# [root@overcloud-controller-0 cinder]# sudo podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@overcloud-controller-0 cinder]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-cinder-volume 16.1.6 4cae7176b5aa 2 months ago 1.17 GB cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-cinder-volume pcmklatest 4cae7176b5aa 2 months ago 1.17 GB uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb 16.1.6 f6dbb15de16e 2 months ago 718 MB cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb pcmklatest f6dbb15de16e 2 months ago 718 MB cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-haproxy pcmklatest cd1788711919 2 months ago 523 MB uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-haproxy 16.1.6 cd1788711919 2 months ago 523 MB [root@overcloud-controller-0 cinder]# ===============
FFU succuded on: https://rhos-ci-staging-jenkins.lab.eng.tlv2.redhat.com/job/DFG-upgrades-ffu-16.2-from-13-latest_cdn-3cont_2comp_2net-ipv4-ovs_vlan-provider-network/ [stack@undercloud-0 ~]$ rpm -qa| grep openstack-tripleo-heat-template openstack-tripleo-heat-templates-11.5.1-2.20210603174829.el8ost.10.noarch [stack@undercloud-0 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.4 (Ootpa)
For OSP version 16.2.0, the following workaround may be used: # remove the mariadb-server package from the controllers $ ansible -i ~/inventory.yaml Controller -b -m yum -a "name=mariadb-server state=absent" This can happen during the steps performed in Chapter 8: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/framework_for_upgrades_13_to_16.2/configuring-the-overcloud-for-a-leapp-upgrade_preparing-overcloud
A better workaround is to include 'sudo yum -y remove mariadb-server*’ in LeappInitCommand in 8.1: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/framework_for_upgrades_13_to_16.1/configuring-the-overcloud-for-a-leapp-upgrade_preparing-overcloud That would ensure that it doesn't add more steps - it just includes more configuration.
(In reply to Jesse Pretorius from comment #13) > A better workaround is to include 'sudo yum -y remove mariadb-server*’ in > LeappInitCommand in 8.1: > https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16. > 1/html/framework_for_upgrades_13_to_16.1/configuring-the-overcloud-for-a- > leapp-upgrade_preparing-overcloud > > That would ensure that it doesn't add more steps - it just includes more > configuration. Revised version, which is necessary in the case where there may be a failure that requires the re-run of the LeappInitCommand. This ensures that if the mariadb-server* packages are not present, the yum command doesn't fail and cause the whole script to fail. parameter_defaults: LeappInitCommand: | sudo yum -y remove mariadb-server* || true
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.1 (Train)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:5067