Created attachment 1834429 [details] db-upgrade.log Description of problem: As part of FFU upgrade (13 to 16.1), OpenStack node upgrade failed. The OS has been upgraded from 7.7 to 8.2 TASK [Make sure Mysql upgrade temporary directory exists] ********************** Monday 18 October 2021 18:12:37 -0400 (0:00:00.070) 0:02:32.120 ******** changed: [overcloud-controller-0] => {"changed": true, "gid": 0, "group": "root", "mode": "0700", "owner": "root", "path": "/tmp/mariadb-upgrade", "secontext": "system_u:object_r:container_file_t:s0", "size": 25, "state": "directory", "uid": 0} TASK [Upgrade Mysql database from a temporary container] *********************** Monday 18 October 2021 18:12:37 -0400 (0:00:00.323) 0:02:32.443 ******** fatal: [overcloud-controller-0]: FAILED! => {"changed": true, "cmd": "podman run --rm --log-driver=k8s-file --log-opt path=/var/log/containers/mysql/db-upgrade.log -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ipa/ca.crt:/etc/ipa/ca.crt:ro -v /etc/puppet:/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json:rw,z -v /var/lib/config-data/puppet-generated/mysql:/var/lib/kolla/config_files/src:ro,z -v /var/lib/mysql:/var/lib/mysql:rw,z -v /tmp/mariadb-upgrade:/var/log/mariadb:rw,z \"cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb:pcmklatest\" /bin/bash -ecx \"kolla_set_configs; if mysqladmin ping --silent; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown\"", "delta": "0:01:01.933135", "end": "2021-10-18 22:13:41.815918", "msg": "non-zero return code", "rc": 124, "start": "2021-10-18 22:12:39.882783", "stderr": "+ kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets\nINFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets\nINFO:__main__:Deleting /etc/my.cnf.d/galera.cnf\nINFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf\nINFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck\nINFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf\nINFO:__main__:Writing out command to execute\n+ mysqladmin ping --silent\n+ chown -R mysql:mysql /var/lib/mysql\n+ chown -R mysql:mysql /var/log/mariadb\n+ timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'\n+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off\n/usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option\nulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]", "stderr_lines": ["+ kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets", "INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets", "INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck", "INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf", "INFO:__main__:Writing out command to execute", "+ mysqladmin ping --silent", "+ chown -R mysql:mysql /var/lib/mysql", "+ chown -R mysql:mysql /var/log/mariadb", "+ timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'", "+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off", "/usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option", "ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]"], "stdout": "211018 22:12:41 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.\n211018 22:12:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql", "stdout_lines": ["211018 22:12:41 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.", "211018 22:12:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql"]} ===== db upgrade.log === 2021-10-18T19:30:28.000842383+00:00 stderr F + kolla_set_configs 2021-10-18T19:30:28.057513303+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json 2021-10-18T19:30:28.073106728+00:00 stderr F INFO:__main__:Validating config file 2021-10-18T19:30:28.073138945+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS 2021-10-18T19:30:28.073249957+00:00 stderr F INFO:__main__:Copying service configuration files 2021-10-18T19:30:28.073426804+00:00 stderr F INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets 2021-10-18T19:30:28.074292634+00:00 stderr F INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets 2021-10-18T19:30:28.075036288+00:00 stderr F INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf 2021-10-18T19:30:28.075297933+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf 2021-10-18T19:30:28.113359491+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck 2021-10-18T19:30:28.114171971+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf 2021-10-18T19:30:28.133941457+00:00 stderr F INFO:__main__:Writing out command to execute 2021-10-18T19:30:28.139521748+00:00 stderr F + mysqladmin ping --silent 2021-10-18T19:30:28.146077371+00:00 stderr F + chown -R mysql:mysql /var/lib/mysql 2021-10-18T19:30:28.154708787+00:00 stderr F + chown -R mysql:mysql /var/log/mariadb 2021-10-18T19:30:28.157003629+00:00 stderr F + timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done' 2021-10-18T19:30:28.157003629+00:00 stderr F + mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off 2021-10-18T19:30:28.362029466+00:00 stdout F 211018 19:30:28 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 2021-10-18T19:30:28.362223846+00:00 stderr F /usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option 2021-10-18T19:30:28.362223846+00:00 stderr F ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit] 2021-10-18T19:30:28.415781999+00:00 stdout F 211018 19:30:28 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 2021-10-18T19:47:22.298286332+00:00 stderr F + kolla_set_configs 2021-10-18T19:47:22.351423054+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json 2021-10-18T19:47:22.351624672+00:00 stderr F INFO:__main__:Validating config file 2021-10-18T19:47:22.351661191+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS 2021-10-18T19:47:22.351751122+00:00 stderr F INFO:__main__:Copying service configuration files 2021-10-18T19:47:22.351863919+00:00 stderr F INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets 2021-10-18T19:47:22.352658648+00:00 stderr F INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets 2021-10-18T19:47:22.353403800+00:00 stderr F INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf 2021-10-18T19:47:22.353586985+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf 2021-10-18T19:47:22.354200874+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck 2021-10-18T19:47:22.354776569+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf 2021-10-18T19:47:22.355289235+00:00 stderr F INFO:__main__:Writing out command to execute 2021-10-18T19:47:22.360242559+00:00 stderr F + mysqladmin ping --silent 2021-10-18T19:47:22.365549100+00:00 stderr F + chown -R mysql:mysql /var/lib/mysql 2021-10-18T19:47:22.374229759+00:00 stderr F + chown -R mysql:mysql /var/log/mariadb 2021-10-18T19:47:22.377126900+00:00 stderr F + timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done' 2021-10-18T19:47:22.377126900+00:00 stderr F + mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off 2021-10-18T19:47:22.594935297+00:00 stdout F 211018 19:47:22 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 2021-10-18T19:47:22.595222682+00:00 stderr F /usr/bin/mysqld_safe: line 696: ulimit: -1: invalid option 2021-10-18T19:47:22.595222682+00:00 stderr F ulimit: usage: ulimit [-SHabcdefiklmnpqrstuvxPT] [limit] 2021-10-18T19:47:22.649664590+00:00 stdout F 211018 19:47:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql ~ ========================= [root@overcloud-controller-0 cinder]# sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@overcloud-controller-0 cinder]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.2 (Ootpa) [root@overcloud-controller-0 cinder]# [root@overcloud-controller-0 cinder]# sudo podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@overcloud-controller-0 cinder]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-cinder-volume 16.1.6 4cae7176b5aa 2 months ago 1.17 GB cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-cinder-volume pcmklatest 4cae7176b5aa 2 months ago 1.17 GB uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb 16.1.6 f6dbb15de16e 2 months ago 718 MB cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-mariadb pcmklatest f6dbb15de16e 2 months ago 718 MB cluster.common.tag/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-haproxy pcmklatest cd1788711919 2 months ago 523 MB uc-train.ctlplane.5b6s1.local:8787/5c5s45-satellite-rhosp-16-rhel-8-product-containers-rhosp-rhel8_openstack-haproxy 16.1.6 cd1788711919 2 months ago 523 MB [root@overcloud-controller-0 cinder]# =============== Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I don't think "ulimit: -1: invalid option" is the real issue here. In our labs we see the same error and FFU completes just fine. There is probably something else happening that we can't see from this log snippet. May I ask you to open a support ticket and attach sosreports (please reference this bz so we can follow up), or upload them here. Thanks.
FFU succeded on: https://rhos-ci-staging-jenkins.lab.eng.tlv2.redhat.com/job/DFG-upgrades-ffu-16.1-from-13-latest_cdn-3cont_2comp_2net-ipv4-ovs_vlan-provider-network/20/ [stack@undercloud-0 ~]$ rpm -qa |grep openstack-tripleo-heat-temp openstack-tripleo-heat-templates-11.3.2-1.20210720153315.el8ost.noarch [stack@undercloud-0 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.2 (Ootpa)
For OSP versions 16.1.0 to 16.1.6, the following workaround may be used: # remove the mariadb-server package from the controllers $ ansible -i ~/inventory.yaml Controller -b -m yum -a "name=mariadb-server state=absent" This can happen during the steps performed in Chapter 8: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/framework_for_upgrades_13_to_16.1/configuring-the-overcloud-for-a-leapp-upgrade_preparing-overcloud
A better workaround is to include 'yum -y remove mariadb-server*’ in LeappInitCommand in 8.1: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/framework_for_upgrades_13_to_16.1/configuring-the-overcloud-for-a-leapp-upgrade_preparing-overcloud That would ensure that it doesn't add more steps - it just includes more configuration.
Hmm, that should read 'sudo yum -y remove ...'
*** Bug 1985067 has been marked as a duplicate of this bug. ***
Hi Jesse, An RHOSP Doc team script extracts the contents of the "Doc Text" field in this BZ for use in the RHOSP release notes. I have edited the "Doc Text" contents to conform to Red Hat style guidelines. (See below). Please add a comment to this BZ indicating whether my doc text edits: 1. are OK, or 2. require some changes. If condition (2) is true, then please provide the required changes. Thanks for your help with this, --Greg PROPOSED DOC TEXT EDIT ---------------------- Before this update, an upgradable `mariadb-server` package in the RHEL repository caused the `rpm` command to upgrade the `mariadb-server` package on the host, interfering with the containerized `mariadb-server` that pre-exists on the same host. With this update, the Red Hat OpenStack Platform (RHOSP) director removes the `mariadb-server` package from any hosts which also have the containerized MariaDB and the RHOSP FFU process continues.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3762
*** Bug 2028505 has been marked as a duplicate of this bug. ***