Bug 1564544

Summary: [UPGRADES] Error during galera-bundle upgrade - ulimit: -1: invalid option
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-tripleo-heat-templatesAssignee: Damien Ciabrini <dciabrin>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: high Docs Contact: Andrew Burden <aburden>
Priority: high    
Version: 13.0 (Queens)CC: abeekhof, aherr, augol, jschluet, mandreou, m.andre, mbultel, mburns, mcornea, mkrcmari
Target Milestone: betaKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.0.2-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:50:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Yurii Prokulevych 2018-04-06 15:15:11 UTC
Description of problem:
-----------------------
During Database role upgrade 'TASK [Upgrade Mysql database from a temporary container]' failed

openstack overcloud upgrade run \
                --nodes Database --playbook all 2>&1
...
u'TASK [Upgrade Mysql database from a temporary container] ***********************',
 u'fatal: [192.168.24.14]: FAILED! => {"changed": true, "cmd": "/usr/bin/docker run --rm --log-driver=syslog -u root --net=host -e \\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/puppet:/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json -v /var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro -v /var/lib/mysql:/var/lib/mysql -v /tmp/mariadb-upgrade:/var/log/mariadb:rw \\"192.168.24.1:8787/rhosp13/openstack-mariadb:pcmklatest\\" /bin/bash -ecx \\"sudo -E kolla_set_configs;  mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off & timeout 60 sh -c \'while ! mysqladmin ping --silent; do sleep 1; done\'; mysql_upgrade; mysqladmin shutdown\\"", "delta": "0:01:00.349754", "end": "2018-04-06 13:58:00.615572", "msg": "non-zero return code", "rc": 124, "start": "2018-04-06 13:57:00.265818", "stderr": "+ sudo -E kolla_set_configs\\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\\nINFO:__main__:Copying service configuration files\\nINFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets\\nINFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets\\nINFO:__main__:Deleting /etc/my.cnf.d/galera.cnf\\nINFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf\\nINFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck\\nINFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf\\nINFO:__main__:Writing out command to execute\\n+ timeout 60 sh -c \'while ! mysqladmin ping --silent; do sleep 1; done\'\\n+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off\\n/usr/bin/mysqld_safe: line 755: ulimit: -1: invalid option\\nulimit: usage: ulimit [-SHacdefilmnpqrstuvx] [limit]", "stderr_lines": ["+ sudo -E kolla_set_configs", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets", "INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets", "INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck", "INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf", "INFO:__main__:Writing out command to execute", "+ timeout 60 sh -c \'while ! mysqladmin ping --silent; do sleep 1; done\'", "+ mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off", "/usr/bin/mysqld_safe: line 755: ulimi'headers'
t: -1: invalid option", "ulimit: usage: ulimit [-SHacdefilmnpqrstuvx] [limit]"], "stdout": "180406 13:57:00 mysqld_safe Logging to \'/var/log/mariadb/mariadb.log\'.\\n180406 13:57:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql\\n180406 13:57:01 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended", "stdout_lines": ["180406 13:57:00 mysqld_safe Logging to \'/var/log/mariadb/mariadb.log\'.", "180406 13:57:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql", "180406 13:57:01 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended"]}']


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-tripleo-heat-templates-8.0.2-0.20180327213843.f25e2d8.el7ost.noarch
192.168.24.1:8787/rhosp13/openstack-mariadb:2018-03-29.1

How reproducible:
-----------------
So far 100%


Steps to Reproduce:
-------------------
1. Upgrade UC
2. Setup latest repos on OC
3. Setup latest containers
4. Run 'overcloud upgrade prepare...'
5. Upgrade Controller role
6. Start upgrade of Database role

Additional info:
----------------

Virtual environment: 3controller + 2compute + 3database + 3messaging + 3ceph + 2networker

Comment 2 Damien Ciabrini 2018-04-17 07:26:35 UTC
This one is a warning due to the way we generate the galera config in OSP. It is not fatal.

The failure was probably due to other is a previous revision of https://review.openstack.org/#/c/560408, and should be fixed in the latest revisions.

Comment 3 Damien Ciabrini 2018-04-23 13:16:06 UTC
Updated patch to point to queens branch. merged end of last week

Comment 9 errata-xmlrpc 2018-06-27 13:50:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086