Bug 1776761
| Summary: | Mysql Upgrade tasks failing with error: reading container (probably exited) json message: EOF | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jose Luis Franco <jfrancoa> |
| Component: | openstack-tripleo-heat-templates | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | Sasha Smolyak <ssmolyak> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 15.0 (Stein) | CC: | apetrich, mburns |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-11-26 17:20:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Ran the failing command in the overcloud node (controller-0) and this was the output: [heat-admin@controller-0 ~]$ sudo podman run --rm --log-driver=k8s-file --log-opt path=LOG_DIR/db-upgrade.log -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/puppet:/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json:rw,z -v /var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro,z -v /var/lib/mysql:/var/lib/mysql:rw,z -v /tmp/mariadb-upgrade:/var/log/mariadb:rw,z \"192.168.24.1:8787/rh-osbs/rhosp15-openstack-mariadb:pcmklatest\" /bin/bash -ecx \"kolla_set_configs; if mysqladmin ping --silent; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown\" unable to pull "192.168.24.1:8787/rh-osbs/rhosp15-openstack-mariadb:pcmklatest": error getting default registries to try: invalid reference format chown: cannot read directory '/var/lib/mysql/mysql': Permission denied chown: changing ownership of '/var/lib/mysql/aria_log_control': Operation not permitted chown: changing ownership of '/var/lib/mysql/aria_log.00000001': Operation not permitted chown: changing ownership of '/var/lib/mysql/ibdata1': Operation not permitted chown: changing ownership of '/var/lib/mysql/ib_logfile1': Operation not permitted chown: changing ownership of '/var/lib/mysql/ib_logfile0': Operation not permitted chown: cannot read directory '/var/lib/mysql/performance_schema': Permission denied chown: changing ownership of '/var/lib/mysql/multi-master.info': Operation not permitted chown: changing ownership of '/var/lib/mysql/grastate.dat': Operation not permitted chown: changing ownership of '/var/lib/mysql/galera.cache': Operation not permitted chown: cannot read directory '/var/lib/mysql/aodh': Permission denied chown: cannot read directory '/var/lib/mysql/cinder': Permission denied chown: cannot read directory '/var/lib/mysql/glance': Permission denied chown: cannot read directory '/var/lib/mysql/gnocchi': Permission denied chown: cannot read directory '/var/lib/mysql/heat': Permission denied chown: cannot read directory '/var/lib/mysql/keystone': Permission denied chown: cannot read directory '/var/lib/mysql/nova': Permission denied chown: cannot read directory '/var/lib/mysql/nova_api': Permission denied chown: cannot read directory '/var/lib/mysql/nova_cell0': Permission denied chown: cannot read directory '/var/lib/mysql/nova_placement': Permission denied chown: cannot read directory '/var/lib/mysql/ovs_neutron': Permission denied chown: cannot read directory '/var/lib/mysql/panko': Permission denied chown: changing ownership of '/var/lib/mysql': Operation not permitted chown: cannot read directory '/var/log/mariadb': Permission denied [1] 31305 191126 17:06:46 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 191126 17:06:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql [1]+ Done mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off Version check failed. Got the following error when calling the 'mysql' command line client ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) FATAL ERROR: Upgrade failed mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)' Check that mysqld is running and that the socket: '/var/lib/mysql/mysql.sock' exists! *** This bug has been marked as a duplicate of bug 1769291 *** |
Description of problem: The OSP14 to OSP15 upgrade procedure is failing in one of the upgrade tasks, te one upgrading MySQL: 2019-11-25 17:40:28 | TASK [Make sure Mysql upgrade temporary directory exists] ********************** 2019-11-25 17:40:28 | Monday 25 November 2019 17:40:25 -0500 (0:00:00.102) 0:12:07.504 ******* 2019-11-25 17:40:28 | changed: [controller-0] => {"changed": true, "gid": 0, "group": "root", "mode": "0700", "owner": "root", "path": "/tmp/mariadb-upgrade", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0} 2019-11-25 17:40:28 | 2019-11-25 17:40:28 | TASK [Upgrade Mysql database from a temporary container] *********************** 2019-11-25 17:40:28 | Monday 25 November 2019 17:40:26 -0500 (0:00:00.425) 0:12:07.930 ******* 2019-11-25 17:40:28 | fatal: [controller-0]: FAILED! => {"changed": true, "cmd": "podman run --rm --log-driver=k8s-file --log-opt path=LOG_DIR/db-upgrade.log -u root --net=host -e \"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\" -v /etc/hosts:/etc/hosts:ro -v /etc/localtime:/etc/localtime:ro -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro -v /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro -v /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro -v /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro -v /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro -v /dev/log:/dev/log -v /etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro -v /etc/puppet:/etc/puppet:ro -v /var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json:rw,z -v /var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro,z -v /var/lib/mysql:/var/lib/mysql:rw,z -v /tmp/mariadb-upgrade:/var/log/mariadb:rw,z \"192.168.24.1:8787/rh-osbs/rhosp15-openstack-mariadb:pcmklatest\" /bin/bash -ecx \"kolla_set_configs; if mysqladmin ping --silent; then exit 0; fi; chown -R mysql:mysql /var/lib/mysql; chown -R mysql:mysql /var/log/mariadb; mysqld_safe --user=mysql --wsrep-provider=none --skip-networking --wsrep-on=off & timeout 60 sh -c 'while ! mysqladmin ping --silent; do sleep 1; done'; mysql_upgrade; mysqladmin shutdown\"", "delta": "0:00:00.290162", "end": "2019-11-25 22:40:26.421552", "msg": "non-zero return code", "rc": 127, "start": "2019-11-25 22:40:26.131390", "stderr": "error reading container (probably exited) json message: EOF", "stderr_lines": ["error reading container (probably exited) json message: EOF"], "stdout": "", "stdout_lines": []} 2019-11-25 17:40:28 | 2019-11-25 17:40:28 | NO MORE HOSTS LEFT ************************************************************* 2019-11-25 17:40:28 | 2019-11-25 17:40:28 | PLAY RECAP ********************************************************************* 2019-11-25 17:40:28 | controller-0 : ok=142 changed=61 unreachable=0 failed=1 skipped=239 rescued=0 ignored=1 2019-11-25 17:40:28 | 2019-11-25 17:40:28 | Monday 25 November 2019 17:40:26 -0500 (0:00:00.646) 0:12:08.577 ******* 2019-11-25 17:40:28 | =============================================================================== Version-Release number of selected component (if applicable): How reproducible: Run Upgrades job: http://staging-jenkins2-qe-playground.usersys.redhat.com/view/DFG/view/upgrades/view/upgrade/job/DFG-upgrades-upgrade-upgrade-14-15_director-rhel-virthost-3cont_2comp-ipv4-vxlan/ Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: