Bug 1470795 - OSP11 -> OSP12 upgrade: mysql_init_bundle container fails after 30 minutes: Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 ins [NEEDINFO]
OSP11 -> OSP12 upgrade: mysql_init_bundle container fails after 30 minutes: E...
Status: MODIFIED
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
high Severity urgent
: ga
: 12.0 (Pike)
Assigned To: Emilien Macchi
Gurenko Alex
: Triaged
Depends On:
Blocks: 1399762
  Show dependency treegraph
 
Reported: 2017-07-13 12:53 EDT by Marius Cornea
Modified: 2017-08-28 11:25 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
ccamacho: needinfo? (augol)


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1706594 None None None 2017-07-26 08:42 EDT
OpenStack gerrit 487313 None None None 2017-07-26 08:44 EDT

  None (edit)
Description Marius Cornea 2017-07-13 12:53:06 EDT
Description of problem:
OSP11 -> OSP12 upgrade: mysql_init_bundle container fails after 30 minutes: 

2017-07-13T16:17:21.054702000Z Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch with  https://review.openstack.org/480202/ applied

How reproducible:
100$

Steps to Reproduce:
1. Deploy OSP11:

#!/bin/bash

openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
--log-file overcloud_deployment_11.log

2. Upgrade undercloud

3. Upgrade overcloud:
openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \

Actual results:
Upgrade fails because puppet failed to run on mysql_init_bundle container:

[root@controller-0 heat-admin]# docker logs mysql_init_bundle -t
2017-07-13T15:46:34.873177000Z Info: Loading facts
2017-07-13T15:46:34.873474000Z Info: Loading facts
2017-07-13T15:46:34.873657000Z Info: Loading facts
2017-07-13T15:46:34.873829000Z Info: Loading facts
2017-07-13T15:46:34.874010000Z Info: Loading facts
2017-07-13T15:46:34.874208000Z Info: Loading facts
2017-07-13T15:46:34.874365000Z Info: Loading facts
2017-07-13T15:46:34.874517000Z Info: Loading facts
2017-07-13T15:46:34.874665000Z Info: Loading facts
2017-07-13T15:46:34.874865000Z Info: Loading facts
2017-07-13T15:46:34.875051000Z Info: Loading facts
2017-07-13T15:46:34.875211000Z Info: Loading facts
2017-07-13T15:46:34.875367000Z Info: Loading facts
2017-07-13T15:46:34.875508000Z Info: Loading facts
2017-07-13T15:46:34.875667000Z Info: Loading facts
2017-07-13T15:46:34.875842000Z Info: Loading facts
2017-07-13T15:46:34.875998000Z Info: Loading facts
2017-07-13T15:46:34.876159000Z Info: Loading facts
2017-07-13T15:46:34.876298000Z Info: Loading facts
2017-07-13T15:46:34.876493000Z Info: Loading facts
2017-07-13T15:46:34.876634000Z Info: Loading facts
2017-07-13T15:46:34.876809000Z Info: Loading facts
2017-07-13T15:46:34.876950000Z Info: Loading facts
2017-07-13T15:46:37.286789000Z Failed to get D-Bus connection: Operation not permitted
2017-07-13T15:46:37.420199000Z Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
2017-07-13T15:46:37.420527000Z Warning: Undefined variable 'deploy_config_name'; 
2017-07-13T15:46:37.420723000Z    (file & line not available)
2017-07-13T15:46:37.512607000Z Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
2017-07-13T15:46:37.795705000Z Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql_bundle.pp", 75]:["unknown", 1]
2017-07-13T15:46:37.796054000Z    (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
2017-07-13T15:46:37.830408000Z Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp", 92]:["unknown", 1]
2017-07-13T15:46:37.830593000Z    (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
2017-07-13T15:46:37.900289000Z Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
2017-07-13T15:46:37.900518000Z    (file & line not available)
2017-07-13T15:46:38.206703000Z Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/aodh/manifests/db/mysql.pp", 58]:["/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp", 160]
2017-07-13T15:46:38.207016000Z    (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
2017-07-13T15:46:39.556095000Z Warning: Unknown variable: '::nova::db::mysql_api::setup_cell0'. at /etc/puppet/modules/nova/manifests/db/mysql.pp:53:28
2017-07-13T15:46:39.765052000Z Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
2017-07-13T15:46:39.765251000Z    (file & line not available)
2017-07-13T15:46:39.903546000Z Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp", 43]:
2017-07-13T15:46:39.903749000Z    (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
2017-07-13T15:46:40.138751000Z Notice: Compiled catalog for controller-0.localdomain in environment production in 2.83 seconds
2017-07-13T15:46:40.374750000Z Info: Applying configuration version '1499960797'
2017-07-13T15:46:40.554134000Z Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker]/ensure: created
2017-07-13T15:46:40.557509000Z Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker-authkey]/ensure: defined content as '{md5}ff554c5032de56dfc1805ac668cf1ac7'
2017-07-13T15:46:40.559010000Z Info: Class[Pacemaker::Corosync]: Unscheduling all events on Class[Pacemaker::Corosync]
2017-07-13T15:46:40.562144000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]/ensure: defined content as '{md5}d78f65c211f4f42b711b497ea257d0d7'
2017-07-13T15:46:40.564699000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/etc/sysconfig/clustercheck]/ensure: defined content as '{md5}955258872109af7f433be323ec2ba78e'
2017-07-13T15:46:40.567583000Z Info: Computing checksum on file /etc/my.cnf.d/galera.cnf
2017-07-13T15:46:40.585310000Z Info: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf.d/galera.cnf to puppet with sum f434e1d5766874c7b9ed08d0c66904ca
2017-07-13T15:46:40.587247000Z Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}f434e1d5766874c7b9ed08d0c66904ca' to '{md5}9f5cf53819f1724916edac6c7f5d10a3'
2017-07-13T15:46:40.587492000Z Info: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Scheduling refresh of Tripleo::Pacemaker::Resource_restart_flag[galera-master]
2017-07-13T15:46:40.587811000Z Info: Tripleo::Pacemaker::Resource_restart_flag[galera-master]: Unscheduling all events on Tripleo::Pacemaker::Resource_restart_flag[galera-master]
2017-07-13T15:46:40.588085000Z Info: Class[Mysql::Server::Config]: Unscheduling all events on Class[Mysql::Server::Config]
2017-07-13T15:46:40.591260000Z Notice: /Stage[main]/Mysql::Server::Installdb/File[/var/log/mariadb/mariadb.log]/ensure: created
2017-07-13T15:46:40.591458000Z Info: Class[Mysql::Server::Installdb]: Unscheduling all events on Class[Mysql::Server::Installdb]
2017-07-13T15:46:41.638956000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Tripleo::Pacemaker::Resource_restart_flag[galera-master]/File[/var/lib/tripleo]/ensure: created
2017-07-13T15:46:41.645866000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Tripleo::Pacemaker::Resource_restart_flag[galera-master]/File[/var/lib/tripleo/pacemaker-restarts]/ensure: created
2017-07-13T15:46:41.646059000Z Info: Tripleo::Pacemaker::Resource_restart_flag[galera-master]: Unscheduling all events on Tripleo::Pacemaker::Resource_restart_flag[galera-master]
2017-07-13T15:46:53.153068000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]/ensure: created
2017-07-13T15:46:53.153328000Z Info: Pacemaker::Resource::Bundle[galera-bundle]: Unscheduling all events on Pacemaker::Resource::Bundle[galera-bundle]
2017-07-13T15:47:00.111800000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]/ensure: created
2017-07-13T15:47:00.112063000Z Info: Pacemaker::Resource::Ocf[galera]: Unscheduling all events on Pacemaker::Resource::Ocf[galera]
2017-07-13T16:17:21.036663000Z Error: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
2017-07-13T16:17:21.037003000Z Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
2017-07-13T16:17:21.037203000Z Info: Class[Tripleo::Profile::Pacemaker::Database::Mysql_bundle]: Unscheduling all events on Class[Tripleo::Profile::Pacemaker::Database::Mysql_bundle]
2017-07-13T16:17:21.046275000Z Info: Creating state file /var/lib/puppet/state/state.yaml
2017-07-13T16:17:21.054702000Z Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

Expected results:
puppet runs correctly and the upgrade process moves forward

Additional info:
Comment 1 Marius Cornea 2017-07-13 13:04:55 EDT
openstack stack failures list overcloud:

http://paste.openstack.org/show/615300/

Note You need to log in before you can comment on or make changes to this bug.