Hide Forgot
Description of problem: OSP11 -> OSP12 upgrade: mysql_init_bundle container fails after 30 minutes: 2017-07-13T16:17:21.054702000Z Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-7.0.0-0.20170710191337.el7ost.noarch with https://review.openstack.org/480202/ applied How reproducible: 100$ Steps to Reproduce: 1. Deploy OSP11: #!/bin/bash openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ --log-file overcloud_deployment_11.log 2. Upgrade undercloud 3. Upgrade overcloud: openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \ -e /home/stack/docker-osp12.yaml \ Actual results: Upgrade fails because puppet failed to run on mysql_init_bundle container: [root@controller-0 heat-admin]# docker logs mysql_init_bundle -t 2017-07-13T15:46:34.873177000Z Info: Loading facts 2017-07-13T15:46:34.873474000Z Info: Loading facts 2017-07-13T15:46:34.873657000Z Info: Loading facts 2017-07-13T15:46:34.873829000Z Info: Loading facts 2017-07-13T15:46:34.874010000Z Info: Loading facts 2017-07-13T15:46:34.874208000Z Info: Loading facts 2017-07-13T15:46:34.874365000Z Info: Loading facts 2017-07-13T15:46:34.874517000Z Info: Loading facts 2017-07-13T15:46:34.874665000Z Info: Loading facts 2017-07-13T15:46:34.874865000Z Info: Loading facts 2017-07-13T15:46:34.875051000Z Info: Loading facts 2017-07-13T15:46:34.875211000Z Info: Loading facts 2017-07-13T15:46:34.875367000Z Info: Loading facts 2017-07-13T15:46:34.875508000Z Info: Loading facts 2017-07-13T15:46:34.875667000Z Info: Loading facts 2017-07-13T15:46:34.875842000Z Info: Loading facts 2017-07-13T15:46:34.875998000Z Info: Loading facts 2017-07-13T15:46:34.876159000Z Info: Loading facts 2017-07-13T15:46:34.876298000Z Info: Loading facts 2017-07-13T15:46:34.876493000Z Info: Loading facts 2017-07-13T15:46:34.876634000Z Info: Loading facts 2017-07-13T15:46:34.876809000Z Info: Loading facts 2017-07-13T15:46:34.876950000Z Info: Loading facts 2017-07-13T15:46:37.286789000Z Failed to get D-Bus connection: Operation not permitted 2017-07-13T15:46:37.420199000Z Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend 2017-07-13T15:46:37.420527000Z Warning: Undefined variable 'deploy_config_name'; 2017-07-13T15:46:37.420723000Z (file & line not available) 2017-07-13T15:46:37.512607000Z Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend 2017-07-13T15:46:37.795705000Z Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql_bundle.pp", 75]:["unknown", 1] 2017-07-13T15:46:37.796054000Z (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation') 2017-07-13T15:46:37.830408000Z Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp", 92]:["unknown", 1] 2017-07-13T15:46:37.830593000Z (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation') 2017-07-13T15:46:37.900289000Z Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules 2017-07-13T15:46:37.900518000Z (file & line not available) 2017-07-13T15:46:38.206703000Z Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/aodh/manifests/db/mysql.pp", 58]:["/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp", 160] 2017-07-13T15:46:38.207016000Z (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation') 2017-07-13T15:46:39.556095000Z Warning: Unknown variable: '::nova::db::mysql_api::setup_cell0'. at /etc/puppet/modules/nova/manifests/db/mysql.pp:53:28 2017-07-13T15:46:39.765052000Z Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules 2017-07-13T15:46:39.765251000Z (file & line not available) 2017-07-13T15:46:39.903546000Z Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp", 43]: 2017-07-13T15:46:39.903749000Z (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation') 2017-07-13T15:46:40.138751000Z Notice: Compiled catalog for controller-0.localdomain in environment production in 2.83 seconds 2017-07-13T15:46:40.374750000Z Info: Applying configuration version '1499960797' 2017-07-13T15:46:40.554134000Z Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker]/ensure: created 2017-07-13T15:46:40.557509000Z Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker-authkey]/ensure: defined content as '{md5}ff554c5032de56dfc1805ac668cf1ac7' 2017-07-13T15:46:40.559010000Z Info: Class[Pacemaker::Corosync]: Unscheduling all events on Class[Pacemaker::Corosync] 2017-07-13T15:46:40.562144000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]/ensure: defined content as '{md5}d78f65c211f4f42b711b497ea257d0d7' 2017-07-13T15:46:40.564699000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/etc/sysconfig/clustercheck]/ensure: defined content as '{md5}955258872109af7f433be323ec2ba78e' 2017-07-13T15:46:40.567583000Z Info: Computing checksum on file /etc/my.cnf.d/galera.cnf 2017-07-13T15:46:40.585310000Z Info: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf.d/galera.cnf to puppet with sum f434e1d5766874c7b9ed08d0c66904ca 2017-07-13T15:46:40.587247000Z Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}f434e1d5766874c7b9ed08d0c66904ca' to '{md5}9f5cf53819f1724916edac6c7f5d10a3' 2017-07-13T15:46:40.587492000Z Info: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Scheduling refresh of Tripleo::Pacemaker::Resource_restart_flag[galera-master] 2017-07-13T15:46:40.587811000Z Info: Tripleo::Pacemaker::Resource_restart_flag[galera-master]: Unscheduling all events on Tripleo::Pacemaker::Resource_restart_flag[galera-master] 2017-07-13T15:46:40.588085000Z Info: Class[Mysql::Server::Config]: Unscheduling all events on Class[Mysql::Server::Config] 2017-07-13T15:46:40.591260000Z Notice: /Stage[main]/Mysql::Server::Installdb/File[/var/log/mariadb/mariadb.log]/ensure: created 2017-07-13T15:46:40.591458000Z Info: Class[Mysql::Server::Installdb]: Unscheduling all events on Class[Mysql::Server::Installdb] 2017-07-13T15:46:41.638956000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Tripleo::Pacemaker::Resource_restart_flag[galera-master]/File[/var/lib/tripleo]/ensure: created 2017-07-13T15:46:41.645866000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Tripleo::Pacemaker::Resource_restart_flag[galera-master]/File[/var/lib/tripleo/pacemaker-restarts]/ensure: created 2017-07-13T15:46:41.646059000Z Info: Tripleo::Pacemaker::Resource_restart_flag[galera-master]: Unscheduling all events on Tripleo::Pacemaker::Resource_restart_flag[galera-master] 2017-07-13T15:46:53.153068000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]/ensure: created 2017-07-13T15:46:53.153328000Z Info: Pacemaker::Resource::Bundle[galera-bundle]: Unscheduling all events on Pacemaker::Resource::Bundle[galera-bundle] 2017-07-13T15:47:00.111800000Z Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]/ensure: created 2017-07-13T15:47:00.112063000Z Info: Pacemaker::Resource::Ocf[galera]: Unscheduling all events on Pacemaker::Resource::Ocf[galera] 2017-07-13T16:17:21.036663000Z Error: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0] 2017-07-13T16:17:21.037003000Z Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0] 2017-07-13T16:17:21.037203000Z Info: Class[Tripleo::Profile::Pacemaker::Database::Mysql_bundle]: Unscheduling all events on Class[Tripleo::Profile::Pacemaker::Database::Mysql_bundle] 2017-07-13T16:17:21.046275000Z Info: Creating state file /var/lib/puppet/state/state.yaml 2017-07-13T16:17:21.054702000Z Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Expected results: puppet runs correctly and the upgrade process moves forward Additional info:
openstack stack failures list overcloud: http://paste.openstack.org/show/615300/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462
clearing needinfo on this closed bug.