1278004 – Reapplying puppet manifets on controller nodes causes cluster errors and long timeouts

Bug 1278004 - Reapplying puppet manifets on controller nodes causes cluster errors and long timeouts

Summary: Reapplying puppet manifets on controller nodes causes cluster errors and long...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	rhosp-director
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	y2
Target Release:	7.0 (Kilo)
Assignee:	Jiri Stransky
QA Contact:	Udi Kalifon
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-11-04 14:20 UTC by James Slagle
Modified:	2015-12-21 16:57 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Configuration records for heat-config were stored under /var/run, which were lost on node reboot. The cause for this was reported and fix (see BZ#1278181) but had an impact on the OSP director; Puppet reapplied old manifests during an Overcloud update, which caused various Pacemaker cluster errors. The fix for BZ#1278181 moves the configuration records from /var/run/heat-config to /var/lib/heat-config. This fix also includes the "heat-config-rebuild-deployed" script to rebuild the configuration records. Now updates are possible without reapplying old Puppet manifests. Make sure to follow the documented update procedure, including running the "heat-config-rebuild-deployed" script on each node.
Clone Of:
Environment:
Last Closed:	2015-12-21 16:57:49 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
os-collect-config.log showing the puppet runs, long delays, and errors (2.03 MB, text/plain) 2015-11-04 14:38 UTC, James Slagle	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2015:2651	0	normal	SHIPPED_LIVE	Red Hat Enterprise Linux OSP 7 director Bug Fix Advisory	2015-12-21 21:50:26 UTC

Description James Slagle 2015-11-04 14:20:53 UTC

I'm updating 7.0 to 7.1. My environment is a virt environment and I start by deploying a 7.0 undercloud, then deploy a 7.0 Overcloud with HA 3 Controllers, 1 compute, 1 ceph, network isolation single nic with vlans.

I'm seeing that before the UpdateDeployment's are even started on the controllers (the hooks haven't even been cleared yet), os-collect-config is reapplying all the puppet manifests.

A change to the NetworkConfig resource on each controller node, which is due to the network isolation changes in 7.1, causes new metadata to be available for os-collect-config.

The updated network configuration is applied fine.

The issue is that for some reason something in the chain decides that all of the puppet manifests need to be re-applied as well. So, during the same run of os-collect-config that was triggered due to the NetworkConfig change, configure.d/55-heat-config applies all the puppet manifests again. Note that it's really the same manifest, but it applies it 7 times due to the 7 steps post deployment process.

Note you haven't even gotten to the actual UpdateDeployment yet. yum_update.sh hasn't started running yet (the hook isn't cleared).

The actual issue that this causes is that all the puppet applies are running simultaneously across all 3 controllers, and something causes puppet to hang for 30 minutes during each step. That means several hours to get through just the update to NetworkDeployment, before UpdateDeployment is even started.

os-collect-config.log attached showing the puppet error (and hang).

Comment 2 James Slagle 2015-11-04 14:38:52 UTC

Created attachment 1089712 [details]
os-collect-config.log showing the puppet runs, long delays, and errors

Here's the relevant output from puppet from the log. Notice the galera errors

Nov 04 06:53:30 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 06:53:30,013] (heat-config) [DEBUG] Running /var/lib/heat-config/hooks/puppet < /var/run/heat-config/deployed/cf02add7-0855-4ff5-802e-0605a0320b97.json
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,529] (heat-config) [INFO] Return code 6
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,529] (heat-config) [INFO] Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 25.49 seconds
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.8]/Mysql_user[heat.2.8]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.8]/Mysql_grant[heat.2.8/heat.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.4]/Mysql_user[heat.2.4]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.4]/Mysql_user[glance.2.4]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.4]/Mysql_grant[glance.2.4/glance.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.8]/Mysql_user[keystone.2.8]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.4]/Mysql_user[keystone.2.4]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.4]/Mysql_grant[keystone.2.4/keystone.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.4]/Mysql_grant[heat.2.4/heat.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.8]/Mysql_user[glance.2.8]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.8]/Mysql_grant[glance.2.8/glance.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.8]/Mysql_user[neutron.2.8]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.8]/Mysql_user[nova.2.8]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.4]/Mysql_user[neutron.2.4]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.4]/Mysql_grant[neutron.2.4/ovs_neutron.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.8]/Mysql_grant[keystone.2.8/keystone.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.8]/Mysql_grant[nova.2.8/nova.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.4]/Mysql_user[cinder.2.4]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.4]/Mysql_grant[cinder.2.4/cinder.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.8]/Mysql_grant[neutron.2.8/ovs_neutron.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Swift/File[/var/run/swift]/group: group changed 'root' to 'swift'
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.4]/Mysql_user[nova.2.4]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.4]/Mysql_grant[nova.2.4/nova.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.8]/Mysql_user[cinder.2.8]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.8]/Mysql_grant[cinder.2.8/cinder.*]: Dependency Exec[galera-ready] has failures: true
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/File[/tmp/ceph-mon-keyring-overcloud-controller-0]/ensure: defined content as '{md5}bf9763d3c633605e2265b92ff2663803'
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-mkfs-overcloud-controller-0]/returns: ++ ceph-mon --id overcloud-controller-0 --show-config-value mon_data
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-mkfs-overcloud-controller-0]/returns: + mon_data=/var/lib/ceph/mon/ceph-overcloud-controller-0
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-mkfs-overcloud-controller-0]/returns: + '[' '!' -d /var/lib/ceph/mon/ceph-overcloud-controller-0 ']'
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-mkfs-overcloud-controller-0]/returns: executed successfully
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-ceph.client.admin.keyring-overcloud-controller-0]/returns: executed successfully
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Service[ceph-mon-overcloud-controller-0]/ensure: ensure changed 'stopped' to 'running'
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[rm-keyring-overcloud-controller-0]/returns: executed successfully
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Pacemaker::Corosync/Exec[enable-not-start-tripleo_cluster]/returns: executed successfully
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Pacemaker::Corosync/Exec[Set password for hacluster user on tripleo_cluster]/returns: executed successfully
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]/returns: executed successfully
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: Pacemaker has reported quorum achieved
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]/message: defined 'message' as 'Pacemaker has reported quorum achieved'
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: Finished catalog run in 1885.33 seconds
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,529] (heat-config) [INFO] Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Error: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Error: /Stage[main]/Main/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.8]/Mysql_user[heat.2.8]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.8]/Mysql_grant[heat.2.8/heat.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.4]/Mysql_user[heat.2.4]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.4]/Mysql_user[glance.2.4]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.4]/Mysql_grant[glance.2.4/glance.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.8]/Mysql_user[keystone.2.8]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.4]/Mysql_user[keystone.2.4]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.4]/Mysql_grant[keystone.2.4/keystone.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.4]/Mysql_grant[heat.2.4/heat.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.8]/Mysql_user[glance.2.8]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.8]/Mysql_grant[glance.2.8/glance.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.8]/Mysql_user[neutron.2.8]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.8]/Mysql_user[nova.2.8]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.4]/Mysql_user[neutron.2.4]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.4]/Mysql_grant[neutron.2.4/ovs_neutron.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.8]/Mysql_grant[keystone.2.8/keystone.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.8]/Mysql_grant[nova.2.8/nova.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.4]/Mysql_user[cinder.2.4]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.4]/Mysql_grant[cinder.2.4/cinder.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.8]/Mysql_grant[neutron.2.8/ovs_neutron.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.4]/Mysql_user[nova.2.4]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.4]/Mysql_grant[nova.2.4/nova.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.8]/Mysql_user[cinder.2.8]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.8]/Mysql_grant[cinder.2.8/cinder.*]: Skipping because of failed dependencies
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,529] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/cf02add7-0855-4ff5-802e-0605a0320b97.pp. [6]
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,536] (heat-config) [INFO] Completed /var/lib/heat-config/hooks/puppet
Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,537] (heat-config) [DEBUG] Running heat-config-notify /var/run/heat-config/deployed/cf02add7-0855-4ff5-802e-0605a0320b97.json < /var/run/heat-config/deployed/cf02add7-0855-4ff5-802e-0605a0320b97.notify.json
Nov 04 07:25:35 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:35,459] (heat-config) [INFO]
Nov 04 07:25:35 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:35,460] (heat-config) [DEBUG] [2015-11-04 07:25:34,915] (heat-config-notify) [DEBUG] Signaling to http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3A812ed725111f42faba2fd979f817648a%3Astacks%2Fovercloud-ControllerNodesPostDeployment-tssok6wjp4ao-ControllerServicesBaseDeployment_Step2-pa6qy6cdeq6q%2Fd1d66eb2-2d1e-4380-b0a2-9bb3ed76f475%2Fresources%2F0?Timestamp=2015-11-03T23%3A30%3A27Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=553e49b7345f40988549d12c9d7156a7&SignatureVersion=2&Signature=JiDbcWIWglPqAyVrXY1yl2nHeinJwCJe%2FKZ%2Bhz9%2Fpz4%3D via POST
Nov 04 07:25:35 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:35,414] (heat-config-notify) [DEBUG] Response <Response [200]>

Comment 3 James Slagle 2015-11-04 14:39:30 UTC

I consistently reproduce this. I took a copy of /var/lib/os-collect-config before I started the update to compare what metadata has changed. Here's the diff:

[root@overcloud-controller-0 lib]# diff -r -u os-collect-config.save os-collect-config
diff -r -u os-collect-config.save/cfn.json os-collect-config/cfn.json
--- os-collect-config.save/cfn.json     2015-11-04 06:45:38.165520672 -0500
+++ os-collect-config/cfn.json  2015-11-04 06:50:19.497339396 -0500
@@ -990,7 +990,7 @@
<SNIP>
\ No newline at end of file
diff -r -u os-collect-config.save/os_config_files.json os-collect-config/os_config_files.json
--- os-collect-config.save/os_config_files.json 2015-11-04 06:45:38.164520693 -0500
+++ os-collect-config/os_config_files.json      2015-11-04 06:50:19.522338847 -0500
@@ -1 +1 @@
-["/var/lib/os-collect-config/heat_local.json", "/var/lib/os-collect-config/ec2.json", "/var/lib/os-collect-config/cfn.json", "/var/lib/os-collect-config/overcloud-CephClusterConfig-msrbgvgtjir3-CephClusterConfigImpl-wwhgidx2y56w.json", "/var/lib/os-collect-config/overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-ControllerConfig-trpbuff7uga2.json", "/var/lib/os-collect-config/overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-q3vvhibkxvxm.json", "/var/lib/os-collect-config/overcloud-ControllerBootstrapNodeConfig-jicsj6rjai3l-BootstrapNodeConfigImpl-ms4q4g4c5ubc.json", "/var/lib/os-collect-config/overcloud-ControllerClusterConfig-t6dwlmz5zrbm.json", "/var/lib/os-collect-config/overcloud-SwiftDevicesAndProxyConfig-egl7atkcl43f-SwiftDevicesAndProxyConfigImpl-lqeev4b7rzef.json", "/var/lib/os-collect-config/overcloud-VipConfig-bcgu4co7pcgv-VipConfigImpl-zytbdyur7li2.json", "/var/lib/os-collect-config/overcloud-allNodesConfig-hpl4rjypby3y-allNodesConfigImpl-qri6psktvroo.json"]
\ No newline at end of file
+["/var/lib/os-collect-config/heat_local.json", "/var/lib/os-collect-config/ec2.json", "/var/lib/os-collect-config/cfn.json", "/var/lib/os-collect-config/overcloud-CephClusterConfig-msrbgvgtjir3-CephClusterConfigImpl-wwhgidx2y56w.json", "/var/lib/os-collect-config/overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-ControllerConfig-trpbuff7uga2.json", "/var/lib/os-collect-config/overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-qbxlryffunn4.json", "/var/lib/os-collect-config/overcloud-ControllerBootstrapNodeConfig-jicsj6rjai3l-BootstrapNodeConfigImpl-ms4q4g4c5ubc.json", "/var/lib/os-collect-config/overcloud-ControllerClusterConfig-t6dwlmz5zrbm.json", "/var/lib/os-collect-config/overcloud-SwiftDevicesAndProxyConfig-egl7atkcl43f-SwiftDevicesAndProxyConfigImpl-lqeev4b7rzef.json", "/var/lib/os-collect-config/overcloud-VipConfig-bcgu4co7pcgv-VipConfigImpl-zytbdyur7li2.json", "/var/lib/os-collect-config/overcloud-allNodesConfig-hpl4rjypby3y-allNodesConfigImpl-qri6psktvroo.json"]
\ No newline at end of file
Only in os-collect-config: overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-qbxlryffunn4.json
Only in os-collect-config: overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-qbxlryffunn4.json.orig


So, I snipped out the diff from cfn.json, but you can see the differences in the directories is wholly contained to cfn.json, os_config_files.json and a new overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-qbxlryffunn4.json which replaces the old NetworkConfig.

It's not clear to me why these changes would cause the puppet manifests to get reapplied.

Comment 4 James Slagle 2015-11-04 14:44:08 UTC

There were also no changes between the puppet manifests:

[root@overcloud-controller-0 heat-config]# diff -u -r heat-config-puppet.save heat-config-puppet
[root@overcloud-controller-0 heat-config]# 


I guess ultimately the reapply of puppet before UpdateDeployment even starts running might be expected. But when it happens, the 30 minute hang on each step is definitely an issue.

Comment 5 James Slagle 2015-11-04 21:23:45 UTC

i figured out what's causing the puppet reapply:
https://bugzilla.redhat.com/show_bug.cgi?id=1278181

As to why the actual reapply causes these puppet errors, that's still unknown. Hopefully if we can fix bug 1278181, then this one won't be an issue, or at least only happen in a way such that Heat would know about it.

Comment 6 James Slagle 2015-11-16 20:17:45 UTC

assuming the puppet errors here are in fact caused by service restarts by puppet when the pacemaker cluster isn't in maintenance mode (likely), then no need to focus on fixing the actual puppet issue here.

What we do likely need is a way to restore the contents of /var/lib/heat-config/deployed, or some mechanism to ensure that when the puppet apply's happen, the cluster is in maintenance mode and taken out of maintenance mode in the end.

Right now, there's no way to orchestrate the maintenance mode in this scenario because os-collect-config triggers the puppet apply without Heat's knowledge at all. So I think a better thing to do here might be to find a way to restore /var/lib/heat-config/deployed somehow. Maybe it would have to be out of band?

Comment 9 Udi Kalifon 2015-12-20 15:00:15 UTC

Successfully updated from 7.1 to 7.2 after rebooting a controller and a compute, and applying the 55-heat-config workaround. Verified.

Comment 11 errata-xmlrpc 2015-12-21 16:57:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2651

Note You need to log in before you can comment on or make changes to this bug.