Bug 1278004
Summary: | Reapplying puppet manifets on controller nodes causes cluster errors and long timeouts | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | James Slagle <jslagle> | ||||
Component: | rhosp-director | Assignee: | Jiri Stransky <jstransk> | ||||
Status: | CLOSED ERRATA | QA Contact: | Udi Kalifon <ukalifon> | ||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 7.0 (Kilo) | CC: | calfonso, dmacpher, dnavale, kbasil, mburns, ohochman, rhel-osp-director-maint, yeylon | ||||
Target Milestone: | y2 | Keywords: | TestOnly, Triaged | ||||
Target Release: | 7.0 (Kilo) | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: |
Configuration records for heat-config were stored under /var/run, which were lost on node reboot. The cause for this was reported and fix (see BZ#1278181) but had an impact on the OSP director; Puppet reapplied old manifests during an Overcloud update, which caused various Pacemaker cluster errors. The fix for BZ#1278181 moves the configuration records from /var/run/heat-config to /var/lib/heat-config. This fix also includes the "heat-config-rebuild-deployed" script to rebuild the configuration records. Now updates are possible without reapplying old Puppet manifests. Make sure to follow the documented update procedure, including running the "heat-config-rebuild-deployed" script on each node.
|
Story Points: | --- | ||||
Clone Of: | Environment: | ||||||
Last Closed: | 2015-12-21 16:57:49 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
James Slagle
2015-11-04 14:20:53 UTC
Created attachment 1089712 [details] os-collect-config.log showing the puppet runs, long delays, and errors Here's the relevant output from puppet from the log. Notice the galera errors Nov 04 06:53:30 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 06:53:30,013] (heat-config) [DEBUG] Running /var/lib/heat-config/hooks/puppet < /var/run/heat-config/deployed/cf02add7-0855-4ff5-802e-0605a0320b97.json Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,529] (heat-config) [INFO] Return code 6 Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,529] (heat-config) [INFO] Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 25.49 seconds Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.8]/Mysql_user[heat.2.8]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.8]/Mysql_grant[heat.2.8/heat.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.4]/Mysql_user[heat.2.4]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.4]/Mysql_user[glance.2.4]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.4]/Mysql_grant[glance.2.4/glance.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.8]/Mysql_user[keystone.2.8]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.4]/Mysql_user[keystone.2.4]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.4]/Mysql_grant[keystone.2.4/keystone.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.4]/Mysql_grant[heat.2.4/heat.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.8]/Mysql_user[glance.2.8]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.8]/Mysql_grant[glance.2.8/glance.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.8]/Mysql_user[neutron.2.8]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.8]/Mysql_user[nova.2.8]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.4]/Mysql_user[neutron.2.4]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.4]/Mysql_grant[neutron.2.4/ovs_neutron.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.8]/Mysql_grant[keystone.2.8/keystone.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.8]/Mysql_grant[nova.2.8/nova.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.4]/Mysql_user[cinder.2.4]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.4]/Mysql_grant[cinder.2.4/cinder.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.8]/Mysql_grant[neutron.2.8/ovs_neutron.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Swift/File[/var/run/swift]/group: group changed 'root' to 'swift' Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.4]/Mysql_user[nova.2.4]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.4]/Mysql_grant[nova.2.4/nova.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:33 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.8]/Mysql_user[cinder.2.8]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.8]/Mysql_grant[cinder.2.8/cinder.*]: Dependency Exec[galera-ready] has failures: true Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/File[/tmp/ceph-mon-keyring-overcloud-controller-0]/ensure: defined content as '{md5}bf9763d3c633605e2265b92ff2663803' Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-mkfs-overcloud-controller-0]/returns: ++ ceph-mon --id overcloud-controller-0 --show-config-value mon_data Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-mkfs-overcloud-controller-0]/returns: + mon_data=/var/lib/ceph/mon/ceph-overcloud-controller-0 Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-mkfs-overcloud-controller-0]/returns: + '[' '!' -d /var/lib/ceph/mon/ceph-overcloud-controller-0 ']' Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-mkfs-overcloud-controller-0]/returns: executed successfully Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[ceph-mon-ceph.client.admin.keyring-overcloud-controller-0]/returns: executed successfully Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Service[ceph-mon-overcloud-controller-0]/ensure: ensure changed 'stopped' to 'running' Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[overcloud-controller-0]/Exec[rm-keyring-overcloud-controller-0]/returns: executed successfully Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Pacemaker::Corosync/Exec[enable-not-start-tripleo_cluster]/returns: executed successfully Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Pacemaker::Corosync/Exec[Set password for hacluster user on tripleo_cluster]/returns: executed successfully Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]/returns: executed successfully Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: Pacemaker has reported quorum achieved Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]/message: defined 'message' as 'Pacemaker has reported quorum achieved' Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Notice: Finished catalog run in 1885.33 seconds Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,529] (heat-config) [INFO] Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false. Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default') Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Error: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0] Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Error: /Stage[main]/Main/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0] Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Mysql_database[glance]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Error: Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.8]/Mysql_user[heat.2.8]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.8]/Mysql_grant[heat.2.8/heat.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.4]/Mysql_user[heat.2.4]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_user[glance@%]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_%]/Mysql_grant[glance@%/glance.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.4]/Mysql_user[glance.2.4]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.4]/Mysql_grant[glance.2.4/glance.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.8]/Mysql_user[keystone.2.8]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.4]/Mysql_user[keystone.2.4]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.4]/Mysql_grant[keystone.2.4/keystone.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_172.16.2.4]/Mysql_grant[heat.2.4/heat.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Mysql_database[cinder]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_user[cinder@%]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.8]/Mysql_user[glance.2.8]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_172.16.2.8]/Mysql_grant[glance.2.8/glance.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Mysql_database[ovs_neutron]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.8]/Mysql_user[neutron.2.8]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Mysql_database[nova]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_user[nova@%]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_%]/Mysql_grant[nova@%/nova.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.8]/Mysql_user[nova.2.8]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.4]/Mysql_user[neutron.2.4]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.4]/Mysql_grant[neutron.2.4/ovs_neutron.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_172.16.2.8]/Mysql_grant[keystone.2.8/keystone.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.8]/Mysql_grant[nova.2.8/nova.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_user[neutron@%]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_%]/Mysql_grant[neutron@%/ovs_neutron.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.4]/Mysql_user[cinder.2.4]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.4]/Mysql_grant[cinder.2.4/cinder.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_user[keystone@%]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_%]/Mysql_grant[keystone@%/keystone.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_172.16.2.8]/Mysql_grant[neutron.2.8/ovs_neutron.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.4]/Mysql_user[nova.2.4]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_172.16.2.4]/Mysql_grant[nova.2.4/nova.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_%]/Mysql_grant[cinder@%/cinder.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.8]/Mysql_user[cinder.2.8]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: Warning: /Stage[main]/Cinder::Db::Mysql/Openstacklib::Db::Mysql[cinder]/Openstacklib::Db::Mysql::Host_access[cinder_172.16.2.8]/Mysql_grant[cinder.2.8/cinder.*]: Skipping because of failed dependencies Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,529] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/cf02add7-0855-4ff5-802e-0605a0320b97.pp. [6] Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,536] (heat-config) [INFO] Completed /var/lib/heat-config/hooks/puppet Nov 04 07:25:34 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:33,537] (heat-config) [DEBUG] Running heat-config-notify /var/run/heat-config/deployed/cf02add7-0855-4ff5-802e-0605a0320b97.json < /var/run/heat-config/deployed/cf02add7-0855-4ff5-802e-0605a0320b97.notify.json Nov 04 07:25:35 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:35,459] (heat-config) [INFO] Nov 04 07:25:35 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:35,460] (heat-config) [DEBUG] [2015-11-04 07:25:34,915] (heat-config-notify) [DEBUG] Signaling to http://192.0.2.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3A812ed725111f42faba2fd979f817648a%3Astacks%2Fovercloud-ControllerNodesPostDeployment-tssok6wjp4ao-ControllerServicesBaseDeployment_Step2-pa6qy6cdeq6q%2Fd1d66eb2-2d1e-4380-b0a2-9bb3ed76f475%2Fresources%2F0?Timestamp=2015-11-03T23%3A30%3A27Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=553e49b7345f40988549d12c9d7156a7&SignatureVersion=2&Signature=JiDbcWIWglPqAyVrXY1yl2nHeinJwCJe%2FKZ%2Bhz9%2Fpz4%3D via POST Nov 04 07:25:35 overcloud-controller-0.localdomain os-collect-config[2469]: [2015-11-04 07:25:35,414] (heat-config-notify) [DEBUG] Response <Response [200]> I consistently reproduce this. I took a copy of /var/lib/os-collect-config before I started the update to compare what metadata has changed. Here's the diff: [root@overcloud-controller-0 lib]# diff -r -u os-collect-config.save os-collect-config diff -r -u os-collect-config.save/cfn.json os-collect-config/cfn.json --- os-collect-config.save/cfn.json 2015-11-04 06:45:38.165520672 -0500 +++ os-collect-config/cfn.json 2015-11-04 06:50:19.497339396 -0500 @@ -990,7 +990,7 @@ <SNIP> \ No newline at end of file diff -r -u os-collect-config.save/os_config_files.json os-collect-config/os_config_files.json --- os-collect-config.save/os_config_files.json 2015-11-04 06:45:38.164520693 -0500 +++ os-collect-config/os_config_files.json 2015-11-04 06:50:19.522338847 -0500 @@ -1 +1 @@ -["/var/lib/os-collect-config/heat_local.json", "/var/lib/os-collect-config/ec2.json", "/var/lib/os-collect-config/cfn.json", "/var/lib/os-collect-config/overcloud-CephClusterConfig-msrbgvgtjir3-CephClusterConfigImpl-wwhgidx2y56w.json", "/var/lib/os-collect-config/overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-ControllerConfig-trpbuff7uga2.json", "/var/lib/os-collect-config/overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-q3vvhibkxvxm.json", "/var/lib/os-collect-config/overcloud-ControllerBootstrapNodeConfig-jicsj6rjai3l-BootstrapNodeConfigImpl-ms4q4g4c5ubc.json", "/var/lib/os-collect-config/overcloud-ControllerClusterConfig-t6dwlmz5zrbm.json", "/var/lib/os-collect-config/overcloud-SwiftDevicesAndProxyConfig-egl7atkcl43f-SwiftDevicesAndProxyConfigImpl-lqeev4b7rzef.json", "/var/lib/os-collect-config/overcloud-VipConfig-bcgu4co7pcgv-VipConfigImpl-zytbdyur7li2.json", "/var/lib/os-collect-config/overcloud-allNodesConfig-hpl4rjypby3y-allNodesConfigImpl-qri6psktvroo.json"] \ No newline at end of file +["/var/lib/os-collect-config/heat_local.json", "/var/lib/os-collect-config/ec2.json", "/var/lib/os-collect-config/cfn.json", "/var/lib/os-collect-config/overcloud-CephClusterConfig-msrbgvgtjir3-CephClusterConfigImpl-wwhgidx2y56w.json", "/var/lib/os-collect-config/overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-ControllerConfig-trpbuff7uga2.json", "/var/lib/os-collect-config/overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-qbxlryffunn4.json", "/var/lib/os-collect-config/overcloud-ControllerBootstrapNodeConfig-jicsj6rjai3l-BootstrapNodeConfigImpl-ms4q4g4c5ubc.json", "/var/lib/os-collect-config/overcloud-ControllerClusterConfig-t6dwlmz5zrbm.json", "/var/lib/os-collect-config/overcloud-SwiftDevicesAndProxyConfig-egl7atkcl43f-SwiftDevicesAndProxyConfigImpl-lqeev4b7rzef.json", "/var/lib/os-collect-config/overcloud-VipConfig-bcgu4co7pcgv-VipConfigImpl-zytbdyur7li2.json", "/var/lib/os-collect-config/overcloud-allNodesConfig-hpl4rjypby3y-allNodesConfigImpl-qri6psktvroo.json"] \ No newline at end of file Only in os-collect-config: overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-qbxlryffunn4.json Only in os-collect-config: overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-qbxlryffunn4.json.orig So, I snipped out the diff from cfn.json, but you can see the differences in the directories is wholly contained to cfn.json, os_config_files.json and a new overcloud-Controller-4p6knggj24jg-0-dqg7cgjja7ha-NetworkConfig-4cejl5bcqnz6-OsNetConfigImpl-qbxlryffunn4.json which replaces the old NetworkConfig. It's not clear to me why these changes would cause the puppet manifests to get reapplied. There were also no changes between the puppet manifests: [root@overcloud-controller-0 heat-config]# diff -u -r heat-config-puppet.save heat-config-puppet [root@overcloud-controller-0 heat-config]# I guess ultimately the reapply of puppet before UpdateDeployment even starts running might be expected. But when it happens, the 30 minute hang on each step is definitely an issue. i figured out what's causing the puppet reapply: https://bugzilla.redhat.com/show_bug.cgi?id=1278181 As to why the actual reapply causes these puppet errors, that's still unknown. Hopefully if we can fix bug 1278181, then this one won't be an issue, or at least only happen in a way such that Heat would know about it. assuming the puppet errors here are in fact caused by service restarts by puppet when the pacemaker cluster isn't in maintenance mode (likely), then no need to focus on fixing the actual puppet issue here. What we do likely need is a way to restore the contents of /var/lib/heat-config/deployed, or some mechanism to ensure that when the puppet apply's happen, the cluster is in maintenance mode and taken out of maintenance mode in the end. Right now, there's no way to orchestrate the maintenance mode in this scenario because os-collect-config triggers the puppet apply without Heat's knowledge at all. So I think a better thing to do here might be to find a way to restore /var/lib/heat-config/deployed somehow. Maybe it would have to be out of band? Successfully updated from 7.1 to 7.2 after rebooting a controller and a compute, and applying the 55-heat-config workaround. Verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:2651 |