Red Hat Bugzilla – Bug 1280066
Operation DROP USER failed for 'root'@'127.0.0.1' puppet error happens on 2 controller nodes out of a 3 node HA deployment
Last modified: 2016-04-18 02:58:04 EDT
Doing an update from 7.0 to latest, during the ControllerNodesPostDeployment, which runs after the update has already been completed, all 3 Controllers failed on the ControllerOvercloudServicesDeployment_Step4 resource. 2 of the Controller nodes fail with this puppet error: Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: [2015-11-10 14:48:13,948] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386.pp. [6] Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: Error: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@127.0.0.1]/ensure: change from present to absent failed: Execution of '/usr/bin/mysql -e DROP USER 'root'@'127.0.0.1'' returned 1: ERROR 1396 (HY000) at line 1: Operation DROP USER failed for 'root'@'127.0.0.1' The other fails on restarting httpd, which is a known issue (bug 1275986), so I suspect it actually succeeded on drop the root user, but there's nothing to tell the other 2 nodes that they don't need to attempt to drop the root user. Could this actually be a bug in the mysql puppet module? Shouldn't it check that the user has already been dropped before attempting to drop it? If it is already doing that, maybe we're hitting a race condition across the cluster between the check and the attempted drop in the puppet code.
Note that on the 2 nodes that failed, the puppet run was started after the node that didn't hit this error. These are from the logs showing when the puppet apply was started: On the node that didn't hit this error: Nov 10 14:47:25 overcloud-controller-1.localdomain os-collect-config[3966]: [2015-11-10 14:47:24,975] (heat-config) [DEBUG] [2015-11-10 14:43:07,125] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/ea9654fc-11b3-45b2-91ea-d0b5e5c5d9b3" FACTER_fqdn="overcloud-controller-1.localdomain" FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5" puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/ea9654fc-11b3-45b2-91ea-d0b5e5c5d9b3.pp On the 2 that did hit the error: Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: [2015-11-10 14:48:13,959] (heat-config) [DEBUG] [2015-11-10 14:43:13,698] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386" FACTER_fqdn="overcloud-controller-0.localdomain" FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5" puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386.pp and Nov 10 14:48:31 overcloud-controller-2.localdomain os-collect-config[3530]: [2015-11-10 14:48:31,264] (heat-config) [DEBUG] [2015-11-10 14:43:12,100] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/1e8e31ce-32ea-4fc6-a5de-744ea12b23bb" FACTER_fqdn="overcloud-controller-2.localdomain" FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5" puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/1e8e31ce-32ea-4fc6-a5de-744ea12b23bb.pp
*** This bug has been marked as a duplicate of bug 1279967 ***