Bug 1280066 - Operation DROP USER failed for 'root'@'' puppet error happens on 2 controller nodes out of a 3 node HA deployment
Operation DROP USER failed for 'root'@'' puppet error happens on 2 c...
Status: CLOSED DUPLICATE of bug 1279967
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jay Dobies
Depends On:
  Show dependency treegraph
Reported: 2015-11-10 16:16 EST by James Slagle
Modified: 2016-04-18 02:58 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-11-10 16:23:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description James Slagle 2015-11-10 16:16:14 EST
Doing an update from 7.0 to latest, during the ControllerNodesPostDeployment, which runs after the update has already been completed, all 3 Controllers failed on the ControllerOvercloudServicesDeployment_Step4 resource.

2 of the Controller nodes fail with this puppet error:
Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: [2015-11-10 14:48:13,948] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386.pp. [6]
Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: Error: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root@]/ensure: change from present to absent failed: Execution of '/usr/bin/mysql -e DROP USER 'root'@''' returned 1: ERROR 1396 (HY000) at line 1: Operation DROP USER failed for 'root'@''

The other fails on restarting httpd, which is a known issue (bug 1275986), so I suspect it actually succeeded on drop the root user, but there's nothing to tell the other 2 nodes that they don't need to attempt to drop the root user.

Could this actually be a bug in the mysql puppet module? Shouldn't it check that the user has already been dropped before attempting to drop it? If it is already doing that, maybe we're hitting a race condition across the cluster between the check and the attempted drop in the puppet code.
Comment 1 James Slagle 2015-11-10 16:18:12 EST
Note that on the 2 nodes that failed, the puppet run was started after the node that didn't hit this error.

These are from the logs showing when the puppet apply was started:

On the node that didn't hit this error:

Nov 10 14:47:25 overcloud-controller-1.localdomain os-collect-config[3966]: [2015-11-10 14:47:24,975] (heat-config) [DEBUG] [2015-11-10 14:43:07,125] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/ea9654fc-11b3-45b2-91ea-d0b5e5c5d9b3"  FACTER_fqdn="overcloud-controller-1.localdomain"  FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/ea9654fc-11b3-45b2-91ea-d0b5e5c5d9b3.pp

On the 2 that did hit the error:

Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: [2015-11-10 14:48:13,959] (heat-config) [DEBUG] [2015-11-10 14:43:13,698] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386"  FACTER_fqdn="overcloud-controller-0.localdomain"  FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386.pp


Nov 10 14:48:31 overcloud-controller-2.localdomain os-collect-config[3530]: [2015-11-10 14:48:31,264] (heat-config) [DEBUG] [2015-11-10 14:43:12,100] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/1e8e31ce-32ea-4fc6-a5de-744ea12b23bb"  FACTER_fqdn="overcloud-controller-2.localdomain"  FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/1e8e31ce-32ea-4fc6-a5de-744ea12b23bb.pp
Comment 3 James Slagle 2015-11-10 16:23:09 EST

*** This bug has been marked as a duplicate of bug 1279967 ***

Note You need to log in before you can comment on or make changes to this bug.