Bug 1280066

Summary:	Operation DROP USER failed for 'root'@'127.0.0.1' puppet error happens on 2 controller nodes out of a 3 node HA deployment
Product:	Red Hat OpenStack	Reporter:	James Slagle <jslagle>
Component:	openstack-tripleo-heat-templates	Assignee:	Jay Dobies <jason.dobies>
Status:	CLOSED DUPLICATE	QA Contact:	yeylon <yeylon>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	mburns, rhel-osp-director-maint, srevivo
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-11-10 21:23:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description James Slagle 2015-11-10 21:16:14 UTC

Doing an update from 7.0 to latest, during the ControllerNodesPostDeployment, which runs after the update has already been completed, all 3 Controllers failed on the ControllerOvercloudServicesDeployment_Step4 resource.

2 of the Controller nodes fail with this puppet error:
Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: [2015-11-10 14:48:13,948] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386.pp. [6]
Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: Error: /Stage[main]/Mysql::Server::Account_security/Mysql_user[root.0.1]/ensure: change from present to absent failed: Execution of '/usr/bin/mysql -e DROP USER 'root'@'127.0.0.1'' returned 1: ERROR 1396 (HY000) at line 1: Operation DROP USER failed for 'root'@'127.0.0.1'


The other fails on restarting httpd, which is a known issue (bug 1275986), so I suspect it actually succeeded on drop the root user, but there's nothing to tell the other 2 nodes that they don't need to attempt to drop the root user.

Could this actually be a bug in the mysql puppet module? Shouldn't it check that the user has already been dropped before attempting to drop it? If it is already doing that, maybe we're hitting a race condition across the cluster between the check and the attempted drop in the puppet code.

Comment 1 James Slagle 2015-11-10 21:18:12 UTC

Note that on the 2 nodes that failed, the puppet run was started after the node that didn't hit this error.

These are from the logs showing when the puppet apply was started:

On the node that didn't hit this error:

Nov 10 14:47:25 overcloud-controller-1.localdomain os-collect-config[3966]: [2015-11-10 14:47:24,975] (heat-config) [DEBUG] [2015-11-10 14:43:07,125] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/ea9654fc-11b3-45b2-91ea-d0b5e5c5d9b3"  FACTER_fqdn="overcloud-controller-1.localdomain"  FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/ea9654fc-11b3-45b2-91ea-d0b5e5c5d9b3.pp

On the 2 that did hit the error:

Nov 10 14:48:14 overcloud-controller-0.localdomain os-collect-config[3604]: [2015-11-10 14:48:13,959] (heat-config) [DEBUG] [2015-11-10 14:43:13,698] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386"  FACTER_fqdn="overcloud-controller-0.localdomain"  FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/48e0c030-a154-45c8-bcb9-254aea2f9386.pp

and

Nov 10 14:48:31 overcloud-controller-2.localdomain os-collect-config[3530]: [2015-11-10 14:48:31,264] (heat-config) [DEBUG] [2015-11-10 14:43:12,100] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/1e8e31ce-32ea-4fc6-a5de-744ea12b23bb"  FACTER_fqdn="overcloud-controller-2.localdomain"  FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-b7xinlxthgsl-ControllerPuppetConfig-rsoqvnfattbx-ControllerPuppetConfigImpl-l5qh474k3dw5"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/1e8e31ce-32ea-4fc6-a5de-744ea12b23bb.pp

Comment 3 James Slagle 2015-11-10 21:23:09 UTC


*** This bug has been marked as a duplicate of bug 1279967 ***