Bug 1160962

Summary: mysql/rabbitmq-server error when puppet execution in HA controller
Product: Red Hat OpenStack Reporter: Dulanjalie <dulanjalie>
Component: rhel-osp-installerAssignee: Mike Burns <mburns>
Status: CLOSED WONTFIX QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.0 (RHEL 6)CC: cwchang, dulanjalie, lnatapov, rhos-maint, sopatwar, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: Installer   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: n1kv
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-10 21:58:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
mariadb.log none

Description Dulanjalie 2014-11-06 03:23:03 UTC 
I have a HA OS setup with three controller. I am seeing this error on the HA contoller :
Notice: /Stage[main]/Quickstack::Galera::Server/Galera::Server/Service[galera]/enable: enable changed 'false' to 'true'
Error: /Stage[main]/Galera::Monitor/Database_user[monitor_user@localhost]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'monitor_user@localhost'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[root@macf01fafe9db2a]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'root@macf01fafe9db2a'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[@macf01fafe9db2a.example.com]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = '@macf01fafe9db2a.example.com'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[@%]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = '@%'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[root@::1]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'root@::1'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[@localhost]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = '@localhost'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[root.0.1]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'root.0.1'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[root.com]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'root.com'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[@macf01fafe9db2a]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = '@macf01fafe9db2a'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Notice: /Stage[main]/Quickstack::Galera::Db/Database[heat]: Dependency Database_user[root.com] has failures: true

How to trigger this:
reboot the controller. the run puppet

Workaround
pcs resource cleanup mysqld
pcs resource cleanup rabbitmq-server
restart puppet agent

Release: A2 OSP5
Comment 3 Leonid Natapov 2014-11-06 13:15:00 UTC 
Hi,
Can you please,provide more information on what exactly you did ?  
The deployment went OK and everything worked correctly until you rebooted 1 node ?

I've tried to reproduce it on my environment and I didn't succeed.

It looks like galera cluster is out of sync and you have inconsistent data across various nodes in the cluster.


Also it would be great if you could attach mariadb.log file.
Comment 4 Dulanjalie 2014-11-06 16:11:16 UTC 
1. I rebooted the HA controller which has DHCP/Metadata/L3 and openvswitch agents running
2. then after it came up i ran puppet agent -tv on all the three controllers. Other two controllers' puppet did not error out. but the one i rebooted gave me the error.

Attached the log file from rebooted HA controller
Comment 5 Dulanjalie 2014-11-06 16:11:48 UTC 
Created attachment 954523 [details]
mariadb.log
Comment 6 Sourabh Patwardhan 2015-09-10 21:58:41 UTC 
Closing this bug - too old (OSP 5).
The scenario causing the issue has been captured as a test-case for a later bug targeting OSP 7 (bug #1240394)