Bug 1160962 - mysql/rabbitmq-server error when puppet execution in HA controller
Summary: mysql/rabbitmq-server error when puppet execution in HA controller
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhel-osp-installer
Version: 5.0 (RHEL 6)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: Installer
Assignee: Mike Burns
QA Contact: Omri Hochman
URL:
Whiteboard: n1kv
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-06 03:23 UTC by Dulanjalie
Modified: 2015-09-10 21:58 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-09-10 21:58:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
mariadb.log (66.07 KB, text/plain)
2014-11-06 16:11 UTC, Dulanjalie
no flags Details

Description Dulanjalie 2014-11-06 03:23:03 UTC
I have a HA OS setup with three controller. I am seeing this error on the HA contoller :
Notice: /Stage[main]/Quickstack::Galera::Server/Galera::Server/Service[galera]/enable: enable changed 'false' to 'true'
Error: /Stage[main]/Galera::Monitor/Database_user[monitor_user@localhost]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'monitor_user@localhost'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[root@macf01fafe9db2a]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'root@macf01fafe9db2a'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[@macf01fafe9db2a.example.com]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = '@macf01fafe9db2a.example.com'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[@%]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = '@%'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[root@::1]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'root@::1'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[@localhost]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = '@localhost'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[root.0.1]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'root.0.1'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[root.com]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = 'root.com'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Error: /Stage[main]/Mysql::Server::Account_security/Database_user[@macf01fafe9db2a]: Could not evaluate: Execution of '/usr/bin/mysql --defaults-file=/root/.my.cnf mysql -NBe select '1' from mysql.user where CONCAT(user, '@', host) = '@macf01fafe9db2a'' returned 1: ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use
Notice: /Stage[main]/Quickstack::Galera::Db/Database[heat]: Dependency Database_user[root.com] has failures: true

How to trigger this:
reboot the controller. the run puppet

Workaround
pcs resource cleanup mysqld
pcs resource cleanup rabbitmq-server
restart puppet agent

Release: A2 OSP5

Comment 3 Leonid Natapov 2014-11-06 13:15:00 UTC
Hi,
Can you please,provide more information on what exactly you did ?  
The deployment went OK and everything worked correctly until you rebooted 1 node ?

I've tried to reproduce it on my environment and I didn't succeed.

It looks like galera cluster is out of sync and you have inconsistent data across various nodes in the cluster.


Also it would be great if you could attach mariadb.log file.

Comment 4 Dulanjalie 2014-11-06 16:11:16 UTC
1. I rebooted the HA controller which has DHCP/Metadata/L3 and openvswitch agents running
2. then after it came up i ran puppet agent -tv on all the three controllers. Other two controllers' puppet did not error out. but the one i rebooted gave me the error.

Attached the log file from rebooted HA controller

Comment 5 Dulanjalie 2014-11-06 16:11:48 UTC
Created attachment 954523 [details]
mariadb.log

Comment 6 Sourabh Patwardhan 2015-09-10 21:58:41 UTC
Closing this bug - too old (OSP 5).
The scenario causing the issue has been captured as a test-case for a later bug targeting OSP 7 (bug #1240394)


Note You need to log in before you can comment on or make changes to this bug.