Bug 1149066

Looks like this is a PackStack issue; CONFIG_MYSQL_HOST has to be set to 127.0.0.1 in the PackStack answer file (and it's set by default to the host IP). See: https://bugs.launchpad.net/packstack/+bug/1369498

I updated this parameter in the answers file, and the installation was able to complete.

That is, the CONFIG_MARIADB_HOST param must be changed. :)

I have not noticed this on any of my recent all-in-one installations, and I haven't been setting anything special for the CONFIG_MARIADB_HOST setting in my answerfile.  I just kicked off a new environment installation using RDO Juno bits, so I will check what it defaults to for me when it's completed.

Out of curiosity, what is the hostname set to on your test system and what IP address does it resolve to?

At least with RDO Juno, the default seems to be to use the host IP (not the loopback) for CONFIG_MYSQL_HOST.  Despite this, packstack is successful.  I will try this with the RHEL OSP5 packages to see what address is used for me there.

Well, I used Icehouse RHELOSP packages, so perhaps whatever it is has been ironed out?
The host name is OpenStack1.lab.local, and the host IP is 10.64.33.19.

Nathan, installing on a VM works fine. However, on new, bare-metal packstack installs (RHELOSP5-onEL7), I get this one consistently:
...
10.64.33.213_keystone.pp:                         [ ERROR ]         
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.64.33.213_keystone.pp
Error: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0]

On the second run with the same answer file, I then get:
10.64.33.213_keystone.pp:                         [ ERROR ]         
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.64.33.213_keystone.pp
Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[_member_]: Could not evaluate: Execution of '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ role-list' returned 1: An unexpected error prevented the server from fulfilling your request. (HTTP 500)

If I then change the CONFIG_MARIADB_HOST parameter to 127.0.0.1, the installation completes.

I am unabled to reproduce this on EL7 with RHEL OSP5.  The fact that it works for you on a VM but not on a bare-metal system suggests that it's environment related.  I'd be happy to look at your system where this issue occurs if you can provide me with access.

Sure, it's just a basic install of RHEL 7. The only bits I've changed since the install are the ifcfg- files so that I can get external access going for the instances. Ping me when you're on IRC (slong), and I'll give you the info. Thanks, Summer

I believe that the issue here is how the firewall rules are being set up along with the way the hostname is configured on the system.  I see this in the 'iptables -L' output on the affected system:

---------------------------------------------------------------------------------
ACCEPT     tcp  --  OpenShift1.lab.local  anywhere             multiport dports mysql /* 001 mariadb incoming mariadb_10.64.33.213 */
---------------------------------------------------------------------------------

The hostname of the system (OpenShift1.lab.local) is being used as the source address in the firewall rule.  On this system, the hostname is not defined properly in /etc/hosts:

---------------------------------------------------------------------------------
[root@OpenShift1 packstack]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
---------------------------------------------------------------------------------

Since there is no host entry for OpenShift1.lab.local, reverse lookup will fail:

---------------------------------------------------------------------------------
[root@OpenShift1 packstack]# nslookup OpenShift1.lab.local
Server:		10.64.63.6
Address:	10.64.63.6#53

** server can't find OpenShift1.lab.local: NXDOMAIN
---------------------------------------------------------------------------------

I believe that the inability to perform a reverse lookup is causing the firewall to block connections to mariadb that are made using the system IP address.  I'll also add that mariadb is not alone here, as rules for AMQP, mongodb, nagios, qemu migration, and other services are all using the system hostname as the source address.

A good way of testing this theory would be to add the following to /etc/hosts prior to running packstack (without setting CONFIG_MARIADB_HOST in the answer file):

----------------------------------------------------------------------------------
10.64.33.213 OpenShift1.lab.local
----------------------------------------------------------------------------------

When you have a chance, please test this on your system with the system hostname properly set.  We can then figure out how to proceed.

Hi Nathan, I tried to do a new packstack with the hostname properly set. But no, that didn't solve it. Any other suggestions?

Agree to close (I didn't hit this issue in Juno either).