Bug 1368231

Summary: Undercloud Install fails with no suitable provider for mysql_database
Product: Red Hat OpenStack Reporter: Sai Sindhur Malleni <smalleni>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: augol, dbecker, dsneddon, dtantsur, jcoufal, jschluet, mburns, morazi, rhel-osp-director-maint, smalleni, tshefi
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-02 12:50:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
cougar06 and 07
none
undercloud.conf and install.log none

Description Sai Sindhur Malleni 2016-08-18 18:17:55 UTC
Description of problem:
Undercloud install fails intermittently

Version-Release number of selected component (if applicable):
OSP10

How reproducible:
Not very reproducible

Steps to Reproduce:
1. Install undercloud
2.
3.

Actual results:
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[nova_api@%] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[nova_api.2.1] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_database[neutron] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[neutron@%] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[neutron.2.1] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_database[heat] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[heat@%] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[heat.2.1] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_database[ironic] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[ironic@%] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[ironic.2.1] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_database[mistral] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[mistral@%] has failures: true
2016-08-16 14:03:20 - Notice: /Firewall[999 drop all]: Dependency Mysql_user[mistral.2.1] has failures: true
2016-08-16 14:03:20 - Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
2016-08-16 14:03:20 - Error: Could not find a suitable provider for mysql_datadir
2016-08-16 14:03:20 - Error: Could not find a suitable provider for mysql_database
2016-08-16 14:03:20 - Error: Could not find a suitable provider for mysql_user
2016-08-16 14:03:20 - Notice: Finished catalog run in 6.92 seconds
+ rc=4
+ set -e
+ echo 'puppet apply exited with exit code 4'
puppet apply exited with exit code 4
+ '[' 4 '!=' 2 -a 4 '!=' 0 ']'

Expected results:
Undercloud should install successfully

Additional info:
Do not have additional logs, as I do not have the environment anymore.

Comment 2 Dan Sneddon 2016-10-14 16:49:44 UTC
(In reply to Sai Sindhur Malleni from comment #0)
> Description of problem:
> Undercloud install fails intermittently

I believe this error may have had to do with an unreachable IP on 192.0.2.1. This may happen if the br-ctlplane bridge isn't enabled properly with the IP from the attached interface (the IP should be moved from the interface to the bridge).

If this re-occurs, can you run the following commands and attach them to this bug?

ip address
sudo ovs-vsctl show
cat ~stack/undercloud.conf

Comment 3 Tzach Shefi 2016-11-01 13:05:38 UTC
Created attachment 1216111 [details]
cougar06 and 07

Comment 4 Tzach Shefi 2016-11-01 13:06:12 UTC
I've hit this on two virt OSPD deployments. 
Cougar06 host is RHEL7.3
Cougar07 host is RHEL 7.2
Both servers are Dell C6105, dual AMD servers 12 core, 64G RAM. 
I mention this cause we have seen this on a fellow server cougar09, reused since. Possible this might happen show up when using AMD CPUs?

Both servers fail with same errors as this bug. 

eth0 of undercloud doesn't have an IP.

on IP a -> eth0 no IP set. 

[root@undercloud-0 ~]# sudo ovs-vsctl show
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)


[root@undercloud-0 ~]# cat ~stack/undercloud.conf
[DEFAULT]
# Network interface on the Undercloud that will be handling the PXE
# boots and DHCP for Overcloud instances. (string value)
local_interface = eth0

Attached output for Dan's request for both servers comment #3.

Comment 6 Tzach Shefi 2016-11-02 07:36:45 UTC
Hi Dan, 
Anything else I should check/upload? 
Would it be ok to resue one of the servers, if so which host would you prefer I keep rhel 7.2 or 7.3? 

Thanks

Comment 7 Dmitry Tantsur 2016-11-02 09:54:29 UTC
Is "local_interface = eth0" really correct? it does not look like you have an IP there. Anyway, please attach your full undercloud.conf and full undercloud installation logs.

Comment 9 Tzach Shefi 2016-11-02 12:48:15 UTC
FYI 
Thanks to mariusc the great :) 

He figured out root cause of problem, lack of CPU support for SSSE3. 
See this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1378501

Cougar06 and 07 have an AMD 4170 HE which by the looks of it lack SSSE3 support. 
And fail to install. 

While my other server cougar09 uses an AMD 4334 CPU, which does include SSSE3 and therefor explains why it did pass. 

Gut feeling was sort of right Intel (creator of SSSE3) has better support for this so it didn't show up on Intel based servers only on AMD.
https://en.wikipedia.org/wiki/SSSE3

Guess this should be re-closed as not a bug? 

Unsure which CPUs the bug reported Sai used. 


Adding files Dmitry asked for just in case.

Comment 10 Tzach Shefi 2016-11-02 12:48:53 UTC
Created attachment 1216530 [details]
undercloud.conf and install.log

Comment 11 Dmitry Tantsur 2016-11-02 12:50:40 UTC
Wow, I would never guess that Oo mariusc++ indeed. I'm closing this as not a bug, thanks for the update.