Bug 1299294

Summary: keystone errors in Overcloud deployment with IPv6 enabled
Product: Red Hat OpenStack Reporter: Dan Sneddon <dsneddon>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 7.0 (Kilo)CC: athomas, dmacpher, gchamoul, gdubreui, jcoufal, mburns, mcornea, rhel-osp-director-maint, sathlang
Target Milestone: y3   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-107.el7ost Doc Type: Bug Fix
Doc Text:
In an IPv6-based Overcloud, RabbitMQ lacked some IPv6-specific options when starting. This caused RabbitMQ to fail on some nodes, which cause other services to fail due to Pacemaker contraints. This fix adds the IPv6-specific options. Now RabbitMQ and other services start successfully.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-18 16:50:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan Sneddon 2016-01-18 00:40:20 UTC 
Description of problem:
When IPv6 is enabled on the latest poodle, the deployment is failing with an error that mentions Keystone. I'm not sure where the error is coming from.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates.noarch     0.8.6-103.el7ost

How reproducible:
100%

Steps to Reproduce:
1. Deploy with IPv6 enabled
2.
3.

Actual results:
Error: Could not prefetch keystone_tenant provider 'openstack': undefined method `collect' for nil:NilClass\u001b[0m\n\u001b[1;31mError: Could not prefetch keystone_role provider 'openstack': undefined method `collect' for nil:NilClass\u001b[0m\n\u001b[1;31m
Error: Could not prefetch keystone_user provider 'openstack': undefined method `collect' for nil:NilClass\u001b[0m\n\u001b[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Could not evaluate: undefined method `empty?' for nil:NilClass\u001b[0m\n\u001b[1;31

Expected results:
This shouldn't happen.

Additional info:
Comment 2 Sofer Athlan-Guyot 2016-01-18 09:25:12 UTC 
This is usually due to the kesytone service has not started properly.

The issue here is that it tries to grab the domain and as the response is coming back empty it triggers this exception.
Comment 5 Gilles Dubreuil 2016-01-19 05:49:28 UTC 
I seems to me OPM is in a wrong state.
Effectively Keystone module is quite old (openstack-puppet-modules-2015.1.8-41.el7ost.noarch latest puddle + poodle (OSP 7):
commit 920421a70d33e751c4ec1a665e74819d1c1354d0
Date:   Tue Jun 30 14:37:26 2015 +0000

All other modules seems way behind as well. As a matter of fact, testing with latest keystone module from stable/liberty branch doesn't work because many dependent modules are behind too.
Comment 6 Sofer Athlan-Guyot 2016-01-19 15:00:01 UTC 
Oki, the problem is in that puppet is using the admin_bind_host in /etc/keystone/keystone.conf value to find the keystone server.  This variable is set to the local ip, not the vip.

Then when puppet try to get to the keystone, it works only on the one where the keystone server is started.

Pacemaker policy for keystone is a clone resource, so only one keystone is started.
Comment 7 Sofer Athlan-Guyot 2016-01-20 05:12:58 UTC 
The problem is actually unrelated to keystone.  The issue here is that rabbimq is failing to start on 2 out of 3 nodes.  As there is a pacemaker constraint, saying the keystone start after rabbitmq, the keystone service was not started on 2 nodes.

The rabbimq tht seems to be missing some IPv6 information to start correctly.
Comment 9 Gilles Dubreuil 2016-01-20 06:25:44 UTC 
Please ignore my comment#5, OSP 7 is Kilo. My confusion came from the fact it's pointing to a commit (the one corresponding to kilo) on liberty branch.
Comment 11 Marius Cornea 2016-02-12 12:44:30 UTC 
I wasn't able to reproduce this with openstack-tripleo-heat-templates-0.8.6-120.el7ost.noarch.
Comment 13 errata-xmlrpc 2016-02-18 16:50:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0264.html