Bug 1317693

Summary:	RabbitMQ failures when deploying RDO with IPv6
Product:	[Community] RDO	Reporter:	Dan Sneddon <dsneddon>
Component:	openstack-tripleo-heat-templates	Assignee:	Emilien Macchi <emacchi>
Status:	CLOSED EOL	QA Contact:	Shai Revivo <srevivo>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	trunk	CC:	emacchi
Target Milestone:	---
Target Release:	Kilo
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-05-19 15:37:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Dan Sneddon 2016-03-14 22:00:44 UTC

Description of problem:
When deploying RDO trunk with IPv6, the deployment fails in post-deployment due to an error in the rabbitmq configuration in os-collect-config.

Version-Release number of selected component (if applicable):
RDO trunk
rabbitmq-server-3.3.5-6.el7.noarch


How reproducible:
100%

Steps to Reproduce:
1. Deploy using network isolation and IPv6
2.
3.

Actual results:
Mar 14 21:41:52 overcloud-controller-0 os-collect-config[3709]: Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]: Dependency Exec[/var/lib/rabbitmq/rabbitmqadmin] has failures: true
Mar 14 21:41:52 overcloud-controller-0 os-collect-config[3709]: Error: curl -k --noproxy localhost --retry 30 --retry-delay 6 -f -L -o /var/lib/rabbitmq/rabbitmqadmin http://guest:guest@fd00:fd00:fd00:2000::13:15672/cli/rabbitmqadmin returned 7 instead of one of [0]
Mar 14 21:41:52 overcloud-controller-0 os-collect-config[3709]: Error: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/Staging::File[rabbitmqadmin]/Exec[/var/lib/rabbitmq/rabbitmqadmin]/returns: change from notrun to 0 failed: curl -k --noproxy localhost --retry 30 --retry-delay 6 -f -L -o /var/lib/rabbitmq/rabbitmqadmin http://guest:guest@fd00:fd00:fd00:2000::13:15672/cli/rabbitmqadmin returned 7 instead of one of [0]
Mar 14 21:41:52 overcloud-controller-0 os-collect-config[3709]: Warning: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]: Skipping because of failed dependencies

Expected results:
Deployment should complete

Additional info:

Comment 1 Emilien Macchi 2016-03-15 00:04:57 UTC

It sounds like brackets are missing when passing rabbitmq parameter: RABBITMQ_NODE_IP_ADDRESS.

Can you check we have it?

Comment 2 Dan Sneddon 2016-03-15 00:09:24 UTC

(In reply to Emilien Macchi from comment #1)
> It sounds like brackets are missing when passing rabbitmq parameter:
> RABBITMQ_NODE_IP_ADDRESS.
> 
> Can you check we have it?

No, we're not adding the brackets in any of the applicable parameters, but these are raw host IPs. The brackets should only appear when being rendered with the port, so I think in this case the brackets should be added in the Puppet modules that convert these IPs to URLs.

[root@overcloud-controller-0 hieradata]# grep rabbit *
all_nodes.yaml:ceilometer::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
all_nodes.yaml:cinder::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
all_nodes.yaml:glance::notify::rabbitmq::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
all_nodes.yaml:heat::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
all_nodes.yaml:keystone::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
all_nodes.yaml:neutron::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
all_nodes.yaml:nova::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
all_nodes.yaml:rabbit_node_ips: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
all_nodes.yaml:sahara::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']

Comment 3 Emilien Macchi 2016-03-15 17:17:39 UTC

Quick update:

Attila Darazs is currently working on [1] to get IPv6 tested by TripleO CI but is stuck because a RabbitMQ issue in Puppet catalog [2], reported by Dan Sneddon.
[1] https://review.openstack.org/#/c/289445
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1317693

[2] is caused by a patch in puppetlabs-rabbitmq [3], that change the way we validate RabbitMQ is working from testing localhost to testing the actual binding IP.
[3] https://github.com/puppetlabs/puppetlabs-rabbitmq/commit/dac8de9d95c5771b7ef7596b73a59d4108138e3a

The problem is that when testing the actual IPv6, it curls fails for some different reasons explained on [4] by Sofer.
[4] https://review.openstack.org/#/c/292664/

So we need to investigate puppetlabs-rabbitmq and puppet-staging to see if whether or not we need to change something there.
For now, I don't think we need to patch anything in TripleO Heat Templates, but we'll see after the investigation.

Comment 4 Emilien Macchi 2016-03-16 13:35:30 UTC

https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/443
https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/444
https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/445

and
https://github.com/puppetlabs/puppetlabs-stdlib/pull/577

Should fix this bug in TripleO upstream.
If it's confirmed to work, we need to update OPM.

Comment 6 Chandan Kumar 2016-05-19 15:37:37 UTC

This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.