Description of problem: When deploying RDO trunk with IPv6, the deployment fails in post-deployment due to an error in the rabbitmq configuration in os-collect-config. Version-Release number of selected component (if applicable): RDO trunk rabbitmq-server-3.3.5-6.el7.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy using network isolation and IPv6 2. 3. Actual results: Mar 14 21:41:52 overcloud-controller-0 os-collect-config[3709]: Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]: Dependency Exec[/var/lib/rabbitmq/rabbitmqadmin] has failures: true Mar 14 21:41:52 overcloud-controller-0 os-collect-config[3709]: Error: curl -k --noproxy localhost --retry 30 --retry-delay 6 -f -L -o /var/lib/rabbitmq/rabbitmqadmin http://guest:guest@fd00:fd00:fd00:2000::13:15672/cli/rabbitmqadmin returned 7 instead of one of [0] Mar 14 21:41:52 overcloud-controller-0 os-collect-config[3709]: Error: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/Staging::File[rabbitmqadmin]/Exec[/var/lib/rabbitmq/rabbitmqadmin]/returns: change from notrun to 0 failed: curl -k --noproxy localhost --retry 30 --retry-delay 6 -f -L -o /var/lib/rabbitmq/rabbitmqadmin http://guest:guest@fd00:fd00:fd00:2000::13:15672/cli/rabbitmqadmin returned 7 instead of one of [0] Mar 14 21:41:52 overcloud-controller-0 os-collect-config[3709]: Warning: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]: Skipping because of failed dependencies Expected results: Deployment should complete Additional info:
It sounds like brackets are missing when passing rabbitmq parameter: RABBITMQ_NODE_IP_ADDRESS. Can you check we have it?
(In reply to Emilien Macchi from comment #1) > It sounds like brackets are missing when passing rabbitmq parameter: > RABBITMQ_NODE_IP_ADDRESS. > > Can you check we have it? No, we're not adding the brackets in any of the applicable parameters, but these are raw host IPs. The brackets should only appear when being rendered with the port, so I think in this case the brackets should be added in the Puppet modules that convert these IPs to URLs. [root@overcloud-controller-0 hieradata]# grep rabbit * all_nodes.yaml:ceilometer::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14'] all_nodes.yaml:cinder::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14'] all_nodes.yaml:glance::notify::rabbitmq::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14'] all_nodes.yaml:heat::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14'] all_nodes.yaml:keystone::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14'] all_nodes.yaml:neutron::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14'] all_nodes.yaml:nova::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14'] all_nodes.yaml:rabbit_node_ips: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14'] all_nodes.yaml:sahara::rabbit_hosts: ['fd00:fd00:fd00:2000::13','fd00:fd00:fd00:2000::15','fd00:fd00:fd00:2000::14']
Quick update: Attila Darazs is currently working on [1] to get IPv6 tested by TripleO CI but is stuck because a RabbitMQ issue in Puppet catalog [2], reported by Dan Sneddon. [1] https://review.openstack.org/#/c/289445 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1317693 [2] is caused by a patch in puppetlabs-rabbitmq [3], that change the way we validate RabbitMQ is working from testing localhost to testing the actual binding IP. [3] https://github.com/puppetlabs/puppetlabs-rabbitmq/commit/dac8de9d95c5771b7ef7596b73a59d4108138e3a The problem is that when testing the actual IPv6, it curls fails for some different reasons explained on [4] by Sofer. [4] https://review.openstack.org/#/c/292664/ So we need to investigate puppetlabs-rabbitmq and puppet-staging to see if whether or not we need to change something there. For now, I don't think we need to patch anything in TripleO Heat Templates, but we'll see after the investigation.
https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/443 https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/444 https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/445 and https://github.com/puppetlabs/puppetlabs-stdlib/pull/577 Should fix this bug in TripleO upstream. If it's confirmed to work, we need to update OPM.
This bug is against a Version which has reached End of Life. If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.