Bug 1373233

I'm adding this comment from the ticket directly here. It contains some analysis of the issue and in the very end I suggest a workaround which was confirmed by Cyril

===========================================================================

Hello,

The bug that we are talking about seems to be (partly) by design and comes back to what I mentioned in my previous email. `rabbit_port` should only be set in the case of one controller only (even though you proved that it works for multiple controllers as well):
~~~
  if $rpc_backend == 'rabbit' or $rpc_backend == 'neutron.openstack.common.rpc.impl_kombu' {
    if ! $rabbit_password {
      fail('When rpc_backend is rabbitmq, you must set rabbit password')
    }
    if $rabbit_hosts {
      neutron_config { 'oslo_messaging_rabbit/rabbit_hosts':     value  => join($rabbit_hosts, ',') }
      neutron_config { 'oslo_messaging_rabbit/rabbit_ha_queues': value  => true }
    } else  {
      neutron_config { 'oslo_messaging_rabbit/rabbit_host':      value => $rabbit_host }
      neutron_config { 'oslo_messaging_rabbit/rabbit_port':      value => $rabbit_port }
      neutron_config { 'oslo_messaging_rabbit/rabbit_hosts':     value => "${rabbit_host}:${rabbit_port}" }
      neutron_config { 'oslo_messaging_rabbit/rabbit_ha_queues': value => false }
    }
~~~

For as long as `rabbit_hosts` is provided, which is the case in any deployment at least of OSP 8 and later, regardless of the number of controller nodes. You can validate this by having a look at 
~~~
[root@overcloud-controller-0 ~]# grep rabbit_host /etc/neutron/neutron.conf 
# rabbit_host=localhost
# rabbit_hosts=$rabbit_host:$rabbit_port
# Deprecated group/name - [DEFAULT]/rabbit_host
# rabbit_host = localhost
# Deprecated group/name - [DEFAULT]/rabbit_hosts
# rabbit_hosts = $rabbit_host:$rabbit_port
rabbit_hosts = 172.16.2.7
[root@overcloud-controller-0 ~]# grep rabbit_host /etc/puppet/hieradata/* -R
/etc/puppet/hieradata/all_nodes.yaml:ceilometer::rabbit_hosts: ['172.16.2.7']
/etc/puppet/hieradata/all_nodes.yaml:cinder::rabbit_hosts: ['172.16.2.7']
/etc/puppet/hieradata/all_nodes.yaml:glance::notify::rabbitmq::rabbit_hosts: ['172.16.2.7']
/etc/puppet/hieradata/all_nodes.yaml:heat::rabbit_hosts: ['172.16.2.7']
/etc/puppet/hieradata/all_nodes.yaml:keystone::rabbit_hosts: ['172.16.2.7']
/etc/puppet/hieradata/all_nodes.yaml:neutron::rabbit_hosts: ['172.16.2.7']
/etc/puppet/hieradata/all_nodes.yaml:nova::rabbit_hosts: ['172.16.2.7']
[root@overcloud-controller-0 ~]# 
~~~

I think that RabbitClientPort could be an option, although I will need to validate this.
===>
~~~
parameters:
  RabbitClientUseSSL:true
  RabbitClientPort: 5671

  controllerExtraConfig:
    rabbitmq::ssl: True
    rabbitmq::ssl_cert: /etc/pki/tls/private/overcloud_endpoint.pem
    rabbitmq::ssl_key: /etc/pki/tls/private/overcloud_endpoint.pem
    rabbitmq::ssl_port: 5671
    neutron::rabbit_port: 5671
    neutron::rabbit_use_ssl: True
~~~

Also, I _do_ have a doubt abouter RabbitClientPort, because of:
~~~
cat /usr/share/openstack-tripleo-heat-templates/overcloud-without-mergepy.yaml
(...)
  RabbitClientUseSSL:
    default: false
    description: >
        Rabbit client subscriber parameter to specify
        an SSL connection to the RabbitMQ host.
    type: string
  RabbitClientPort:
    default: 5672
    description: Set rabbit subscriber port, change this if using SSL
    type: number
(...)
~~~

Which sets neutron::rabbit_port, which of course doesn't work (see above)
~~~
/usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml:        rabbit_client_port: {get_param: RabbitClientPort}
/usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml:                cinder::rabbit_port: {get_input: rabbit_client_port}
/usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml:                heat::rabbit_port: {get_input: rabbit_client_port}
/usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml:                keystone::rabbit_port: {get_input: rabbit_client_port}
/usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml:                neutron::rabbit_port: {get_input: rabbit_client_port}
/usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml:                ceilometer::rabbit_port: {get_input: rabbit_client_port}
/usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml:                nova::rabbit_port: {get_input: rabbit_client_port}
~~~

Which means that we'd have to look for a way to change the port for `rabbit_nodes`
~~~
/usr/share/openstack-tripleo-heat-templates/puppet/manifests/overcloud_controller.pp:  $rabbit_nodes = hiera('rabbit_node_ips')
~~~

Which is set by the following
~~~
/usr/share/openstack-tripleo-heat-templates/overcloud-without-mergepy.yaml:      rabbit_node_ips: {get_attr: [ControllerIpListMap, net_ip_map, {get_param: [ServiceNetMap, RabbitMqNetwork]}]}
~~~

Unfortunately,  cannot find anything that looks as if it were made for this port change ...

If `RabbitClientPort` does not help (as I would expect right now), a workaround for this issue could be the following:
~~~
parameter_defaults:
  controllerExtraConfig:
    neutron::config::server_config:
      oslo_messaging_rabbit/rabbit_port:
        value: '5671'

   controller_classes:
      - ::neutron::config
~~~

I am going to try both variations in my lab, but you may actually me faster (I just came back from vacation and have to setup my labs again).

Regards,

Andreas

As per the analysis in #c2, puppet-oslo is sidestepping this parameter. The suggested workaround might work but it may be preferable to amend the rabbit_hosts value to be host_ip:port.

This also affects all services, not just neutron.

adding only 'neutron::rabbit_port: 5671' to an environment file and deploying did not set the value as expected. Do I need to do a full SSL deployment to see this change?

You should set the heat template parameter, not the hieradata. There is logic in the puppet-tripleo module that consumes the heat data and ensures the proper placement of rabbit host and port data.

Target release is 10 in this case, and templates are completely different in 10. In any event, OOB the rabbitmq is set so the initial issue of not being able to deploy is not relevant and is irrelevant in 7,8 and 9 since those are already out in the wild.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html