Bug 1128170

i'll create a pull request for openstack puppet modules

The latest OPM RPM hast the required functionality openstack-puppet-modules-2014.1-21.8.el6ost

Ivan,

Tried to reproduce the issue on latest RHEL 6.5/RHOS setup and observed error message with
rabbitmq. Could you please check the steps below. Let me know if you need more information or
need to access the setup for any triage purpose.

Steps
==========
0) Installed RHOS allinone using packstack (openstack-puppet-modules-2014.1-21.8.el6ost did get installed).

1) Stopped openstack services using "openstack-service stop"

2) Created and added the certificates required for SSL communication as below,

    a) mkdir /etc/pki/rabbitmq
    b) cd /etc/pki/rabbitmq
    c) Generated .keyfile using "openssl genrsa -des3 -out 10.35.115.14.key 1024
    d) Generated certificate signing request using "openssl req -new -key 10.35.115.14.key -out 10.35.115.14.csr"
    e) Generated a self signed certificate using "openssl x509 -req -days 365 -in 10.35.115.14.csr -signkey 10.35.115.14.key -out 10.35.115.14.crt"

3) Edited /etc/rabbitmq/rabbitmq.config as below,
   [
     {rabbit, [
        {ssl_listeners, [5671]},
        {ssl_options, [{certfile,"/etc/pki/rabbitmq/10.35.115.14.crt"},
                       {keyfile,"/etc/pki/rabbitmq/10.35.115.14.key"},
                       {verify,verify_peer},
                       {fail_if_no_peer_cert,false}]}
      ]}
    ].
    
4) Enabled SSL Communication Between Rabbitmq and Cinder as below,

    openstack-config --set /etc/cinder/cinder.conf DEFAULT rabbit_use_ssl True
    openstack-config --set /etc/cinder/cinder.conf DEFAULT kombu_ssl_certfile /etc/pki/rabbitmq/10.35.115.14.crt
    openstack-config --set /etc/cinder/cinder.conf DEFAULT kombu_ssl_keyfile /etc/pki/rabbitmq/10.35.115.14.key
    
5) Restarted Openstack services using ""openstack-service start"

6) Tried to create volume types and volumes and observed the following error in /var/log/cinder/api.log and volume.log.
       2014-09-26 17:55:51.107 9692 ERROR oslo.messaging._drivers.impl_rabbit [req-045b85d7-d29c-402e-85b6-523bfb9bb1a8 2debfa0585244fb998d3feb23555eb64 
       fcdab35549824c3b93892120be0f936b - - -] AMQP server on 10.35.115.14:5672 is unreachable: 
       [Errno 336265225] _ssl.c:341: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib. 
       Trying again in 30 seconds.
       
      
   Looks like the message broker is unreachable. Rabbitmq status below shows that it is running.   
   Any idea why this may be happening? 

rabbitmqctl status
===================
Status of node 'rabbit@stone-vds1' ...
[{pid,9625},
 {running_applications,[{rabbit,"RabbitMQ","3.1.5"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {xmerl,"XML parser","1.3.5"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,[{total,70274544},
          {connection_procs,2406248},
          {queue_procs,1020408},
          {plugins,0},
          {other_proc,13821056},
          {mnesia,252824},
          {mgmt_db,0},
          {msg_index,90528},
          {other_ets,860736},
          {binary,29779168},
          {code,16428070},
          {atom,594537},
          {other_system,5020969}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,6684953804},
 {disk_free_limit,1000000000},
 {disk_free,122720583680},
 {file_descriptors,[{total_limit,924},
                    {total_used,48},
                    {sockets_limit,829},
                    {sockets_used,46}]},
 {processes,[{limit,1048576},{used,588}]},
 {run_queue,0},
 {uptime,85976}]
...done.

The root of the problem is you're still using port 5672. You need to use 5671.

IMHO you shouldn't set the kombu_* variables. AFAICT they are used for identity so you'd be presenting the server with its own cert.

I'd double-check file permissions as well on the cert and key.

I wonder if you'll ultimately have SSL trust issues with a self-signed cert.

Rob,

Thank you. 

1) I see that I have used 5671 in the ssl config. I changed it to 5672, instead of making the change on rabbitmq side. Is that ok?

[
  {rabbit, [
     {ssl_listeners, [5672]},
     {ssl_options, [{certfile,"/etc/pki/rabbitmq/10.35.115.14.crt"},
                    {keyfile,"/etc/pki/rabbitmq/10.35.115.14.key"},
                    {verify,verify_peer},
                    {fail_if_no_peer_cert,false}]}
   ]}
].


2) Removed kombu variables set before.

3) Here are the permissions for the certificate and key file,

-rw-r--r--. 1 root root 948 Sep 26 16:12 /etc/pki/rabbitmq/10.35.115.14.crt
-rw-r--r--. 1 root root 951 Sep 26 16:08 /etc/pki/rabbitmq/10.35.115.14.key

ERROR MESSAGE
==============

I still see the following error. This time a different one about an unknown protocol.

2014-09-29 16:48:18.259 18560 ERROR oslo.messaging._drivers.impl_rabbit [req-71914bd1-92fc-4796-8c7d-515e905102c2 2debfa0585244fb998d3feb23555eb64 fcdab35549824c3b93892120be0f936b - - -] AMQP server on 10.35.115.14:5672 is unreachable: [Errno 1] _ssl.c:492: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Trying again in 5 seconds.

Thanks
Prasanth

I think you need to check the rabbit server logs to see if the server came up successfully with SSL enabled.

I'll attach a simple python script that can be used to test rabbit SSL directly to exclude any client (mis)configuration issues.

You just need to change the hostname and rabbit password in the file. You can get the password rabbit_password in from nova.conf.

Created attachment 942414 [details]
rabbit AMQP test script

Edit the script to set the FQDN (in your case IP address I guess) and the rabbit password, then run: python rabbit.py and you should get "Hello World!" back.

Here is the result from the script. Also, the rabbitmq logs show that it is accepting connectiions, but then closes it out with an error report. Is there any way to get more verbose error message?


[root@stone-vds1 ~]# python rabbit.py 
Traceback (most recent call last):
  File "rabbit.py", line 12, in <module>
    lConnection = amqp.Connection(host=server, userid=username, password=password, ssl=True)
  File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/connection.py", line 129, in __init__
    self.transport = create_transport(host, connect_timeout, ssl)
  File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 279, in create_transport
    return SSLTransport(host, connect_timeout, ssl)
  File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 179, in __init__
    super(SSLTransport, self).__init__(host, connect_timeout)
  File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 91, in __init__
    self._setup_transport()
  File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 193, in _setup_transport
    self.sslobj = ssl.wrap_socket(self.sock)
  File "/usr/lib64/python2.6/ssl.py", line 342, in wrap_socket
    suppress_ragged_eofs=suppress_ragged_eofs)
  File "/usr/lib64/python2.6/ssl.py", line 120, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.6/ssl.py", line 279, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [Errno 1] _ssl.c:492: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Exception socket.error: (107, 'Transport endpoint is not connected') in <bound method SSLTransport.__del__ of <amqplib.client_0_8.transport.SSLTransport object at 0x7f4c3bc67f10>> ignored




RABBITMQ LOGS
======================

=INFO REPORT==== 29-Sep-2014::20:00:08 ===
accepting AMQP connection <0.3116.34> (10.35.115.14:38608 -> 10.35.115.14:5672)

=ERROR REPORT==== 29-Sep-2014::20:00:08 ===
closing AMQP connection <0.3116.34> (10.35.115.14:38608 -> 10.35.115.14:5672):
{bad_header,<<22,3,1,0,244,1,0,0>>}

=INFO REPORT==== 29-Sep-2014::20:00:26 ===
accepting AMQP connection <0.3190.34> (10.35.115.14:38609 -> 10.35.115.14:5672)

=ERROR REPORT==== 29-Sep-2014::20:00:26 ===
closing AMQP connection <0.3190.34> (10.35.115.14:38609 -> 10.35.115.14:5672):
{bad_header,<<22,3,1,0,244,1,0,0>>}

=INFO REPORT==== 29-Sep-2014::20:00:38 ===
accepting AMQP connection <0.3207.34> (10.35.115.14:38617 -> 10.35.115.14:5672)

=ERROR REPORT==== 29-Sep-2014::20:00:38 ===
closing AMQP connection <0.3207.34> (10.35.115.14:38617 -> 10.35.115.14:5672):
{bad_header,<<22,3,1,0,244,1,0,0>>}

=INFO REPORT==== 29-Sep-2014::20:00:38 ===
accepting AMQP connection <0.3237.34> (10.35.115.14:38618 -> 10.35.115.14:5672)

=ERROR REPORT==== 29-Sep-2014::20:00:38 ===
closing AMQP connection <0.3237.34> (10.35.115.14:38618 -> 10.35.115.14:5672):
{bad_header,<<22,3,1,0,244,1,0,0>>}

I don't know rabbitmq-server well so don't know a way to get more verbose debugging.

I don't think you can force the port to 5672 like this. I'm surprised the server started at all. The default RABBITMQ_NODE_PORT is 5672 and this listener is always configured, so you created a conflicting configuration. I'd stick with 5671 as it is the standard AMQP SSL port and it is what the OpenStack clients should be using when it comes to SSL.

I see the same error as you when I try to connect an SSL client to a non-SSL server.

My rabbitmq.config looks like:

[
  {rabbit, [

    {ssl_listeners, [5671]},
    {ssl_options, [{cacertfile,"/etc/ipa/ca.crt"},
                    {certfile,"/etc/pki/tls/certs/set4client1.private.example-amqp.crt"},
                    {keyfile,"/etc/pki/tls/private/set4client1.private.example-amqp.key"},
                    {verify,verify_none},
                    {fail_if_no_peer_cert,false}]},
    {default_user, <<"openstack">>},
    {default_pass, <<"9ca8271764fe05655579bac5738cdd89">>}
  ]},
  {kernel, [
    
  ]}
].

After making the modifications above and generating certificates based on 
https://openstack.redhat.com/Securing_Services_Foreman, I was able to verify
rabbitmq communication with SSL enabled. Moving the bug to Verified.


SSL Listener comes up after enabling Rabbimq with SSL
=====================================================

=INFO REPORT==== 30-Sep-2014::00:45:42 ===
started SSL Listener on [::]:5671


CINDER VOLUME OPERATION
=======================

Logs show that cinder operation is successful and is happening through SSL port 5671

2014-09-30 00:45:56.588 20460 AUDIT cinder.service [-] Starting cinder-volume node (version 2014.1.2)
2014-09-30 00:45:56.589 20460 INFO cinder.volume.manager [req-a66fb2d3-4426-4f84-9e94-3075cc98e1cd - - - - -] Starting volume driver LVMISCSIDriver (2.0.0)
2014-09-30 00:45:57.818 20460 INFO cinder.volume.manager [req-a66fb2d3-4426-4f84-9e94-3075cc98e1cd - - - - -] Updating volume status
2014-09-30 00:45:58.098 20460 INFO oslo.messaging._drivers.impl_rabbit [req-a66fb2d3-4426-4f84-9e94-3075cc98e1cd - - - - -] Connected to AMQP server on 10.35.115.14:5671
2014-09-30 00:45:58.132 20460 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.35.115.14:5671
2014-09-30 00:46:29.148 20460 INFO cinder.volume.manager [req-be13c55d-8008-407b-9059-720d978c3d6f - - - - -] Updating volume status


LOGS FROM DEVELOPER SCRIPT
===========================
The following logs show output from developer's script testing that basic SSL is working.

[root@stone-vds1 ~]# python rabbit.py 
Hello World!
[root@stone-vds1 ~]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1350.html

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days