Created attachment 941967 [details] Logs files Description of problem: When using Packstack to install RHOS5 AIO over RHEL7 with SSL enabled Horizon and AMQP, rabbitmq service fails to start. 10.35.160.137_amqp.pp: [ ERROR ] Applying Puppet manifests [ ERROR ] ERROR : Error appeared during Puppet run: 10.35.160.137_amqp.pp Error: Could not start Service[rabbitmq-server]: Execution of '/usr/bin/systemctl start rabbitmq-server' returned 1: Job for rabbitmq-server.service failed. See 'systemctl status rabbitmq-server.service' and 'journalctl -xn' for details. Version-Release number of selected component (if applicable): RHEL7 openstack-packstack-2014.1.1-0.41.dev1251.el7ost.noarch openstack-packstack-puppet-2014.1.1-0.41.dev1251.el7ost.noarch RHOS5 repo from 17-Sep-2014 How reproducible: Every time happened on three setups. Steps to Reproduce: 1. Generate answer file, other than usual setting (attached answer file) set these two: CONFIG_HORIZON_SSL=y CONFIG_AMQP_ENABLE_SSL=y 2. Run packstack with answer file 3. Fails to start rabbitmq.service. 4. Manual restart doesn't help, for debugging disabled firewall / selinux still can't start service. Actual results: Failed to start rabbitmq service, packstack run failed, see attached logs. See attached rabbit {rabbit,failure_during_boot, {case_clause,{error,{already_started,<0.275.0>}}}}}}}} Not sure this is OK or wrong, but under /etc/rabbitmq/ssl there are no files. Expected results: Packstack should manage to start rabbtimq service, plus complete SSL based AIO deployment successfully. Additional info: lbezdick (thanks) pointed me to an upstream puppet problem: https://bugs.launchpad.net/puppet-neutron/+bug/1356083 Also suggested using new amqp.pp from https://review.openstack.org/#/c/99649/ deployment passed amqp step but failed later on, this time on _nova.pp: AMQP server on 10.35.160.137:5671 is unreachableAdd Nova logs under folder called nova.pp Added both original amqp.pp.org and new version amqp.pp.
Hi Tzach, Could you please provide rabbitmq logs? Regards, Gilles
It seems, although the CONFIG_AMQP_SSL_SELF_SIGNED option is present in the answer file, that certificate file is missing. The default self-signed certificate and key files are supposed to be: /etc/pki/tls/certs/amqp_selfcert.pem /etc/pki/tls/private/amqp_selfkey.pem Could you please verify and provide files content if existing?
Certificate files were created, attaching them again plus rabbitmq logs.
Created attachment 942175 [details] Certificate files
Created attachment 942176 [details] Selfcert
Created attachment 942177 [details] rabbitmq log
Hi Tzach, Thanks for the update. Could you please confirm following: After using the new manifest amqp.pp mentioned by Lukas, Rabbitmq service installs correctly and is up and running? If that the case then it's a different issue and I believe we need to track it down separately. Regards
Hi Gilles, Confirm after using new manifest amqp.pp service looks up and running. Service is up and running: [root@cougar08 rabbitmq]# systemctl -t service -a | grep rabbit rabbitmq-server.service loaded active running RabbitMQ broker Port is open and listening: Rabbitmq ssl port 5671 looks OK [root@cougar08 rabbitmq]# netstat -lnp | grep 5671 tcp6 0 0 :::5671 :::* LISTEN 13589/beam.smp [root@cougar08 rabbitmq]# firewall rule added: -A INPUT -s 10.35.160.137/32 -p tcp -m multiport --dports 5671,5672 -m comment --comment "001 amqp incoming amqp_10.35.160.137" -j ACCEPT Yet I still can't explain how then Nova can't reach rabbitmq Notice Nova's journalctl -xn error -> 2014-09-28 11:20:20.967 17723 ERROR oslo.messaging._drivers.impl_rabbit [req-0e21fb20-fbc0-47bd-851f-172210e65d63 - - - - -] AMQP server on 10.35.160.137:5671 is unreachable: Socket closed. Trying again in 30 seconds. Sep 28 11:20:21 cougar08.scl.lab.tlv.redhat.com cinder-backup[17630]: 2014-09-28 11:20:21.066 17630 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.35.160.137:5671 is unreachable: Socket closed. Trying again in 30 seconds. Do you still recommend following it up as a new bug for Nova?
Yes, it seems the initial issue has been fixed, having Rabbitmq listening on ssl port. The other openstack services which cannot reach rabbitmq seems to be related to comment#1 mentioning upstream issue, should effectively be followed up separately. The amqp.pp patch will also make its way to the build.
Created new Packstack bug for Nova bug described on comment#8 https://bugzilla.redhat.com/show_bug.cgi?id=1147823
current icehouse packstack code does not have this problem.