Bug 1097306

Summary: Packstack allinone installation fails when using qpidd as a messaging service
Product: Red Hat OpenStack Reporter: Petr Matousek <pematous>
Component: openstack-packstackAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED ERRATA QA Contact: Ami Jeain <ajeain>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.0 (RHEL 7)CC: acathrow, adahms, aortega, derekh, eglynn, esammons, fpercoco, jross, lbezdick, lhh, ltoscano, lzhaldyb, mmagr, pematous, psedlak, yeylon, zkraus
Target Milestone: rc   
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-packstack-2014.1.1-0.17.dev1102.el7ost Doc Type: Bug Fix
Doc Text:
Previously, the PackStack all-in-one installation process would time out and fail under certain conditions when QPID was specified as the messaging service. This was caused by the logic used to interact with QPID, whereby PackStack would specify a deprecated option for QPID during installation of Glance and would specify a different location for the qpidd.conf file for Red Hat Enterprise Linux 7.0. Now, this logic has been revised to update the deprecated option and specify the same location for the qpidd.conf file for all versions of Red Hat Enterprise Linux, allowing the all-in-one installation process to complete all affected steps successfully.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-08 15:38:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1102040, 1143749    
Bug Blocks:    
Attachments:
Description Flags
amqp_qpidd_018.patch none

Description Petr Matousek 2014-05-13 14:16:27 UTC 
Description of problem:

Openstack packstack allinone installation fails with timeout on provisioning when installing with qpid as a messaging service.

By rerunning the command manually (the command that timed out, /usr/bin/glance ..) I also got the timeout, but when I followed he recommendation in the RDO workaround web page:
http://openstack.redhat.com/Workarounds#glance:_Error_communicating_with_http:.2F.2F192.168.8.96:9292_timed_out

The issue was fixed and the new image was added. 
But by rerunning the packstack installation (with --answer-file) I got again to the original timeout issue (the notifier_strategy is set again to qpid in /etc/glance/glance-api.conf by packstack). Please see additional info for details.

So, I'm wondering if the the notifications shall be enabled when installing with qpidd?

Version-Release number of selected component (if applicable):
qpid-cpp-*-0.18-22

How reproducible:
100%

Steps to Reproduce:
1. install qpid-0.18-22 packages
2. packstack --allinone --amqp-server=qpid
3. Installation fails

Actual results:
Installation fails

Expected results:
Installation succeeds

Additional info:

# cat /var/tmp/packstack/20140513-051329-bboJzs/manifests/10.16.65.238_provision.pp.log
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for intel-s3eb1-01.rhts.eng.bos.redhat.com in environment production in 0.21 seconds
Notice: /Stage[main]/Main/Firewall[000 forward in]/ensure: created
Notice: /Stage[main]/Main/Firewall[000 nat]/ensure: created
Error: Execution of '/usr/bin/glance -T services -I glance -K bb55dfcf8d1743f4 -N http://10.16.65.238:35357/v2.0/ add name=cirros is_public=Yes container_format=bare disk_format=qcow2 copy_from=http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img' returned 1: Error communicating with http://10.16.65.238:9292 timed out

Error: /Stage[main]/Openstack::Provision/Glance_image[cirros]/ensure: change from absent to present failed: Execution of '/usr/bin/glance -T services -I glance -K bb55dfcf8d1743f4 -N http://10.16.65.238:35357/v2.0/ add name=cirros is_public=Yes container_format=bare disk_format=qcow2 copy_from=http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img' returned 1: Error communicating with http://10.16.65.238:9292 timed out

Notice: /Stage[main]/Main/Firewall[000 forward out]/ensure: created
Error: Could not prefetch neutron_network provider 'neutron': Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: Connection to neutron failed: Maximum attempts reached

Error: Execution of '/usr/bin/neutron net-create --format=shell --tenant_id=1869ca35720e4f62b3de1a8c8ebc04de --router:external=True public' returned 1: Connection to neutron failed: [Errno 111] Connection refused

Error: /Stage[main]/Openstack::Provision/Neutron_network[public]/ensure: change from absent to present failed: Execution of '/usr/bin/neutron net-create --format=shell --tenant_id=1869ca35720e4f62b3de1a8c8ebc04de --router:external=True public' returned 1: Connection to neutron failed: [Errno 111] Connection refused

Error: Could not prefetch neutron_subnet provider 'neutron': Execution of '/usr/bin/neutron subnet-list --format=csv --column=id --quote=none' returned 1: Connection to neutron failed: Maximum attempts reached

Notice: /Stage[main]/Openstack::Provision/Neutron_subnet[public_subnet]: Dependency Neutron_network[public] has failures: true
Warning: /Stage[main]/Openstack::Provision/Neutron_subnet[public_subnet]: Skipping because of failed dependencies
Notice: /Stage[main]/Openstack::Provision/Neutron_l3_ovs_bridge[br-ex]: Dependency Neutron_network[public] has failures: true
Warning: /Stage[main]/Openstack::Provision/Neutron_l3_ovs_bridge[br-ex]: Skipping because of failed dependencies
Notice: /Stage[main]/Openstack::Provision/Keystone_tenant[demo]/ensure: created
Error: Could not prefetch neutron_router provider 'neutron': Execution of '/usr/bin/neutron router-list --format=csv --column=id --quote=none' returned 1: Connection to neutron failed: Maximum attempts reached

Notice: /Stage[main]/Openstack::Provision/Neutron_router[router1]: Dependency Neutron_network[public] has failures: true
Warning: /Stage[main]/Openstack::Provision/Neutron_router[router1]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/neutron net-create --format=shell --tenant_id=084b9ca965294eac810a64d08e04b323 private' returned 1: Connection to neutron failed: [Errno 111] Connection refused

Error: /Stage[main]/Openstack::Provision/Neutron_network[private]/ensure: change from absent to present failed: Execution of '/usr/bin/neutron net-create --format=shell --tenant_id=084b9ca965294eac810a64d08e04b323 private' returned 1: Connection to neutron failed: [Errno 111] Connection refused

Notice: /Stage[main]/Openstack::Provision/Neutron_subnet[private_subnet]: Dependency Neutron_network[private] has failures: true
Warning: /Stage[main]/Openstack::Provision/Neutron_subnet[private_subnet]: Skipping because of failed dependencies
Error: Could not prefetch neutron_router_interface provider 'neutron': Execution of '/usr/bin/neutron subnet-list --format=csv --column=id --quote=none' returned 1: Connection to neutron failed: Maximum attempts reached

Notice: /Stage[main]/Openstack::Provision/Neutron_router_interface[router1:private_subnet]: Dependency Neutron_network[public] has failures: true
Notice: /Stage[main]/Openstack::Provision/Neutron_router_interface[router1:private_subnet]: Dependency Neutron_network[private] has failures: true
Warning: /Stage[main]/Openstack::Provision/Neutron_router_interface[router1:private_subnet]: Skipping because of failed dependencies
Notice: /Stage[main]/Openstack::Provision/Keystone_tenant[alt_demo]/ensure: created
Notice: /Stage[main]/Openstack::Provision/Keystone_user[alt_demo]/ensure: created
Notice: /Stage[main]/Openstack::Provision/Keystone_user[demo]/ensure: created
Notice: Finished catalog run in 629.08 seconds


===

# packstack --allinone --amqp-server=qpid
<...>
10.16.65.238_ring_swift.pp:                          [ DONE ]         
Applying 10.16.65.238_swift.pp
Applying 10.16.65.238_provision.pp
10.16.65.238_swift.pp:                               [ DONE ]        
10.16.65.238_provision.pp:                        [ ERROR ]          
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.16.65.238_provision.pp
Error: Execution of '/usr/bin/glance -T services -I glance -K bb55dfcf8d1743f4 -N http://10.16.65.238:35357/v2.0/ add name=cirros is_public=Yes container_format=bare disk_format=qcow2 copy_from=http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img' returned 1: Error communicating with http://10.16.65.238:9292 timed out
You will find full trace in log /var/tmp/packstack/20140513-051329-bboJzs/manifests/10.16.65.238_provision.pp.log
Please check log file /var/tmp/packstack/20140513-051329-bboJzs/openstack-setup.log for more information

Additional information:
 * A new answerfile was created in: /root/packstack-answers-20140513-051330.txt
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * File /root/keystonerc_admin has been created on OpenStack client host 10.16.65.238. To use the command line tools you need to source the file.
 * To access the OpenStack Dashboard browse to http://10.16.65.238/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.

# /usr/bin/glance -T services -I glance -K bb55dfcf8d1743f4 -N http://10.16.65.238:35357/v2.0/ add name=cirros is_public=Yes container_format=bare disk_format=qcow2 copy_from=http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img 
<timeout>
# sed -i 's/notifier_strategy = qpid/notifier_strategy = noop/' /etc/glance/glance-api.conf
# systemctl restart openstack-glance-api.service
# # /usr/bin/glance -T services -I glance -K bb55dfcf8d1743f4 -N http://10.16.65.238:35357/v2.0/ add name=cirros is_public=Yes container_format=bare disk_format=qcow2 copy_from=http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img
Added new image with ID: dd2a05e5-b281-4beb-be82-5bd0304b06cb
^^^^ SUCCESS

# packstack --allinone --amqp-server=qpid
<...>
10.16.65.238_provision.pp:                        [ ERROR ]          
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.16.65.238_provision.pp
Error: Execution of '/usr/bin/glance -T services -I glance -K bb55dfcf8d1743f4 -N http://10.16.65.238:35357/v2.0/ add name=cirros is_public=Yes container_format=bare disk_format=qcow2 copy_from=http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img' returned 1: Error communicating with http://10.16.65.238:9292 timed out
You will find full trace in log /var/tmp/packstack/20140513-051329-bboJzs/manifests/10.16.65.238_provision.pp.log
Please check log file /var/tmp/packstack/20140513-051329-bboJzs/openstack-setup.log for more information
Comment 1 Petr Matousek 2014-05-13 14:23:01 UTC 
I forget to list the openstack packages version, please see below:

# rpm -qa | grep openstack
openstack-keystone-2014.1-2.el7ost.noarch
openstack-neutron-openvswitch-2014.1-14.el7ost.noarch
redhat-access-plugin-openstack-5.0.0-2.el7ost.noarch
openstack-swift-proxy-1.13.1-1.el7ost.noarch
openstack-glance-2014.1-2.el7ost.noarch
openstack-swift-container-1.13.1-1.el7ost.noarch
openstack-puppet-modules-2014.1-9.el7ost.noarch
openstack-nova-common-2014.1-2.el7ost.noarch
openstack-nova-console-2014.1-2.el7ost.noarch
openstack-nova-compute-2014.1-2.el7ost.noarch
openstack-nova-scheduler-2014.1-2.el7ost.noarch
openstack-neutron-2014.1-14.el7ost.noarch
python-django-openstack-auth-1.1.5-2.el7ost.noarch
openstack-dashboard-theme-2014.1-4.el7ost.noarch
openstack-swift-object-1.13.1-1.el7ost.noarch
openstack-packstack-2014.1.1-0.11.1.dev1055.el7ost.noarch
openstack-nova-api-2014.1-2.el7ost.noarch
openstack-nova-conductor-2014.1-2.el7ost.noarch
openstack-nova-cert-2014.1-2.el7ost.noarch
openstack-dashboard-2014.1-4.el7ost.noarch
openstack-swift-1.13.1-1.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-utils-2014.1-1.el7ost.noarch
openstack-ceilometer-common-2014.1-2.1.el7ost.noarch
openstack-swift-account-1.13.1-1.el7ost.noarch
openstack-packstack-puppet-2014.1.1-0.11.1.dev1055.el7ost.noarch
openstack-cinder-2014.1-3.el7ost.noarch
openstack-nova-novncproxy-2014.1-2.el7ost.noarch
Comment 2 Dafna Ron 2014-05-13 14:24:49 UTC 
why is this opened on glance and not on packstack or qpid?
Comment 3 Petr Matousek 2014-05-13 14:38:37 UTC 
(In reply to Dafna Ron from comment #2)
> why is this opened on glance and not on packstack or qpid?

I'm very new to openstack, so I'm not very familiar with the components so far, sorry.

-> changing to packstack component.
Comment 4 Martin Magr 2014-05-13 20:23:53 UTC 
If QPID is set as AMQP backend then notification_strategy has to be set to qpid. Changing back to Glance. Not much can be done from our point unless.
Comment 5 Dafna Ron 2014-05-14 08:43:43 UTC 
(In reply to Martin Magr from comment #4)
> If QPID is set as AMQP backend then notification_strategy has to be set to
> qpid. Changing back to Glance. Not much can be done from our point unless.

I'm not sure why you moved this to glance. 
I see we fail to download cirus image but there are multiple errors for Neutron... 
i would think this is related to iptables configuration and not to glance. 

Notice: /Stage[main]/Main/Firewall[000 forward out]/ensure: created
Error: Could not prefetch neutron_network provider 'neutron': Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: Connection to neutron failed: Maximum attempts reached

Error: Execution of '/usr/bin/neutron net-create --format=shell --tenant_id=1869ca35720e4f62b3de1a8c8ebc04de --router:external=True public' returned 1: Connection to neutron failed: [Errno 111] Connection refused

Error: /Stage[main]/Openstack::Provision/Neutron_network[public]/ensure: change from absent to present failed: Execution of '/usr/bin/neutron net-create --format=shell --tenant_id=1869ca35720e4f62b3de1a8c8ebc04de --router:external=True public' returned 1: Connection to neutron failed: [Errno 111] Connection refused
Comment 7 Martin Magr 2014-05-14 10:58:15 UTC 
Hmm, Dafna is right, iptables might be the cause. Petr, can you please paste here output of "sudo iptables -L"? Thanks in advance
Comment 9 Zdenek Kraus 2014-05-14 14:13:25 UTC 
Let have failed install of openstack via packstack with qpid.

I've added a little print of driver beeing loaded at glance-api by oslo.messaging.transport and stevedore.driver like this:
/usr/lib/python2.7/site-packages/oslo/messaging/transport.py
        mgr = driver.DriverManager('oslo.messaging.drivers',
                                   url.transport,
                                   invoke_on_load=True,
                                   invoke_args=[conf, url],
                                   invoke_kwds=kwargs)
        print mgr.driver

diff:
183a184
>       print mgr.driver


and when I launch /usr/bin/glance-api -d this show up:
<oslo.messaging._drivers.impl_rabbit.RabbitDriver object at 0x357fa90>
<oslo.messaging._drivers.impl_qpid.QpidDriver object at 0x429f950>
<oslo.messaging._drivers.impl_rabbit.RabbitDriver object at 0x446b850>
<oslo.messaging._drivers.impl_qpid.QpidDriver object at 0x446b910>
<oslo.messaging._drivers.impl_rabbit.RabbitDriver object at 0x446d490>
<oslo.messaging._drivers.impl_qpid.QpidDriver object at 0x446d550>
<oslo.messaging._drivers.impl_rabbit.RabbitDriver object at 0x446e050>
<oslo.messaging._drivers.impl_qpid.QpidDriver object at 0x446e150>
<oslo.messaging._drivers.impl_rabbit.RabbitDriver object at 0x446ed10>
<oslo.messaging._drivers.impl_qpid.QpidDriver object at 0x446edd0>
<oslo.messaging._drivers.impl_rabbit.RabbitDriver object at 0x450ed90>
<oslo.messaging._drivers.impl_qpid.QpidDriver object at 0x450ee90>
<oslo.messaging._drivers.impl_rabbit.RabbitDriver object at 0x4518090>
<oslo.messaging._drivers.impl_qpid.QpidDriver object at 0x4518190>


is this expected?
Comment 10 Dafna Ron 2014-05-14 14:34:42 UTC 
1. I think we should find out if we are supporting new installations using qpid or just supporting old ones? 
2. I think we should install either rabbit or qpid. so when selecting qpid we should not see anything related to rabbit. 
can you please run rpm -qa |grep rabbit; rpm -qa |grep qpid and see if the packages are installed at all? 

another option - perhaps some of the services packages have dependencies for rabbit?
Comment 11 Petr Matousek 2014-05-14 14:39:06 UTC 
Note: I have made a successful installation with qpid as the amqp back-end without neutron installed.

The timeout reported in the description is workarounded by setting notifier_strategy = noop in /etc/glance/glance-api.conf.
The we end up with the following error:
ERROR : Error appeared during Puppet run: <ip>_provision.pp
Error: Could not prefetch neutron_network provider 'neutron': Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: Connection to neutron failed: Maximum attempts reached
You will find full trace in log /var/tmp/packstack/20140514-095117-Tlljex/manifests/10.16.68.52_provision.pp.log

If the answer file is modified to not install neutron, the installation success.

Honestly, I don't have a clue what does this setup mean and if qpidd is utilized at all (no traffic seen on qpidd side). This is obviously not solution, just a note.
Comment 12 Zdenek Kraus 2014-05-14 14:46:03 UTC 
Dafna,

[root ~]# rpm -qa |grep rabbit
[root ~]# rpm -qa |grep qpid
qpid-cpp-server-0.18-22.el7.x86_64
python-qpid-qmf-0.18-20.el7.x86_64
ruby-qpid-qmf-0.18-20.el7.x86_64
qpid-qmf-0.18-20.el7.x86_64
qpid-cpp-client-ssl-0.18-22.el7.x86_64
qpid-cpp-server-cluster-0.18-22.el7.x86_64
qpid-cpp-server-rdma-0.18-22.el7.x86_64
rh-qpid-cpp-tests-0.18-22.el7.x86_64
qpid-cpp-server-store-0.18-22.el7.x86_64
qpid-cpp-server-ssl-0.18-22.el7.x86_64
qpid-tools-0.18-10.el7.noarch
qpid-cpp-server-ha-0.18-22.el7.x86_64
qpid-cpp-client-rdma-0.18-22.el7.x86_64
qpid-cpp-client-devel-0.18-22.el7.x86_64
qpid-cpp-server-devel-0.18-22.el7.x86_64
qpid-tests-0.18-2.el7.noarch
qpid-qmf-devel-0.18-20.el7.x86_64
qpid-qmf-debuginfo-0.18-20.el7.x86_64
qpid-cpp-client-0.18-22.el7.x86_64
python-qpid-0.18-10.el7.noarch
qpid-cpp-debuginfo-0.18-22.el7.x86_64

on that machine I've installed openstac with only qpid.

but I've discovered that only on those QpidDrivers the transport and notifier is created. Also I have openstack with rabbit installed succesfully and there are also only half of driver used (but obviously there are all of them rabbits).

so IMO this is not the issue.
Comment 14 Petr Matousek 2014-05-19 12:39:09 UTC 
(In reply to Petr Matousek from comment #11)
> Note: I have made a successful installation with qpid as the amqp back-end
> without neutron installed.

This is just an update to the note above, it is not true that the neutron component is not installed, summarizing:

I can perform successful installation with qpidd following way:
1. packstack --allinone --amqp-server=qpid
2. installation fails on the issue described in comment 0
3. disable neutron installation in the answer file (neutron was already installed in 1.)
4. disable notifications in /etc/glance/glance-api.conf
5. rerun packstack installation 
6. installation succeeds

The installation seems to be working, ie:
- openstack-reports all the services running (including neutron),
- user is able to create instances, use the console, etc.. via the dashboard 
- some traffic seen on qpidd side when working with the dashboard (reply queues creation, ceilometer logs)

Again, this is not a solution, just a note.
Comment 15 Petr Matousek 2014-05-20 15:30:43 UTC 
There was another workaround suggested by rhos qe folks. There shall be two prerequisites met in order to make a successful installation: 
1. selinux in permissive mode
2. symbolic link /etc/qpid/qpidd.conf -> /etc/qpidd.conf

Unfortunately, this does not help and the installation ends again on the issue reported in comment 0.
Comment 16 Lukas Bezdicka 2014-05-23 15:05:11 UTC 
For glance:
notifier_driver=qpid instead of notifier_strategy = qpid
For qpid:
drop the template case for rhos 7


so openstack-puppet-moudles need to have at least puppet-glance with patch 13b832ddf26a2ba4defcf051a0e20940df71cdcd

and we'll have to patch packstack package for this.
Comment 17 Petr Matousek 2014-05-26 12:37:55 UTC 
When the qpid.pp for glance is updated to use the notifier_driver instead of the notifier_strategy, the originally reported error is gone, but the installation fails again immediately on the next error, that can also be seen in comment 0 logs:

ERROR : Error appeared during Puppet run: <host_ip>_provision.pp
Error: Could not prefetch neutron_network provider 'neutron': Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: Connection to neutron failed: Maximum attempts reached

Note: this was seen on fresh installation.
Logs may be provided on demand.
Comment 18 Luigi Toscano 2014-05-27 16:02:35 UTC 
Just a quick note: I see this error also when using rabbitmq.

Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: <host_ip>_provision.pp
Error: Could not prefetch neutron_network provider 'neutron': Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: Connection to neutron failed: Maximum attempts reached
You will find full trace in log /var/tmp/packstack/20140527-114456-DjC6dN/manifests/<host_ip>_provision.pp.log


# rpm -qa |grep rabbit
rabbitmq-server-3.1.5-6.0.el7ost.noarch
# rpm -qa |grep qpid
python-qpid-0.18-10.el7.noarch
(I didn't install the python-qpid package)


Packstack:
[root@hp-sl4545g7-01 ~]# rpm -qa |grep rabbit
rabbitmq-server-3.1.5-6.0.el7ost.noarch
[root@hp-sl4545g7-01 ~]# rpm -qa |grep qpid
python-qpid-0.18-10.el7.noarch
Comment 19 Lukas Bezdicka 2014-05-27 17:27:24 UTC 
Created attachment 899613 [details]
amqp_qpidd_018.patch
Comment 21 Martin Magr 2014-05-28 11:06:51 UTC 
*** Bug 1098493 has been marked as a duplicate of this bug. ***
Comment 23 Ami Jeain 2014-06-15 13:45:31 UTC 
verified using the command:
#packstack --allinone --amqp-backend=qpid

# rpm -qa |grep openstack-packstack
openstack-packstack-2014.1.1-0.22.dev1117.el7ost.noarch
Comment 27 errata-xmlrpc 2014-07-08 15:38:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0846.html