Bug 1103800

Summary: rhos5 is unable to create instances
Product: Red Hat OpenStack Reporter: Petr Matousek <pematous>
Component: python-oslo-messagingAssignee: Pádraig Brady <pbrady>
Status: CLOSED ERRATA QA Contact: Petr Matousek <pematous>
Severity: high Docs Contact:
Priority: high    
Version: 5.0 (RHEL 7)CC: apevec, esammons, fpercoco, iboverma, jross, kgiusti, lhh, lzhaldyb, nyechiel, tross, tshefi, zkraus
Target Milestone: rcKeywords: TestBlocker
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-oslo-messaging-1.3.0.2-4 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1106851 1106854 1106858 1106863 1106893 (view as bug list) Environment:
Last Closed: 2014-07-08 15:13:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1106851, 1106854, 1106858, 1106863, 1106893    

Description Petr Matousek 2014-06-02 15:12:37 UTC
Description of problem:

Error appears when creating an instance: NoValidHost

Note: the scheduler gets empty list of 'weighed_hosts' in filter_scheduler.py, List of hosts shall be returned, ie: [WeighedHost [host: bender.idm.lab.bos.redhat.com, weight: 1.0]]

This issue does _NOT_ appear when using rabbitmq as a messaging service.
This issue does _NOT_ appear when using python-qpid-0.24-1.

Note: no errors seen in qpidd log

Version-Release number of selected component (if applicable):
python-oslo-messaging-1.3.0.2-2
python-qpid-0.18-12
openstack-packstack-2014.1.1-0.16.dev1100

How reproducible:
100%

Steps to Reproduce:
1. yum -y install http://team.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm
2. rhos-release 5
3. workaround 1103695 (change nagios-plugins-ping -> monitoring-plugins-ping)
4. packstack --allinone --amqp-backend=qpid
5. login to rhos5 Dashboard
6. Project -> Instances -> Launch instance
7. fill the requested fields and press 'Launch'
8. Error by creating the instance: No valid host was found
File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 116, in schedule_run_instance raise exception.NoValidHost(reason="") 

Actual results:
Unable to create rhos5 instance
Unable to get list of hosts.

Expected results:
Propper list of hosts is returned.
rhos5 instance may be successfully created.

Additional info:

Comment 7 Petr Matousek 2014-06-04 14:53:02 UTC
Note: this issue most probably causes also 'nova x509-create-cert' timeout.

# . keystonerc_admin
# nova x509-create-cert /root/accrc/admin/admin-pk.pem /root/accrc/admin/admin-cert.pem
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-cfe83282-61ae-432a-9b19-cb07c62310a0)

or
(ERROR: HTTPConnectionPool(host='<host_ip>', port=8774): Request timed out. (timeout=600.0)) - seen in CI jobs.

This shall be retested as well while verifying this bug.

Expected output:
# nova x509-create-cert /root/accrc/admin/admin-pk.pem /root/accrc/admin/admin-cert.pem
Wrote private key to /root/accrc/admin/admin-pk.pem
Wrote x509 certificate to /root/accrc/admin/admin-cert.pem

Comment 8 Russell Bryant 2014-06-09 14:30:43 UTC
*** Bug 1106374 has been marked as a duplicate of this bug. ***

Comment 9 Russell Bryant 2014-06-09 17:47:38 UTC
In addition to the oslo.messaging library, this fix needs to make it into the stable/icehouse branch of all projects that use impl_qpid from oslo-incubator.  The projects affected are: Neutron, Heat, Ceilometer, Trove.

I'll be cloning this bug to those projects to make sure the fix gets there if it hasn't already.

Comment 10 Russell Bryant 2014-06-09 20:42:56 UTC
This is the stable/icehouse patch for oslo.messaging that needs to be backported for this bug: https://review.openstack.org/98882

Comment 12 Petr Matousek 2014-06-17 06:33:21 UTC
Note: This issue is fixed. RHEL-OSP5 instances may be created without any issues.

packages under test:
python-oslo-messaging-1.3.0.2-4.el7ost
openstack-neutron-2014.1-30.el7ost
openstack-heat-2014.1-1.5.el7ost
openstack-ceilometer-2014.1-2.3.el7ost 
openstack-puppet-modules-2014.1-14.1.el7ost
openstack-packstack-puppet-2014.1.1-0.22.dev1117.el7ost
openstack-packstack-2014.1.1-0.22.dev1117.el7ost
qpid-cpp-server-0.18-24
python-qpid-0.18-12.el7

Comment 13 Petr Matousek 2014-06-20 13:20:50 UTC
Issue fixed, packages available in latest puddle (2014-06-19.2). Comment 7 verified as well.

packages under test:
python-oslo-messaging-1.3.0.2-4.el7ost

-> VERIFIED

Comment 15 errata-xmlrpc 2014-07-08 15:13:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0845.html