Bug 1000972 - Cinder doesn't close qpid connections after certain error conditions
Summary: Cinder doesn't close qpid connections after certain error conditions
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z3
: 3.0
Assignee: Xavier Queralt
QA Contact: Yogev Rabl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-26 08:51 UTC by Xavier Queralt
Modified: 2016-04-26 14:31 UTC (History)
8 users (show)

Fixed In Version: openstack-cinder-2013.1.4-1.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, connections created by Cinder while trying to reconnect to the qpid server wouldn't be closed if an error was triggered during the process. As a result the service had to be restarted to close unused connections. This has been fixed so that, even on an error, qpid always closes the old connection before trying to reconnect.
Clone Of:
Environment:
Last Closed: 2013-11-18 15:13:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1172922 0 None None None Never
OpenStack gerrit 43713 0 None None None Never
Red Hat Product Errata RHBA-2013:1510 0 normal SHIPPED_LIVE Red Hat OpenStack 3.0 bug fix and enhancement advisory 2013-11-18 20:11:18 UTC

Description Xavier Queralt 2013-08-26 08:51:03 UTC
Version-Release number of selected component (if applicable):
openstack-cinder-api-2013.1.2-2.el6ost.noarch

In conditions where cinder gets a connection to qpid but there is a problem at the application layer (such as a bad qpid password is specified), cinder will continuously retry the connection without closing previous connections.

Steps to reproduce:
- Preconditions: At least 1 cinder service using qpid for messaging, with authentication turned on.
- Specify a wrong qpid password in cinder.conf on the node
- cinder will continuously retry with the wrong password and print errors such as this to the compute log:
2013-04-25 16:37:52.269 ERROR cinder.openstack.common.rpc.impl_qpid [req-1d15f33c-5b2d-4ee1-aaa1-ab0140a56608 None None] Unable to connect to AMQP server: connection-forced: Authentication failed(320). Sleeping 60 seconds

Actual results:
- Each time cinder retries the connection, it will create another connection to qpid and not close the previous connections.

Expected results:


Additional info:

Comment 1 Omri Hochman 2013-08-26 11:19:23 UTC
The issue originally opened against nova in BZ#975882.  According to comment#16 the bug reproduce when cinder services are running.

Steps to reproduce: 
-------------------
1)Change in qpidd.conf -> auth=yes 

2)Restart qpidd '/etc/init.d/qpidd restart'

3)Create qpidd user and password by running (Enter: guest/guest) : 
'saslpasswd2 -f /var/lib/qpidd/qpidd.sasldb -u QPID guest'

4)Check the created qpidd user/password by running : 
'sasldblistusers2 -f /var/lib/qpidd/qpidd.sasldb'

5)Attempt to boot instance ( should work!! ) 

6)Change in nova.conf : qpid_password=guest  --> qpid_password=badguest 

7)Restart all openstack services

8)Attempt to boot instance. ( should stuck!! )

9)While the boot command stuck, check the number of opened sessions by running: "watch -d 'netstat -n |grep 5672 | wc -l' "

The number of open sessions should not constantly increase, but the number should reduce as well, when there are connections that are being closed. 

Note: 

The number of connections will reduce when stopping cinder services.

More info: 
-----------
https://docspace.corp.redhat.com/docs/DOC-148763

Comment 6 Yogev Rabl 2013-10-31 07:03:17 UTC
verified on versions: 

openstack-nova-compute-2013.1.4-1.el6ost.noarch
openstack-dashboard-2013.1.4-1.el6ost.noarch
openstack-utils-2013.1-8.1.el6ost.noarch
openstack-glance-2013.1.4-1.el6ost.noarch
openstack-selinux-0.1.2-10.el6ost.noarch
openstack-cinder-2013.1.4-2.el6ost.noarch
openstack-nova-api-2013.1.4-1.el6ost.noarch
openstack-nova-console-2013.1.4-1.el6ost.noarch
openstack-nova-conductor-2013.1.4-1.el6ost.noarch
openstack-nova-novncproxy-0.4-6.el6ost.noarch
openstack-nova-network-2013.1.4-1.el6ost.noarch
openstack-nova-cert-2013.1.4-1.el6ost.noarch
python-django-openstack-auth-1.0.6-2.el6ost.noarch
redhat-access-plugin-openstack-1.2.0-6.el6ost.noarch
openstack-keystone-2013.1.4-1.el6ost.noarch
openstack-nova-common-2013.1.4-1.el6ost.noarch
openstack-nova-scheduler-2013.1.4-1.el6ost.noarch
openstack-packstack-2013.1.1-0.33.dev695.el6ost.noarc

Comment 8 errata-xmlrpc 2013-11-18 15:13:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1510.html


Note You need to log in before you can comment on or make changes to this bug.