Bug 1000972

Summary: Cinder doesn't close qpid connections after certain error conditions
Product: Red Hat OpenStack Reporter: Xavier Queralt <xqueralt>
Component: openstack-cinderAssignee: Xavier Queralt <xqueralt>
Status: CLOSED ERRATA QA Contact: Yogev Rabl <yrabl>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.0CC: apevec, breeler, dallan, eharney, hateya, ohochman, xqueralt, yeylon
Target Milestone: z3Keywords: ZStream
Target Release: 3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-2013.1.4-1.el6ost Doc Type: Bug Fix
Doc Text:
Previously, connections created by Cinder while trying to reconnect to the qpid server wouldn't be closed if an error was triggered during the process. As a result the service had to be restarted to close unused connections. This has been fixed so that, even on an error, qpid always closes the old connection before trying to reconnect.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-18 15:13:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xavier Queralt 2013-08-26 08:51:03 UTC
Version-Release number of selected component (if applicable):
openstack-cinder-api-2013.1.2-2.el6ost.noarch

In conditions where cinder gets a connection to qpid but there is a problem at the application layer (such as a bad qpid password is specified), cinder will continuously retry the connection without closing previous connections.

Steps to reproduce:
- Preconditions: At least 1 cinder service using qpid for messaging, with authentication turned on.
- Specify a wrong qpid password in cinder.conf on the node
- cinder will continuously retry with the wrong password and print errors such as this to the compute log:
2013-04-25 16:37:52.269 ERROR cinder.openstack.common.rpc.impl_qpid [req-1d15f33c-5b2d-4ee1-aaa1-ab0140a56608 None None] Unable to connect to AMQP server: connection-forced: Authentication failed(320). Sleeping 60 seconds

Actual results:
- Each time cinder retries the connection, it will create another connection to qpid and not close the previous connections.

Expected results:


Additional info:

Comment 1 Omri Hochman 2013-08-26 11:19:23 UTC
The issue originally opened against nova in BZ#975882.  According to comment#16 the bug reproduce when cinder services are running.

Steps to reproduce: 
-------------------
1)Change in qpidd.conf -> auth=yes 

2)Restart qpidd '/etc/init.d/qpidd restart'

3)Create qpidd user and password by running (Enter: guest/guest) : 
'saslpasswd2 -f /var/lib/qpidd/qpidd.sasldb -u QPID guest'

4)Check the created qpidd user/password by running : 
'sasldblistusers2 -f /var/lib/qpidd/qpidd.sasldb'

5)Attempt to boot instance ( should work!! ) 

6)Change in nova.conf : qpid_password=guest  --> qpid_password=badguest 

7)Restart all openstack services

8)Attempt to boot instance. ( should stuck!! )

9)While the boot command stuck, check the number of opened sessions by running: "watch -d 'netstat -n |grep 5672 | wc -l' "

The number of open sessions should not constantly increase, but the number should reduce as well, when there are connections that are being closed. 

Note: 

The number of connections will reduce when stopping cinder services.

More info: 
-----------
https://docspace.corp.redhat.com/docs/DOC-148763

Comment 6 Yogev Rabl 2013-10-31 07:03:17 UTC
verified on versions: 

openstack-nova-compute-2013.1.4-1.el6ost.noarch
openstack-dashboard-2013.1.4-1.el6ost.noarch
openstack-utils-2013.1-8.1.el6ost.noarch
openstack-glance-2013.1.4-1.el6ost.noarch
openstack-selinux-0.1.2-10.el6ost.noarch
openstack-cinder-2013.1.4-2.el6ost.noarch
openstack-nova-api-2013.1.4-1.el6ost.noarch
openstack-nova-console-2013.1.4-1.el6ost.noarch
openstack-nova-conductor-2013.1.4-1.el6ost.noarch
openstack-nova-novncproxy-0.4-6.el6ost.noarch
openstack-nova-network-2013.1.4-1.el6ost.noarch
openstack-nova-cert-2013.1.4-1.el6ost.noarch
python-django-openstack-auth-1.0.6-2.el6ost.noarch
redhat-access-plugin-openstack-1.2.0-6.el6ost.noarch
openstack-keystone-2013.1.4-1.el6ost.noarch
openstack-nova-common-2013.1.4-1.el6ost.noarch
openstack-nova-scheduler-2013.1.4-1.el6ost.noarch
openstack-packstack-2013.1.1-0.33.dev695.el6ost.noarc

Comment 8 errata-xmlrpc 2013-11-18 15:13:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1510.html