Red Hat Bugzilla – Bug 1000972
Cinder doesn't close qpid connections after certain error conditions
Last modified: 2016-04-26 10:31:29 EDT
Version-Release number of selected component (if applicable): openstack-cinder-api-2013.1.2-2.el6ost.noarch In conditions where cinder gets a connection to qpid but there is a problem at the application layer (such as a bad qpid password is specified), cinder will continuously retry the connection without closing previous connections. Steps to reproduce: - Preconditions: At least 1 cinder service using qpid for messaging, with authentication turned on. - Specify a wrong qpid password in cinder.conf on the node - cinder will continuously retry with the wrong password and print errors such as this to the compute log: 2013-04-25 16:37:52.269 ERROR cinder.openstack.common.rpc.impl_qpid [req-1d15f33c-5b2d-4ee1-aaa1-ab0140a56608 None None] Unable to connect to AMQP server: connection-forced: Authentication failed(320). Sleeping 60 seconds Actual results: - Each time cinder retries the connection, it will create another connection to qpid and not close the previous connections. Expected results: Additional info:
The issue originally opened against nova in BZ#975882. According to comment#16 the bug reproduce when cinder services are running. Steps to reproduce: ------------------- 1)Change in qpidd.conf -> auth=yes 2)Restart qpidd '/etc/init.d/qpidd restart' 3)Create qpidd user and password by running (Enter: guest/guest) : 'saslpasswd2 -f /var/lib/qpidd/qpidd.sasldb -u QPID guest' 4)Check the created qpidd user/password by running : 'sasldblistusers2 -f /var/lib/qpidd/qpidd.sasldb' 5)Attempt to boot instance ( should work!! ) 6)Change in nova.conf : qpid_password=guest --> qpid_password=badguest 7)Restart all openstack services 8)Attempt to boot instance. ( should stuck!! ) 9)While the boot command stuck, check the number of opened sessions by running: "watch -d 'netstat -n |grep 5672 | wc -l' " The number of open sessions should not constantly increase, but the number should reduce as well, when there are connections that are being closed. Note: The number of connections will reduce when stopping cinder services. More info: ----------- https://docspace.corp.redhat.com/docs/DOC-148763
verified on versions: openstack-nova-compute-2013.1.4-1.el6ost.noarch openstack-dashboard-2013.1.4-1.el6ost.noarch openstack-utils-2013.1-8.1.el6ost.noarch openstack-glance-2013.1.4-1.el6ost.noarch openstack-selinux-0.1.2-10.el6ost.noarch openstack-cinder-2013.1.4-2.el6ost.noarch openstack-nova-api-2013.1.4-1.el6ost.noarch openstack-nova-console-2013.1.4-1.el6ost.noarch openstack-nova-conductor-2013.1.4-1.el6ost.noarch openstack-nova-novncproxy-0.4-6.el6ost.noarch openstack-nova-network-2013.1.4-1.el6ost.noarch openstack-nova-cert-2013.1.4-1.el6ost.noarch python-django-openstack-auth-1.0.6-2.el6ost.noarch redhat-access-plugin-openstack-1.2.0-6.el6ost.noarch openstack-keystone-2013.1.4-1.el6ost.noarch openstack-nova-common-2013.1.4-1.el6ost.noarch openstack-nova-scheduler-2013.1.4-1.el6ost.noarch openstack-packstack-2013.1.1-0.33.dev695.el6ost.noarc
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1510.html