Bug 975882 - Nova doesn't close qpid connections after certain error conditions
Nova doesn't close qpid connections after certain error conditions
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
3.0
Unspecified Unspecified
medium Severity medium
: z2
: 3.0
Assigned To: Xavier Queralt
yfried
: Rebase, Reopened, ZStream
Depends On: 993100
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-19 10:14 EDT by yfried
Modified: 2016-04-26 11:39 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Rebase: Bug Fixes Only
Doc Text:
Rebase package(s) to version: 2013.1.3 Highlights and important bug fixes: https://launchpad.net/nova/+milestone/2013.1.3
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-03 16:19:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1172922 None None None Never
OpenStack gerrit 33476 None None None Never

  None (edit)
Description yfried 2013-06-19 10:14:07 EDT
Version-Release number of selected component (if applicable):
openstack-nova-compute-2013.1.2-2.el6ost.noarch


In conditions where nova gets a connection to qpid but there is a problem at the application layer (such as a bad qpid password is specified), nova will continuously retry the connection without closing previous connections.

Steps to reproduce:
- Preconditions: At least 1 nova compute node using qpid for messaging, with authentication turned on.
- Specify a wrong qpid password in nova.conf on the compute node
- nova will continuously retry with the wrong password and print errors such as this to the compute log:
2013-04-25 16:37:52.269 ERROR nova.openstack.common.rpc.impl_qpid [req-1d15f33c-5b2d-4ee1-aaa1-ab0140a56608 None None] Unable to connect to AMQP server: connection-forced: Authentication failed(320). Sleeping 60 seconds

Actual results:
- Each time nova retries the connection, it will create another connection to qpid and not close the previous connections.

Expected results:


Additional info:
Comment 1 yfried 2013-06-20 02:30:59 EDT
link to same quantum bug (closed)
https://bugzilla.redhat.com/show_bug.cgi?id=962385#c9
Comment 2 Perry Myers 2013-06-20 08:52:34 EDT
Since this only occurs when an incorrect user/password is used, it implies that the issue only occurs before you've got a functional cloud infrastructure (can't have a working cloud w/o having qpid connectivity, etc)

So, I think this is worth release noting and fixing in the next release.
Comment 3 Alan Pevec 2013-06-20 12:04:28 EDT
(In reply to Perry Myers from comment #2)
> Since this only occurs when an incorrect user/password is used

Reconnect situation can happen any time there's an issue with connection during operation e.g. temporary network outage.

https://github.com/openstack/nova/blob/2013.1.2/nova/openstack/common/rpc/impl_qpid.py#L372
Comment 5 Alan Pevec 2013-07-09 06:31:03 EDT
Fixed in Nova havana-1 https://github.com/openstack/nova/commit/b4826d85c25a56ad95ffb76c467cdb459daba0c4
Comment 6 Alan Pevec 2013-07-09 07:33:19 EDT
Will be included in stable/grizzly 2013.1.3 release.
Comment 11 Omri Hochman 2013-08-19 09:53:00 EDT
Steps to reproduce: 
-------------------
1)Change in qpidd.conf -> auth=yes 

2)Restart qpidd '/etc/init.d/qpidd restart'

3)Create qpidd user and password by running (Enter: guest/guest) : 
'saslpasswd2 -f /var/lib/qpidd/qpidd.sasldb -u QPID guest'

4)Check the created qpidd user/password by running : 
'sasldblistusers2 -f /var/lib/qpidd/qpidd.sasldb'

5)Attempt to boot instance ( should work!! ) 

6)Change ion nova.conf : qpid_password=guest  --> qpid_password=badguest 

7)Attempt to boot instance. ( should stuck!! )

8)While the boot command stuck, check the number of opened sessions by running: "watch -d 'netstat -n |grep 5672 | wc -l' "

The number of open sessions should not constantly increase, but the number should reduce as well, when there are connections that are being closed. 

Note:
-------
When the bug reproduces - 'netstat -n |grep 5672' will show increase number of open sessions, that are being stopped only when stopping the nova command (ie - "boot").

Some sessions will only be closed when password is being restored.


More info: 
-----------
https://docspace.corp.redhat.com/docs/DOC-148763
Comment 16 Omri Hochman 2013-08-26 04:54:11 EDT
This issue is not reproducible when stopping cinder service - the number of connections reduces and it work as it should. 

According that info - this nova bug will be closed and copied to cinder Bz#1000972.
Comment 17 Omri Hochman 2013-08-26 07:24:55 EDT
Verified - with openstack-nova-2013.1.3-2.
> This issue is not reproducible when stopping cinder service - the number of
> connections reduces and it work as it should. 
> 
> According that info - this nova bug will be closed and copied to cinder
> Bz#1000972.
Comment 19 errata-xmlrpc 2013-09-03 16:19:13 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-1199.html

Note You need to log in before you can comment on or make changes to this bug.