Bug 1307158

Summary: When nova APIs are used to execute glance functions (image details, list, etc), glance connections are not reused or closed.
Product: Red Hat OpenStack Reporter: nalmond
Component: python-glanceclientAssignee: Cyril Roelandt <cyril>
Status: CLOSED ERRATA QA Contact: Avi Avraham <aavraham>
Severity: medium Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: cyril, dhill, dmaley, ealcaniz, eglynn, fpercoco, jruzicka, mschuppe, mtessun, sgotliv, srevivo, tcleveng
Target Milestone: asyncKeywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: python-glanceclient-0.14.2-5.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-08 12:11:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description nalmond 2016-02-12 21:08:26 UTC
'no more files' errors are produced by nova and/or glance if enough 'nova image-list' commands are run in a row. There are automated processes (ceilometer) that are running image-related nova commands at a rate of about 1-3 every second. This is causing nova to fail within a few hours. Restarting either nova-api or glance-api clears the connections; however, if nova fails, then nova-api will *need* be to restarted.

This was resolved by applying these upstream commits directly:

https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=f98ab688eff8fff4bdb5f650da3516715d62f232

https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=ef9fd9fca05f8da8325ccaa6632e34d1321130bf

Relevant upstream bugs:
https://bugs.launchpad.net/python-glanceclient/+bug/1423939
https://bugs.launchpad.net/python-glanceclient/+bug/1423165

Comment 2 Flavio Percoco 2016-02-15 16:29:04 UTC
Cyril,

Could you help with this one?

Comment 3 Cyril Roelandt 2016-02-18 16:21:21 UTC
OK, so, both these patches seem to be in stable/liberty. They appear to be in stable/kilo as well. We will need to backport them to stable/juno, though.

Comment 4 tcleveng 2016-02-18 16:41:54 UTC
Update from the customer:

Call today with Stephen Pierce conversation

The customer wanted to inform the bug engineers that they are completely satisfied with the patches they have implemented in their environment. The question moving forward in this bug would be what are we (RedHat) going to do to provide this as a supported configuration?

1. Offer an exception
2. backport the patches into an OSP update

The customer has stated they would like to get an official package from RedHat for these changes that way there is no question on supportability.

Can someone on here provide an action plan for the best proposed outcome?

Thanks.
Taylor Clevenger
Platform TAM

Comment 5 Cyril Roelandt 2016-02-19 13:53:34 UTC
So I wanted to backport this patch to Juno : https://review.openstack.org/#/c/157516/5 , but not only does the "glanceclient/v1/images.py" file exist in rhos-6.0-patches, nor does the "paginate" function that this patch fixes. Flavio, do you know whether this feature used to be somewhere else in the code?

As for https://review.openstack.org/#/c/156975/1 , I have a similar issue : the code is so different in Juno that I even wonder whether the patched feature existed back then. Flavio, WDYT?

Comment 6 Flavio Percoco 2016-02-19 18:36:27 UTC
If the feature doesn't exist and the backport is too invasive, I'd recommend not doing it (as you also hinted). The key thing is to make sure the backport won't break the current release.

It is possible the feature doesn't exist in early versions of glanceclient.

Comment 9 Cyril Roelandt 2016-03-17 14:19:19 UTC
Please discard my earlier comment :)

Comment 10 Mike McCune 2016-03-28 22:26:18 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 17 Cyril Roelandt 2016-05-09 16:58:14 UTC
Flavio tells me there already is a package that includes the fix for this issue, but I can't find it. Flavio?

Comment 18 Flavio Percoco 2016-05-09 17:53:15 UTC
Cyril, It's in the Fixed In field. If that's not the right package, we should clear the field out, set this bug back to assigned and act accordingly.

Comment 21 Cyril Roelandt 2016-05-12 17:34:34 UTC
Yes, did anyone try with version python-glanceclient-0.14.2-5.el7ost ?

If this happens on other components, I think new bugs should be opened against these components.

Comment 22 David Hill 2016-05-17 14:50:55 UTC
Yes and I can confirm it solves our issue for glance.

Comment 25 Avi Avraham 2016-06-01 08:19:25 UTC
According to comment 22 and 23 the patch was verified by the customer.
python-glanceclient-0.14.2-5.el7ost is now installed as part of rhos6 puddle 
The automation tests succsefully passed refer to 
https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS6/job/rhos-jenkins-rhos-6.0-puddle-rhel-7.2-all-in-one-packstack-nova-flatdhcp-qpidd-tempest-git-all/lastCompletedBuild/

Comment 27 errata-xmlrpc 2016-06-08 12:11:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1213