Bug 1307158 - When nova APIs are used to execute glance functions (image details, list, etc), glance connections are not reused or closed.
Summary: When nova APIs are used to execute glance functions (image details, list, etc...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-glanceclient
Version: 6.0 (Juno)
Hardware: Unspecified
OS: Linux
high
medium
Target Milestone: async
: 6.0 (Juno)
Assignee: Cyril Roelandt
QA Contact: Avi Avraham
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-02-12 21:08 UTC by nalmond
Modified: 2019-11-14 07:27 UTC (History)
12 users (show)

Fixed In Version: python-glanceclient-0.14.2-5.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-08 12:11:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1213 0 normal SHIPPED_LIVE openstack-glance bug fix advisory 2016-06-08 16:09:18 UTC

Description nalmond 2016-02-12 21:08:26 UTC
'no more files' errors are produced by nova and/or glance if enough 'nova image-list' commands are run in a row. There are automated processes (ceilometer) that are running image-related nova commands at a rate of about 1-3 every second. This is causing nova to fail within a few hours. Restarting either nova-api or glance-api clears the connections; however, if nova fails, then nova-api will *need* be to restarted.

This was resolved by applying these upstream commits directly:

https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=f98ab688eff8fff4bdb5f650da3516715d62f232

https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=ef9fd9fca05f8da8325ccaa6632e34d1321130bf

Relevant upstream bugs:
https://bugs.launchpad.net/python-glanceclient/+bug/1423939
https://bugs.launchpad.net/python-glanceclient/+bug/1423165

Comment 2 Flavio Percoco 2016-02-15 16:29:04 UTC
Cyril,

Could you help with this one?

Comment 3 Cyril Roelandt 2016-02-18 16:21:21 UTC
OK, so, both these patches seem to be in stable/liberty. They appear to be in stable/kilo as well. We will need to backport them to stable/juno, though.

Comment 4 tcleveng 2016-02-18 16:41:54 UTC
Update from the customer:

Call today with Stephen Pierce conversation

The customer wanted to inform the bug engineers that they are completely satisfied with the patches they have implemented in their environment. The question moving forward in this bug would be what are we (RedHat) going to do to provide this as a supported configuration?

1. Offer an exception
2. backport the patches into an OSP update

The customer has stated they would like to get an official package from RedHat for these changes that way there is no question on supportability.

Can someone on here provide an action plan for the best proposed outcome?

Thanks.
Taylor Clevenger
Platform TAM

Comment 5 Cyril Roelandt 2016-02-19 13:53:34 UTC
So I wanted to backport this patch to Juno : https://review.openstack.org/#/c/157516/5 , but not only does the "glanceclient/v1/images.py" file exist in rhos-6.0-patches, nor does the "paginate" function that this patch fixes. Flavio, do you know whether this feature used to be somewhere else in the code?

As for https://review.openstack.org/#/c/156975/1 , I have a similar issue : the code is so different in Juno that I even wonder whether the patched feature existed back then. Flavio, WDYT?

Comment 6 Flavio Percoco 2016-02-19 18:36:27 UTC
If the feature doesn't exist and the backport is too invasive, I'd recommend not doing it (as you also hinted). The key thing is to make sure the backport won't break the current release.

It is possible the feature doesn't exist in early versions of glanceclient.

Comment 9 Cyril Roelandt 2016-03-17 14:19:19 UTC
Please discard my earlier comment :)

Comment 10 Mike McCune 2016-03-28 22:26:18 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 17 Cyril Roelandt 2016-05-09 16:58:14 UTC
Flavio tells me there already is a package that includes the fix for this issue, but I can't find it. Flavio?

Comment 18 Flavio Percoco 2016-05-09 17:53:15 UTC
Cyril, It's in the Fixed In field. If that's not the right package, we should clear the field out, set this bug back to assigned and act accordingly.

Comment 21 Cyril Roelandt 2016-05-12 17:34:34 UTC
Yes, did anyone try with version python-glanceclient-0.14.2-5.el7ost ?

If this happens on other components, I think new bugs should be opened against these components.

Comment 22 David Hill 2016-05-17 14:50:55 UTC
Yes and I can confirm it solves our issue for glance.

Comment 25 Avi Avraham 2016-06-01 08:19:25 UTC
According to comment 22 and 23 the patch was verified by the customer.
python-glanceclient-0.14.2-5.el7ost is now installed as part of rhos6 puddle 
The automation tests succsefully passed refer to 
https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS6/job/rhos-jenkins-rhos-6.0-puddle-rhel-7.2-all-in-one-packstack-nova-flatdhcp-qpidd-tempest-git-all/lastCompletedBuild/

Comment 27 errata-xmlrpc 2016-06-08 12:11:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1213


Note You need to log in before you can comment on or make changes to this bug.