Bug 1307158 - When nova APIs are used to execute glance functions (image details, list, etc), glance connections are not reused or closed.
When nova APIs are used to execute glance functions (image details, list, etc...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-glanceclient (Show other bugs)
6.0 (Juno)
Unspecified Linux
high Severity medium
: async
: 6.0 (Juno)
Assigned To: Cyril Roelandt
Avi Avraham
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-12 16:08 EST by nalmond
Modified: 2016-06-14 06:48 EDT (History)
12 users (show)

See Also:
Fixed In Version: python-glanceclient-0.14.2-5.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-08 08:11:01 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nalmond 2016-02-12 16:08:26 EST
'no more files' errors are produced by nova and/or glance if enough 'nova image-list' commands are run in a row. There are automated processes (ceilometer) that are running image-related nova commands at a rate of about 1-3 every second. This is causing nova to fail within a few hours. Restarting either nova-api or glance-api clears the connections; however, if nova fails, then nova-api will *need* be to restarted.

This was resolved by applying these upstream commits directly:

https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=f98ab688eff8fff4bdb5f650da3516715d62f232

https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=ef9fd9fca05f8da8325ccaa6632e34d1321130bf

Relevant upstream bugs:
https://bugs.launchpad.net/python-glanceclient/+bug/1423939
https://bugs.launchpad.net/python-glanceclient/+bug/1423165
Comment 2 Flavio Percoco 2016-02-15 11:29:04 EST
Cyril,

Could you help with this one?
Comment 3 Cyril Roelandt 2016-02-18 11:21:21 EST
OK, so, both these patches seem to be in stable/liberty. They appear to be in stable/kilo as well. We will need to backport them to stable/juno, though.
Comment 4 tcleveng 2016-02-18 11:41:54 EST
Update from the customer:

Call today with Stephen Pierce conversation

The customer wanted to inform the bug engineers that they are completely satisfied with the patches they have implemented in their environment. The question moving forward in this bug would be what are we (RedHat) going to do to provide this as a supported configuration?

1. Offer an exception
2. backport the patches into an OSP update

The customer has stated they would like to get an official package from RedHat for these changes that way there is no question on supportability.

Can someone on here provide an action plan for the best proposed outcome?

Thanks.
Taylor Clevenger
Platform TAM
Comment 5 Cyril Roelandt 2016-02-19 08:53:34 EST
So I wanted to backport this patch to Juno : https://review.openstack.org/#/c/157516/5 , but not only does the "glanceclient/v1/images.py" file exist in rhos-6.0-patches, nor does the "paginate" function that this patch fixes. Flavio, do you know whether this feature used to be somewhere else in the code?

As for https://review.openstack.org/#/c/156975/1 , I have a similar issue : the code is so different in Juno that I even wonder whether the patched feature existed back then. Flavio, WDYT?
Comment 6 Flavio Percoco 2016-02-19 13:36:27 EST
If the feature doesn't exist and the backport is too invasive, I'd recommend not doing it (as you also hinted). The key thing is to make sure the backport won't break the current release.

It is possible the feature doesn't exist in early versions of glanceclient.
Comment 9 Cyril Roelandt 2016-03-17 10:19:19 EDT
Please discard my earlier comment :)
Comment 10 Mike McCune 2016-03-28 18:26:18 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 17 Cyril Roelandt 2016-05-09 12:58:14 EDT
Flavio tells me there already is a package that includes the fix for this issue, but I can't find it. Flavio?
Comment 18 Flavio Percoco 2016-05-09 13:53:15 EDT
Cyril, It's in the Fixed In field. If that's not the right package, we should clear the field out, set this bug back to assigned and act accordingly.
Comment 21 Cyril Roelandt 2016-05-12 13:34:34 EDT
Yes, did anyone try with version python-glanceclient-0.14.2-5.el7ost ?

If this happens on other components, I think new bugs should be opened against these components.
Comment 22 David Hill 2016-05-17 10:50:55 EDT
Yes and I can confirm it solves our issue for glance.
Comment 25 Avi Avraham 2016-06-01 04:19:25 EDT
According to comment 22 and 23 the patch was verified by the customer.
python-glanceclient-0.14.2-5.el7ost is now installed as part of rhos6 puddle 
The automation tests succsefully passed refer to 
https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS6/job/rhos-jenkins-rhos-6.0-puddle-rhel-7.2-all-in-one-packstack-nova-flatdhcp-qpidd-tempest-git-all/lastCompletedBuild/
Comment 27 errata-xmlrpc 2016-06-08 08:11:01 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1213

Note You need to log in before you can comment on or make changes to this bug.