Red Hat Bugzilla – Bug 1307158
When nova APIs are used to execute glance functions (image details, list, etc), glance connections are not reused or closed.
Last modified: 2016-06-14 06:48:27 EDT
'no more files' errors are produced by nova and/or glance if enough 'nova image-list' commands are run in a row. There are automated processes (ceilometer) that are running image-related nova commands at a rate of about 1-3 every second. This is causing nova to fail within a few hours. Restarting either nova-api or glance-api clears the connections; however, if nova fails, then nova-api will *need* be to restarted. This was resolved by applying these upstream commits directly: https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=f98ab688eff8fff4bdb5f650da3516715d62f232 https://git.openstack.org/cgit/openstack/python-glanceclient/commit/?id=ef9fd9fca05f8da8325ccaa6632e34d1321130bf Relevant upstream bugs: https://bugs.launchpad.net/python-glanceclient/+bug/1423939 https://bugs.launchpad.net/python-glanceclient/+bug/1423165
Cyril, Could you help with this one?
OK, so, both these patches seem to be in stable/liberty. They appear to be in stable/kilo as well. We will need to backport them to stable/juno, though.
Update from the customer: Call today with Stephen Pierce conversation The customer wanted to inform the bug engineers that they are completely satisfied with the patches they have implemented in their environment. The question moving forward in this bug would be what are we (RedHat) going to do to provide this as a supported configuration? 1. Offer an exception 2. backport the patches into an OSP update The customer has stated they would like to get an official package from RedHat for these changes that way there is no question on supportability. Can someone on here provide an action plan for the best proposed outcome? Thanks. Taylor Clevenger Platform TAM
So I wanted to backport this patch to Juno : https://review.openstack.org/#/c/157516/5 , but not only does the "glanceclient/v1/images.py" file exist in rhos-6.0-patches, nor does the "paginate" function that this patch fixes. Flavio, do you know whether this feature used to be somewhere else in the code? As for https://review.openstack.org/#/c/156975/1 , I have a similar issue : the code is so different in Juno that I even wonder whether the patched feature existed back then. Flavio, WDYT?
If the feature doesn't exist and the backport is too invasive, I'd recommend not doing it (as you also hinted). The key thing is to make sure the backport won't break the current release. It is possible the feature doesn't exist in early versions of glanceclient.
Please discard my earlier comment :)
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Flavio tells me there already is a package that includes the fix for this issue, but I can't find it. Flavio?
Cyril, It's in the Fixed In field. If that's not the right package, we should clear the field out, set this bug back to assigned and act accordingly.
Yes, did anyone try with version python-glanceclient-0.14.2-5.el7ost ? If this happens on other components, I think new bugs should be opened against these components.
Yes and I can confirm it solves our issue for glance.
According to comment 22 and 23 the patch was verified by the customer. python-glanceclient-0.14.2-5.el7ost is now installed as part of rhos6 puddle The automation tests succsefully passed refer to https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS6/job/rhos-jenkins-rhos-6.0-puddle-rhel-7.2-all-in-one-packstack-nova-flatdhcp-qpidd-tempest-git-all/lastCompletedBuild/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1213