Bug 1302413 - optimize metadata service caching logic to avoid unnecessary data retrieval from conductor
optimize metadata service caching logic to avoid unnecessary data retrieval f...
Status: CLOSED EOL
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
5.0 (RHEL 7)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 5.0 (RHEL 7)
Assigned To: Eoghan Glynn
nlevinki
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-27 13:58 EST by Eoghan Glynn
Modified: 2017-06-30 09:31 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-30 09:31:03 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1540526 None None None 2016-02-25 11:25 EST
Launchpad 1549814 None None None 2016-02-25 11:00 EST
OpenStack gerrit 158066 None None None 2016-02-25 11:30 EST
OpenStack gerrit 276861 None None None 2016-02-25 11:28 EST

  None (edit)
Description Eoghan Glynn 2016-01-27 13:58:05 EST
Currently the metadata service retrieves data unneccessarily from the conductor in at least two separate cases:

1. where the same data was already retrieved and cached by another metadata service worker running on the same node

2. when a retrieval for that same data is already in flight

This unneccessary load on the conductor could be reduced by using a shared cache across all workers running on the same node, and recording when a retrieval for some URL is already in progress so that if that same data is requested again before the initial fetch has completed, then the subsequent request can simply await it's arrival in the shared cached as opposed to independently re-fetching it in parallel.
Comment 2 Sven Anderson 2016-02-25 11:19:54 EST
The metadata caching is flawed in general, but there is a symptomatic fix for this related performance issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1244852

upstream, which pre-fetches some of data before caching, here:

https://github.com/openstack/nova/commit/3a761270581d1ac61a3b4669c130d211f1ad5a17#diff-969229657f01b56c336e01497df732d7R1226

and here:

https://github.com/openstack/nova/commit/cc41015d463e11ac11bbaaac0b5c441329dc5f0b#diff-567f52edc17aff6c473d69c341a4cb0cR513


Unfortunately the second change introduced the pre-fetch as a side-effect, so it cannot be backported as is.
Comment 3 Sven Anderson 2016-02-29 06:23:37 EST
I have submitted two upstream changes that are related to this.

Disabling memached for metadata caching:
https://review.openstack.org/#/c/285530

No parallel queries of the same data (this addresses point 2 in the description):
https://review.openstack.org/#/c/285562
Comment 5 Sven Anderson 2016-03-04 06:27:05 EST
Clarification: the upstream changes are stop-gap fixes (and might not get accepted). The main problem of caching the db queries instead of a whole python object, in order to make it shareable between different processes, is still unaddressed.
Comment 6 Stephen Gordon 2016-06-16 13:27:10 EDT
Sven can you help me understand what if anything remains that is backportable here versus needing to be addressed in a later release (Newton/Ocata)?
Comment 7 Sven Anderson 2017-01-03 11:33:11 EST
Stephen, Diana was working on that after me. I have no idea about the current status.
Comment 8 Scott Lewis 2017-06-30 09:31:03 EDT
Red Hat OpenStack Platform version 5 is now End-of-Life, and as such will not have further updates. See https://access.redhat.com/support/policy/updates/openstack/platform/ for full support lifecycle details.

Note You need to log in before you can comment on or make changes to this bug.