Currently the metadata service retrieves data unneccessarily from the conductor in at least two separate cases: 1. where the same data was already retrieved and cached by another metadata service worker running on the same node 2. when a retrieval for that same data is already in flight This unneccessary load on the conductor could be reduced by using a shared cache across all workers running on the same node, and recording when a retrieval for some URL is already in progress so that if that same data is requested again before the initial fetch has completed, then the subsequent request can simply await it's arrival in the shared cached as opposed to independently re-fetching it in parallel.
The metadata caching is flawed in general, but there is a symptomatic fix for this related performance issue: https://bugzilla.redhat.com/show_bug.cgi?id=1244852 upstream, which pre-fetches some of data before caching, here: https://github.com/openstack/nova/commit/3a761270581d1ac61a3b4669c130d211f1ad5a17#diff-969229657f01b56c336e01497df732d7R1226 and here: https://github.com/openstack/nova/commit/cc41015d463e11ac11bbaaac0b5c441329dc5f0b#diff-567f52edc17aff6c473d69c341a4cb0cR513 Unfortunately the second change introduced the pre-fetch as a side-effect, so it cannot be backported as is.
I have submitted two upstream changes that are related to this. Disabling memached for metadata caching: https://review.openstack.org/#/c/285530 No parallel queries of the same data (this addresses point 2 in the description): https://review.openstack.org/#/c/285562
Clarification: the upstream changes are stop-gap fixes (and might not get accepted). The main problem of caching the db queries instead of a whole python object, in order to make it shareable between different processes, is still unaddressed.
Sven can you help me understand what if anything remains that is backportable here versus needing to be addressed in a later release (Newton/Ocata)?
Stephen, Diana was working on that after me. I have no idea about the current status.
Red Hat OpenStack Platform version 5 is now End-of-Life, and as such will not have further updates. See https://access.redhat.com/support/policy/updates/openstack/platform/ for full support lifecycle details.