When the metadata for a resource is requested from Heat, Heat fetches the value of every attribute of the resource, even though this data is not returned by the API. This generally means at least one and possibly multiple pointless ReST API calls to the OpenStack service underlying the resource. The metadata in Heat gets polled regularly by in-guest agents like os-collect-config, because it is the mechanism by which software deployments are triggered. In a large deployment (like a substantial TripleO cloud), there is a lot of polling... enough to put a substantial load on Heat. This is probably one of the reasons for the load. In addition, because polling from a server generally uses a Heat "stack user" account from a different keystone domain rather than the stack owner's account, the resource cannot actually be found and the log is polluted with a lot of useless messages of the form: WARNING heat.engine.clients.os.nova [req-2c4da8b3-f441-43c3-bb59-bbe1a6665a6f None 2240eae5cccc439298daa101fc8557bd-96361e11-11f8-45d1-a89b-732401f] Instance (099bb7bb-f22d-4739-a077-4776a09bbda9) not found: Instance 099bb7bb-f22d-4739-a077-4776a09bbda9 could not be found. (HTTP 404) And of course nova-api also logs the 404s.
on latest version as of now, none of those messages show in the logs.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-0266.html