Bug 1304854

Summary:	Metadata polling unnecessarily calls Nova
Product:	Red Hat OpenStack	Reporter:	Zane Bitter <zbitter>
Component:	openstack-heat	Assignee:	Zane Bitter <zbitter>
Status:	CLOSED ERRATA	QA Contact:	Amit Ugol <augol>
Severity:	high	Docs Contact:
Priority:	high
Version:	7.0 (Kilo)	CC:	mburns, rhel-osp-director-maint, sbaker, shardy, ssainkar, yeylon
Target Milestone:	z4	Keywords:	ZStream
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-heat-2015.1.2-8.el7ost	Doc Type:	Bug Fix
Doc Text:	Previously, when the metadata for a resource was requested from Heat, Heat fetched the value of every attribute of the resource, even though this data was not returned by the API. This meant at least one and possibly multiple pointless ReST API calls to the OpenStack service underlying the resource. The metadata in Heat gets polled regularly by in-guest agents like os-collect-config, because it is the mechanism by which software deployments are triggered. Because polling from a server generally uses a Heat "stack user" account from a different keystone domain rather than the stack owner's account, the resource cannot actually be found and "404 Not Found" messages were accumulating in the logs of both heat-engine and nova-api. Heat no longer calculates attribute values when only the metadata for a resource is requested.	Story Points:	---
Clone Of:
Clones:	1304856 (view as bug list)		Environment:
Last Closed:	2016-02-18 16:43:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1304856

Description Zane Bitter 2016-02-04 19:43:55 UTC

When the metadata for a resource is requested from Heat, Heat fetches the value of every attribute of the resource, even though this data is not returned by the API. This generally means at least one and possibly multiple pointless ReST API calls to the OpenStack service underlying the resource.

The metadata in Heat gets polled regularly by in-guest agents like os-collect-config, because it is the mechanism by which software deployments are triggered. In a large deployment (like a substantial TripleO cloud), there is a lot of polling... enough to put a substantial load on Heat. This is probably one of the reasons for the load.

In addition, because polling from a server generally uses a Heat "stack user" account from a different keystone domain rather than the stack owner's account, the resource cannot actually be found and the log is polluted with a lot of useless messages of the form:

WARNING heat.engine.clients.os.nova [req-2c4da8b3-f441-43c3-bb59-bbe1a6665a6f None 2240eae5cccc439298daa101fc8557bd-96361e11-11f8-45d1-a89b-732401f] Instance (099bb7bb-f22d-4739-a077-4776a09bbda9) not found: Instance 099bb7bb-f22d-4739-a077-4776a09bbda9 could not be found. (HTTP 404)

And of course nova-api also logs the 404s.

Comment 3 Amit Ugol 2016-02-16 12:17:36 UTC

on latest version as of now, none of those messages show in the logs.

Comment 5 errata-xmlrpc 2016-02-18 16:43:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-0266.html