Bug 1438015

Summary:	Heat doesn't renew token leading to authorization failure in deployments going past default 240 min timeout
Product:	Red Hat OpenStack	Reporter:	Sai Sindhur Malleni <smalleni>
Component:	instack-undercloud	Assignee:	Rabi Mishra <ramishra>
Status:	CLOSED ERRATA	QA Contact:	Ronnie Rasouli <rrasouli>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	10.0 (Newton)	CC:	mburns, ramishra, rhel-osp-director-maint, sbaker, shardy, srevivo, tvignaud, zbitter
Target Milestone:	rc	Keywords:	Triaged, ZStream
Target Release:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	instack-undercloud-7.1.1-0.20170616135935.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-13 21:22:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sai Sindhur Malleni 2017-03-31 15:55:35 UTC

Description of problem:
When doing large scale deployments where timeout needs to be set higher than 240 mins, we see authorization failures in deployment command around the time default timeout 240 mins reached due to keystone token expiration set to 14400s (240mins). Heat doesn't seem to renew token from keystone, so the workaround is to also bump keystone token expiration time to the timeout value used in overcloud deploy command. We see this

2017-03-18 19:54:22Z [overcloud.Compute]: CREATE_FAILED Resource CREATE failed: Unauthorized: resources[81].resources.NovaCompute: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f3373924-3da4-4349-8b46-b2430ad3dc3f)
2017-03-18 19:54:22Z [overcloud.Compute]: CREATE_FAILED Unauthorized: resources.Compute.resources[81].resources.NovaCompute: The request you have made requires authentication. (HTTP 401) (Request-ID: req- f3373924-3da4-4349-8b46-b2430ad3dc3f)
2017-03-18 19:54:23Z [overcloud]: CREATE_FAILED Resource CREATE failed: Unauthorized: resources.Compute.resources[81].resources.NovaCompute: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f3373924-3da4-4349-8b46-b2430ad3dc3f)
Version-Release number of selected component (if applicable):
RHOP 10

How reproducible:
100%

Steps to Reproduce:
1. Do large scale deployments
2. Bump timeout in deploy command to > 240 mins.
3.

Actual results:
Although timeout was set to 360minutes, stack create failed because of authorization errors.

Expected results:
Deployment should continue until timeout passed to the overcloud deploy command

Additional info:

Comment 1 Rabi Mishra 2017-04-17 06:00:52 UTC

To allow re-authentication on token expiry, such that long-running tasks may complete, heat has a flag 'reauthentication_auth_method', which can be set to 'trusts' in heat.conf. This would allow for trust to be used in place of user token.

Comment 2 Zane Bitter 2017-04-18 15:27:52 UTC

Do we need to change something in TripleO to make that the default?

Comment 3 Zane Bitter 2017-05-18 14:12:27 UTC

Fixed upstream, but backports are not feasible due to reliance on new features as well as bug fixes in other projects. Retargeting for OSP12.

Comment 8 Ronnie Rasouli 2017-12-05 08:49:11 UTC

fixed landed on downstream

Comment 11 errata-xmlrpc 2017-12-13 21:22:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462