Bug 1569751

Summary: OSP10->11->12->13 undercloud upgrade post validation fail: Task 'Check Keystone' failed:\nHost: localhost\nMessage: Status code was not [201]: HTTP Error 404: Not Found\n\nFailure! The validation failed for all hosts:\n*
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-validationsAssignee: Carlos Camacho <ccamacho>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: augol, ccamacho, dbecker, jfrancoa, jjoyce, jschluet, lbezdick, mbracho, mburns, morazi, sclewis, slinaber, tvignaud, yprokule
Target Milestone: rcKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-validations-8.4.1-5.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:52:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2018-04-19 23:54:13 UTC
Description of problem:
OSP10->11->12->13 undercloud upgrade post validation fail: 

2018-04-19 19:48:37,946 INFO: Starting and waiting for validation groups ['post-upgrade'] 
2018-04-19 19:49:35,343 ERROR: ERROR error running the validation groups ['post-upgrade']   {"stderr": "[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of become which is a\n generic framework . This feature will be removed in version 2.8. Deprecation \nwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\n", "stdout": "Task 'Check Keystone' failed:\nHost: localhost\nMessage: Status code was not [201]: HTTP Error 404: Not Found\n\nFailure! The validation failed for all hosts:\n* localhost\n"} Mistral execution ID: ef78e9c4-3f5e-4776-87eb-91992d5b02fd

The error is only related to the post upgrade validation and it does not block the upgrade.

Version-Release number of selected component (if applicable):
instack-undercloud-8.4.0-4.el7ost.noarch
openstack-tripleo-validations-8.4.0-2.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10
2. Upgrade undercloud to OSP11
3. Upgrade undercloud to OSP12
4. Upgrade undercloud to OSP13

Actual results:
post-upgrade undercloud validation fail:
2018-04-19 19:49:35,343 ERROR: ERROR error running the validation groups ['post-upgrade']   {"stderr": "[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of become which is a\n generic framework . This feature will be removed in version 2.8. Deprecation \nwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\n", "stdout": "Task 'Check Keystone' failed:\nHost: localhost\nMessage: Status code was not [201]: HTTP Error 404: Not Found\n\nFailure! The validation failed for all hosts:\n* localhost\n"} Mistral execution ID: ef78e9c4-3f5e-4776-87eb-91992d5b02fd

Expected results:
post-upgrade validations succeed

Additional info:

Comment 2 Yurii Prokulevych 2018-04-23 09:50:26 UTC
Got same issue during regular 12 -> 13 major upgrade.
Package: openstack-tripleo-validations-8.4.0-2.el7ost.noarch

Comment 3 Yurii Prokulevych 2018-04-23 10:05:18 UTC
So after running validation the problem seems to be a malformed url used for 'Check Keystone' task:
    https://10.0.0.101:13000/v2.0/v3/auth/tokens

The url is defined as 'url: "{{ overcloud_keystone_url }}/v3/auth/tokens"'
and 'overcloud_keystone_url' that is generated with:

 tripleo-ansible-inventory --static-yaml-inventory inventory

has a trailing 'v2.0' suffix

Comment 4 Jose Luis Franco 2018-04-23 12:06:19 UTC
overcloud_keystone_url parameter gets its value from the KeystoneURL stack parameter:

https://github.com/openstack/tripleo-common/blob/master/tripleo_common/inventory.py#L177-L179

This stack output is at the same time defined in tripleo-heat-templates from the value of KeystonePublic.uri

https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/overcloud.j2.yaml#L933

Which in stable/pike (which is the version we come from before upgrading the undercloud) is defined appending /v2.0:

https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/network/endpoints/endpoint_map.yaml#L4085

For that reason, Yurii has defined in its inventory the value: https://10.0.0.101:13000/v2.0/ for 'overcloud_keystone_url'.

I'm preparing a patch for it so we leave the version aside when creating overcloud_keystone_url.

Comment 7 Carlos Camacho 2018-05-09 16:43:43 UTC
We need the blocker flag in this please. The fix is merged downstream but we cant create the package

Comment 8 Maria Bracho 2018-05-09 17:23:04 UTC
acked.

Comment 15 errata-xmlrpc 2018-06-27 13:52:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086