Bug 1717169

Summary: [upgrades][13->14] Overcloud upgrade failed on underfined variable error 'python_interpreter'
Product: Red Hat OpenStack Reporter: Archit Modi <amodi>
Component: openstack-tripleo-heat-templatesAssignee: Ade Lee <alee>
Status: CLOSED ERRATA QA Contact: Pavan <pkesavar>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: alee, dmendiza, jagee, jfrancoa, jgrosso, jjoyce, jschluet, mburns, pkesavar, sasha, slinaber, tvignaud
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.3.1-0.20190513171730.9995be9.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1718963 1718964 (view as bug list) Environment:
Last Closed: 2019-07-02 20:09:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1691280, 1711990, 1712040, 1712041, 1712357, 1713694, 1713696, 1713751, 1713754, 1718963, 1718964    
Attachments:
Description Flags
overcloud-upgrade-run.log none

Description Archit Modi 2019-06-04 20:12:43 UTC
Description of problem: Deploy RHOS 13 with TLS everywhere enabled and the mentioned topology. While upgrading from RHos 13 to Rhos 14, the upgrade fails on undefined variable error "python_interpreter"

Topology: 3 controller, 2 compute, 1 freeipa, TLS everywhere enabled

Steps to Reproduce:
1. Deploy RHOS 13 with mentioned topology
2. Start upgrade procedure from Rhos 13 to Rhos 14
3. Upgrade fails at Ovecloud upgrade stage

Actual results:

2019-06-04 01:56:20 | 
2019-06-04 01:56:20 | TASK [create enrollment script] ************************************************
2019-06-04 01:56:20 | Tuesday 04 June 2019  01:56:19 -0400 (0:00:05.920)       0:02:03.783 ********** 
2019-06-04 01:56:20 | fatal: [compute-1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'python_interpreter' is undefined\n\nThe error appears to have been in '/var/lib/mistral/748701bf-d539-4283-a00c-2b3dc94e8938/Compute/host_prep_tasks.yaml': line 117, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    - hostname\n  - copy:\n    ^ here\n"}
2019-06-04 01:56:20 | fatal: [compute-0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'python_interpreter' is undefined\n\nThe error appears to have been in '/var/lib/mistral/748701bf-d539-4283-a00c-2b3dc94e8938/Compute/host_prep_tasks.yaml': line 117, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    - hostname\n  - copy:\n    ^ here\n"}
2019-06-04 01:56:20 | 
2019-06-04 01:56:20 | NO MORE HOSTS LEFT *************************************************************
2019-06-04 01:56:20 | 
2019-06-04 01:56:20 | PLAY RECAP *********************************************************************
2019-06-04 01:56:20 | compute-0                  : ok=97   changed=24   unreachable=0    failed=1   
2019-06-04 01:56:20 | compute-1                  : ok=97   changed=24   unreachable=0    failed=1   
2019-06-04 01:56:20 | 

Expected results:
Upgrade passed

Additional info:

Comment 1 Archit Modi 2019-06-04 20:14:05 UTC
Created attachment 1577272 [details]
overcloud-upgrade-run.log

Comment 3 Jose Luis Franco 2019-06-06 05:43:24 UTC
So, having a look at the error and checking the involved template. This seems to be an error in an undefined ansible variable:

https://github.com/openstack/tripleo-heat-templates/blob/stable/rocky/extraconfig/services/ipaclient.yaml#L124

The ipaclient template has a function in the host_prep_tasks which references an undefined "python_interpreter" variable added in https://review.opendev.org/#/c/661103/:

function get_fqdn {
                    # Get the instance hostname out of the metadata
                    fqdn=`echo $data | {{ python_interpreter }} -c 'import json,sys;obj=json.load(sys.stdin);print(obj.get("join", {}).get("hostname", ""))'`
                    if [ -z "$fqdn"]; then
                        echo "Unable to determine hostname"
                        return 1
                    fi
                    return 0
                }

The fix should be easy, so I'm moving it to the Security DFG who are in charge of this service.

Comment 8 Ade Lee 2019-06-13 17:24:39 UTC
*** Bug 1719970 has been marked as a duplicate of this bug. ***

Comment 19 errata-xmlrpc 2019-07-02 20:09:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1672