update - I was able to proceed by manually disabling the check in /usr/libexec/heat-config/hooks/hiera on all controllers and ran the ansible upgrade steps... after the upgrade was complete the 'old' data is now completely gone:

[root@overcloud-compute-0 os-collect-config]# os-apply-config --key hiera.datafiles --type raw --key-default empty
empty

[root@overcloud-controller-1 ~]# os-apply-config --key hiera.datafiles --type raw --key-default empty
empty

First try to overcome this problem by using a more deterministic approach to the check.  Hard to test.  Must be deployed and patch on all running nodes:

On each node before the composable upgrade:


     curl https://review.openstack.org/changes/453726/revisions/current/patch?download | base64 -d | sed -e 's,heat-config-hiera/install.d/hook-hiera.py,hiera,' | sudo patch  -d /usr/libexec/heat-config/hooks/ -p1

(In reply to Sofer Athlan-Guyot from comment #2)
> First try to overcome this problem by using a more deterministic approach to
> the check.  Hard to test.  Must be deployed and patch on all running nodes:
> 
> On each node before the composable upgrade:
> 
> 
>      curl
> https://review.openstack.org/changes/453726/revisions/current/patch?download
> | base64 -d | sed -e 's,heat-config-hiera/install.d/hook-hiera.py,hiera,' |
> sudo patch  -d /usr/libexec/heat-config/hooks/ -p1

I'm also running this (mostly to get to the upgraded state again so we can check that) on my 2nd env... fwiw you can also apply to the upgrade init by adding this to the end of your /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml

    curl "https://review.openstack.org/gitweb?p=openstack/heat-agents.git;a=blob_plain;f=heat-config-hiera/install.d/hook-hiera.py;hb=55508e3fb05cf99f003b8309119d6861e85ad082" -o /usr/libexec/heat-config/hooks/hiera

adding DFG:DF for visibility and see if anyone has better suggestion than https://review.openstack.org/#/c/453726/3/heat-config-hiera/install.d/hook-hiera.py

also adding DFG:CloudApp (heat folks) who are the ones that can actually land the proposed change anyway

One option might be to package up this:

http://git.openstack.org/cgit/openstack/tripleo-puppet-elements/tree/elements/hiera/10-hiera-disable

Installing this file into the /usr/libexec/os-refresh-config location would remove the old hook files before they cause problems I think and effectively disable the old Hiera data as well. Because this runs before 50- the existing heat checks could stay in place right?

I suppose we'd need a way to deploy this file though. Perhaps we could inline it into another project so it gets installed via an RPM. Or alternatively use the 'DeploymentArtifacts' mechanism to deploy it as a tarball or rpm. The DeploymentArtifacts mechanism runs very early on

Dan

I see major-upgrade-composable-steps.yaml is doing the same as 10-hiera-disable, which is good.

It looks like Marios is correct, https://review.openstack.org/#/c/453726/ should be proposed to stable/ocata only. I'll keep an eye on it and propose an upstream heat-agents-1.0.1 as soon as it lands.

Patch proposed as a stable/ocata only review.

adding a note here on request, the fix for this by Sofer in the trackers above in https://review.openstack.org/#/c/454556/ openstack/heat-agents. To make the fix available we need a release of heat-agents and sbaker kindly sorted that with heat-agents-1.0.1 https://review.openstack.org/#/c/454912 (added to trackers). 

This isn't yet appearing in downstream puddle, at least it wasn't during my upgrade run today/from this morning (openstack-heat-agents-1.0.0-3.el7ost.noarch). You can apply the fix as a workaround until it does appear in 11 by "patching" the UpgradeInitCommand from the upstream review (proceed cautiously, first line backs up the templates):

    # sudo cp -r /usr/share/openstack-tripleo-heat-templates /usr/share/openstack-tripleo-heat-templates.ORIG_BZ_1439121
    # https://review.openstack.org/#/c/454556/ Simplify legacy hieradata check to avoid false positive.
    echo '    curl "https://review.openstack.org/gitweb?p=openstack/heat-agents.git;a=blob_plain;f=heat-config-hiera/install.d/hook-hiera.py;h=db69f69c517e2693222080862db52abfaa3fbdc5;hb=769d0de4d9acb07563202a2ef3d3a0aa1b9542d1" -o /usr/libexec/heat-config/hooks/hiera' \
          | sudo tee -a /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml > /dev/null

So +1 morazi moving to POST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245