Bug 1654449

The hiera data file is created correctly in /etc/puppet/hieradata/.  The issue is it is unused when configuring the containers.  /var/lib/docker-puppet/docker-puppet.py re-proposes the uuid fact:

From /var/lib/docker-puppet/docker-puppet.py:

sh_script = '/var/lib/docker-puppet/docker-puppet.sh'
with open(sh_script, 'w') as script_file:
    os.chmod(script_file.name, 0755)
    script_file.write("""#!/bin/bash
    set -ex
    mkdir -p /etc/puppet
    cp -a /tmp/puppet-etc/* /etc/puppet
    rm -Rf /etc/puppet/ssl # not in use and causes permission errors
    echo "{\\"step\\": $STEP}" > /etc/puppet/hieradata/docker.json
    TAGS=""
    if [ -n "$PUPPET_TAGS" ]; then
        TAGS="--tags \"$PUPPET_TAGS\""
    fi

    # Create a reference timestamp to easily find all files touched by
    # puppet. The sync ensures we get all the files we want due to
    # different timestamp.
    origin_of_time=/var/lib/config-data/${NAME}.origin_of_time
    touch $origin_of_time
    sync

    set +e
    FACTER_hostname=$HOSTNAME FACTER_uuid=docker /usr/bin/puppet apply --summarize \
    --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules $TAGS /etc/config.pp

Ah you need https://review.openstack.org/#/c/607647/. Let me check if it's backported downstream yet.

I used to the following change as a test to work around the issue.  Not sure if this is viable but it correctly applied per node settings.

$ git diff .
diff --git a/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml b/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_no
index c39b485..39f8cd6 100644
--- a/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml
+++ b/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml
@@ -40,7 +40,7 @@ resources:
         input = sys.stdin.readline() or '{}'
         cnt = json.loads(input)
         print json.dumps(cnt.get('${node_id}', {}))
-        " > /etc/puppet/hieradata/${node_id}.json
+        " > /etc/puppet/hieradata/per_node.json
 
   NodeSpecificDeployment:
     type: OS::Heat::SoftwareDeployment
diff --git a/openstack-tripleo-heat-templates/puppet/role.role.j2.yaml b/openstack-tripleo-heat-templates/puppet/role.role.j2.yaml
index e9b208b..eba9d41 100644
--- a/openstack-tripleo-heat-templates/puppet/role.role.j2.yaml
+++ b/openstack-tripleo-heat-templates/puppet/role.role.j2.yaml
@@ -532,6 +532,7 @@ resources:
       config:
         hierarchy:
           - '"%{::uuid}"'
+          - per_node
           - heat_config_%{::deploy_config_name}
           - config_step
           - {{role.name.lower()}}_extraconfig

Duplicate of : https://bugzilla.redhat.com/show_bug.cgi?id=1635334

Fixed-In: puppet-tripleo-8.3.4-13.el7ost openstack-tripleo-heat-templates-8.0.4-36.el7ost

I'll mark it duplicate once confirmed.

Thanks Rabi; thought I was working with the current release.  I will confirm; sorry for the noise.

Looks like there is still an issue with the current version.

rpm -q openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-8.0.7-4.el7ost.noarch


As I saw yesterday it appears the uuid fact does not resolve inside a container so the  "%{::uuid}" hierarchy does not work even now that it's not being replaced.

To show the issue using Steve's awesome blog on debugging containerized deployments (http://hardysteven.blogspot.com/2018/06/tripleo-containerized-deployments.html).

On a compute node we see that debug did not get set to true as defined in the per node config.

[root@overcloud-compute-0 ~]# facter uuid
a8fe66ff-3c42-42fb-804b-7612ba92af41
[root@overcloud-compute-0 ~]# cat /etc/puppet/hieradata/a8fe66ff-3c42-42fb-804b-7612ba92af41.json 
{"nova::debug": "True"}

[root@overcloud-compute-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf DEFAULT debug
False

We make a small change to docker-puppet.py to show ::uuid does not work inside the container.

[root@overcloud-compute-0 docker-puppet]# git diff docker-puppet.py 
diff --git a/docker-puppet.py b/docker-puppet.py
index 8f0736c..8157f3e 100644
--- a/docker-puppet.py
+++ b/docker-puppet.py
@@ -225,6 +225,7 @@ with open(sh_script, 'w') as script_file:
     set +e
     # $::deployment_type in puppet-tripleo
     export FACTER_deployment_type=containers
+    echo "uuid set to: $(facter uuid)"
     FACTER_hostname=$HOSTNAME /usr/bin/puppet apply --summarize \
     --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules $TAGS /etc/config.pp
     rc=$?

We re-apply the containerized puppet and show that uuid is not available to facter:

[root@overcloud-compute-0 ~]# jq '[.[]|select(.config_volume | contains("nova"))]' /var/lib/docker-puppet/docker-puppet.json | tee /tmp/nova.json
[
  {
    "config_image": "172.16.6.1:8787/rhosp13/openstack-nova-compute:13.0-70",
    "step_config": "# TODO(emilien): figure how to deal with libvirt profile.\n# We'll probably treat it like we do with Neutron plugins.\n# Until then, just include it in the default nova-compute role.\ninclude tripleo::profile::base::nova::compute::libvirt\n\ninclude ::tripleo::profile::base::database::mysql::client",
    "config_volume": "nova_libvirt",
    "puppet_tags": "nova_config,nova_paste_api_ini"
  },
  {
    "config_image": "172.16.6.1:8787/rhosp13/openstack-nova-compute:13.0-70",
    "step_config": "include tripleo::profile::base::nova::libvirt\n\ninclude ::tripleo::profile::base::database::mysql::client",
    "config_volume": "nova_libvirt",
    "puppet_tags": "libvirtd_config,nova_config,file,libvirt_tls_password"
  },
  {
    "step_config": "include ::tripleo::profile::base::sshd\ninclude tripleo::profile::base::nova::migration::target",
    "config_volume": "nova_libvirt",
    "config_image": "172.16.6.1:8787/rhosp13/openstack-nova-compute:13.0-70"
  }
]

export NET_HOST='true'
export DEBUG='true'
export PROCESS_COUNT=1
export CONFIG=/tmp/nova.json
[root@overcloud-compute-0 docker-puppet]# python /var/lib/docker-puppet/docker-puppet.py |grep 'uuid set to'
2018-11-29 13:09:51,088 DEBUG: 89842 -- uuid set to: 
+ echo 'uuid set to: '

Verified that openstack-tripleo-heat-templates-8.0.7-19.el7ost resolves the issue for me.

-- Reproduce the issue with older templates

(undercloud) [stack@undercloud13 templates]$ rpm -q openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-8.0.7-4.el7ost.noarch

openstack overcloud deploy --stack overcloud \
[..]
-e /home/stack/templates/per_node_hiera.yaml


(undercloud) [stack@undercloud13 templates]$ cat /home/stack/templates/per_node_hiera.yaml
resource_registry:
  OS::TripleO::ComputeExtraConfigPre: /home/stack/templates/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml
  OS::TripleO::ControllerExtraConfigPre: /home/stack/templates/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml

parameter_defaults:
  NodeDataLookup: |
     {"a8fe66ff-3c42-42fb-804b-7612ba92af41": {"nova::debug": "True"},
      "55907dbc-250b-42a8-a67c-1489cdfecd7f": {"nova::debug": "True"}}

[root@overcloud-compute-0 ~]# dmidecode |grep -i uuid
        UUID: a8fe66ff-3c42-42fb-804b-7612ba92af41


[root@overcloud-controller-0 ~]# dmidecode |grep -i uuid
        UUID: 55907dbc-250b-42a8-a67c-1489cdfecd7f


[root@overcloud-compute-0 ~]# cat /etc/puppet/hieradata/a8fe66ff-3c42-42fb-804b-7612ba92af41.json 
{"nova::debug": "True"}
[root@overcloud-compute-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf DEFAULT debug
False

[root@overcloud-controller-0 ~]# cat /etc/puppet/hieradata/55907dbc-250b-42a8-a67c-1489cdfecd7f.json 
{"nova::debug": "True"}
[root@overcloud-controller-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug                                      
False

-- Upgrade templates and verify fix.

(undercloud) [stack@undercloud13 templates]$ sudo rpm -Uvh openstack-tripleo-heat-templates-8.0.7-23.el7ost.noarch.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:openstack-tripleo-heat-templates-################################# [ 50%]
Cleaning up / removing...
   2:openstack-tripleo-heat-templates-################################# [100%]

cp -R /usr/share/openstack-tripleo-heat-templates /home/stack/templates/openstack-tripleo-heat-templates

openstack overcloud deploy --stack overcloud \
[..]
-e /home/stack/templates/per_node_hiera.yaml

[root@overcloud-compute-0 ~]# cat /etc/puppet/hieradata/a8fe66ff-3c42-42fb-804b-7612ba92af41.json
{"nova::debug": "True"}
[root@overcloud-compute-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf DEFAULT debug
True

[root@overcloud-controller-0 ~]# cat /etc/puppet/hieradata/55907dbc-250b-42a8-a67c-1489cdfecd7f.json 
{"nova::debug": "True"}
[root@overcloud-controller-0 ~]# crudini --get /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf DEFAULT debug
True

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0068