Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1924862

Summary:	NodeDataLookup does not work due to extra quotation in /etc/puppet/hiera.yaml for uuid lookup
Product:	Red Hat OpenStack	Reporter:	Kellen Gattis <kgattis>
Component:	openstack-tripleo-heat-templates	Assignee:	David Vallee Delisle <dvd>
Status:	CLOSED ERRATA	QA Contact:	Joe H. Rahme <jhakimra>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	16.1 (Train)	CC:	asalvati, bbowen, dvd, jamsmith, jzaher, mburns, nacurry, slinaber
Target Milestone:	z6	Keywords:	Patch, Triaged
Target Release:	16.1 (Train on RHEL 8.2)	Flags:	kgattis: needinfo-
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-11.3.2-1.20210104205662.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-26 13:50:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:	Train
Embargoed:

Description Kellen Gattis 2021-02-03 18:49:04 UTC

Description of problem:
We would like to use NodeDataLookup (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/advanced_overcloud_customization/chap-configuration_hooks#sect-Customizing_Hieradata_for_Individual_Nodes) to customize the pin sets on particular compute nodes and it is not working.

Additional information:

Using "puppet lookup --debug nova::compute::cpu_dedicated_set" to confirm whether the override was being picked up, we noticed that it wasn't and discovered that /etc/puppet/hiera.yaml has some extra quotation in it preventing the lookup from working.

# grep -A1 hierarchy /etc/puppet/hiera.yaml 
:hierarchy:
- '"%{::uuid}"'

In order for the above lookup to work, it either needs to be "%{::uuid}" or '%{::uuid}"

Observe the following:
# dmidecode --s system-uuid 
c06c60c2-ec19-4237-b7eb-507f9e303efd

# cat /etc/puppet/hieradata/c06c60c2-ec19-4237-b7eb-507f9e303efd.json 
{
        "nova::compute::cpu_dedicated_set": [ "1-4" ]
}

# puppet lookup --debug nova::compute::cpu_dedicated_set 2>/dev/null | grep -A50 'Searching for "nova::compute::cpu_dedicated_set"'
  Searching for "nova::compute::cpu_dedicated_set"
    Global Data Provider (hiera configuration version 3)
      Using configuration "/etc/puppet/hiera.yaml"
      Hierarchy entry "json"
        Path "/etc/puppet/hieradata/"c06c60c2-ec19-4237-b7eb-507f9e303efd".json"
          Original path: ""%{::uuid}""
          Path not found
        Path "/etc/puppet/hieradata/fqdn.json"
          Original path: "fqdn"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/docker_puppet.json"
          Original path: "docker_puppet"
          Path not found
        Path "/etc/puppet/hieradata/ansible_managed.json"
          Original path: "ansible_managed"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/heat_config_.json"
          Original path: "heat_config_%{::deploy_config_name}"
          Path not found
        Path "/etc/puppet/hieradata/config_step.json"
          Original path: "config_step"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/role_extraconfig.json"
          Original path: "role_extraconfig"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/extraconfig.json"
          Original path: "extraconfig"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/pci_passthrough_whitelist.json"
          Original path: "pci_passthrough_whitelist"
          Path not found
        Path "/etc/puppet/hieradata/service_names.json"
          Original path: "service_names"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/service_configs.json"
          Original path: "service_configs"
          Found key: "nova::compute::cpu_dedicated_set" value: []
--- []

Notice how this section was unsuccessful:
      Hierarchy entry "json"
        Path "/etc/puppet/hieradata/"c06c60c2-ec19-4237-b7eb-507f9e303efd".json"
          Original path: ""%{::uuid}""
          Path not found

After modifying /etc/puppet/hiera.yaml to use either "%{::uuid}" or '%{::uuid}' and running the command again, we get this:
# puppet lookup --debug nova::compute::cpu_dedicated_set 2>/dev/null | grep -A50 'Searching for "nova::compute::cpu_dedicated_set"'
  Searching for "nova::compute::cpu_dedicated_set"
    Global Data Provider (hiera configuration version 3)
      Using configuration "/etc/puppet/hiera.yaml"
      Hierarchy entry "json"
        Path "/etc/puppet/hieradata/c06c60c2-ec19-4237-b7eb-507f9e303efd.json"
          Original path: "%{::uuid}"
          Found key: "nova::compute::cpu_dedicated_set" value: [
            "1-4"
          ]
---
- 1-4

Looking at the code, the hiera.yaml seems to come from here:
https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.j2.yaml#L706

Workaround for issue:

I created the following patch for /usr/share/openstack-tripleo-heat-templates/overcloud.j2.yaml:

--- a/usr/share/openstack-tripleo-heat-templates/overcloud.j2.yaml      2020-08-21 20:32:05.000000000 -0400
+++ b/usr/share/openstack-tripleo-heat-templates/overcloud.j2.yaml      2021-02-03 13:05:56.727134779 -0500
@@ -587,7 +587,7 @@
             - {get_param: {{role.name}}ExtraConfig}
         extraconfig: {get_param: ExtraConfig}
         hieradata_files:
-          - '"%{::uuid}"'
+          - "%{::uuid}"
           - fqdn
           - docker_puppet # Optionally provided by container-puppet.py
           - ansible_managed
@@ -603,7 +603,7 @@
           - all_nodes # provided by tripleo-hieradata
           - vip_data # provided by tripleo-hieradata
           - net_ip_map
-          - '"%{::osfamily}"'
+          - "%{::osfamily}"
           # The following are required for compatibility with the Controller role
           # where some vendor integrations added hieradata via ExtraConfigPre
           - neutron_bigswitch_data # Optionally provided by Controller/ComputeExtraConfigPre

The result is that /etc/puppet/hiera.yaml gets created like this on the nodes:
---
:backends:
  - json
:json:
  :datadir: /etc/puppet/hieradata
:hierarchy:
- '%{::uuid}'
- fqdn
- docker_puppet
- ansible_managed
- heat_config_%{::deploy_config_name}
- config_step
- role_extraconfig
- extraconfig
- pci_passthrough_whitelist
- service_names
- service_configs
- cloud_domain
- bootstrap_node
- all_nodes
- vip_data
- net_ip_map
- '%{::osfamily}'
- neutron_bigswitch_data
- upgrade

The single quotes work fine in /etc/puppet/hiera.yaml and nova.conf gets updated as expected when puppet runs:

# grep -E '^cpu_dedicated_set' /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf 
cpu_dedicated_set=1-4

Comment 1 David Vallee Delisle 2021-02-06 00:51:10 UTC

[1] Reproduced the issue

[2] Changing /etc/puppet/hiera.yaml yield the correct values

Testing the fix in overcloud.j2.yaml


[1]
~~~
  Searching for "nova::compute::cpu_dedicated_set"
    Global Data Provider (hiera configuration version 3)
      Using configuration "/etc/puppet/hiera.yaml"
      Hierarchy entry "json"
        Path "/etc/puppet/hieradata/"09e1660c-caf5-44a4-b0d4-a65af62cfc16".json"
          Original path: ""%{::uuid}""
          Path not found
        Path "/etc/puppet/hieradata/fqdn.json"
          Original path: "fqdn"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/docker_puppet.json"
          Original path: "docker_puppet"
          Path not found
        Path "/etc/puppet/hieradata/ansible_managed.json"
          Original path: "ansible_managed"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/heat_config_.json"
          Original path: "heat_config_%{::deploy_config_name}"
          Path not found
        Path "/etc/puppet/hieradata/config_step.json"
          Original path: "config_step"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/role_extraconfig.json"
          Original path: "role_extraconfig"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/extraconfig.json"
          Original path: "extraconfig"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/pci_passthrough_whitelist.json"
          Original path: "pci_passthrough_whitelist"
          Path not found
        Path "/etc/puppet/hieradata/service_names.json"
          Original path: "service_names"
          No such key: "nova::compute::cpu_dedicated_set"
        Path "/etc/puppet/hieradata/service_configs.json"
          Original path: "service_configs"
~~~

[2]
~~~
      Using configuration "/etc/puppet/hiera.yaml"
      Hierarchy entry "json"
        Merge strategy hash
          Path "/etc/puppet/hieradata/09e1660c-caf5-44a4-b0d4-a65af62cfc16.json"
            Original path: "%{::uuid}"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/fqdn.json"
            Original path: "fqdn"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/docker_puppet.json"
            Original path: "docker_puppet"
            Path not found
          Path "/etc/puppet/hieradata/ansible_managed.json"
            Original path: "ansible_managed"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/heat_config_.json"
            Original path: "heat_config_%{::deploy_config_name}"
            Path not found
          Path "/etc/puppet/hieradata/config_step.json"
            Original path: "config_step"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/role_extraconfig.json"
            Original path: "role_extraconfig"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/extraconfig.json"
            Original path: "extraconfig"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/pci_passthrough_whitelist.json"
            Original path: "pci_passthrough_whitelist"
            Path not found
          Path "/etc/puppet/hieradata/service_names.json"
            Original path: "service_names"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/service_configs.json"
            Original path: "service_configs"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/cloud_domain.json"
            Original path: "cloud_domain"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/bootstrap_node.json"
            Original path: "bootstrap_node"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/all_nodes.json"
            Original path: "all_nodes"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/vip_data.json"
            Original path: "vip_data"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/net_ip_map.json"
            Original path: "net_ip_map"
            No such key: "lookup_options"
          Path "/etc/puppet/hieradata/RedHat.json"
            Original path: "%{::osfamily}"
            Path not found
          Path "/etc/puppet/hieradata/neutron_bigswitch_data.json"
            Original path: "neutron_bigswitch_data"
            Path not found
          Path "/etc/puppet/hieradata/upgrade.json"
            Original path: "upgrade"
            Path not found
    Module data provider for module "nova" not found
  Searching for "nova::compute::cpu_dedicated_set"
    Global Data Provider (hiera configuration version 3)
      Using configuration "/etc/puppet/hiera.yaml"
      Hierarchy entry "json"
        Path "/etc/puppet/hieradata/09e1660c-caf5-44a4-b0d4-a65af62cfc16.json"
          Original path: "%{::uuid}"
          Found key: "nova::compute::cpu_dedicated_set" value: [
            "1-4"
          ]
~~~

Comment 2 David Vallee Delisle 2021-02-06 02:19:50 UTC

Confirmed the issue is solved with proposed patch, will push hard to have this merged to 16.1z5.

Comment 19 errata-xmlrpc 2021-05-26 13:50:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.6 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2097