Bug 1924952 - NodeDataLookup does not work due to invalid json input in puppet/extraconfig/pre_deploy/per_node.yaml
Summary: NodeDataLookup does not work due to invalid json input in puppet/extraconfig/...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.1 (Train)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: z6
: 16.1 (Train on RHEL 8.2)
Assignee: David Vallee Delisle
QA Contact: Jason Grosso
URL:
Whiteboard:
: 1941097 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-04 01:05 UTC by Kellen Gattis
Modified: 2024-10-01 17:26 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.3.2-1.20210104205662.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-26 13:50:41 UTC
Target Upstream Version: Train
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 775347 0 None MERGED per_node is not parsing generated json 2021-05-13 12:29:14 UTC
Red Hat Bugzilla 1924862 0 medium CLOSED NodeDataLookup does not work due to extra quotation in /etc/puppet/hiera.yaml for uuid lookup 2024-10-01 17:26:11 UTC
Red Hat Issue Tracker OSP-439 0 None None None 2022-10-03 16:39:37 UTC
Red Hat Product Errata RHBA-2021:2097 0 None None None 2021-05-26 13:51:11 UTC

Description Kellen Gattis 2021-02-04 01:05:14 UTC
Description of problem:
Following the instructions in https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/advanced_overcloud_customization/chap-configuration_hooks#sect-Customizing_Hieradata_for_Individual_Nodes
leads to a json decoding error.


Steps to Reproduce:
1.
Create a template with the following:
resource_registry:
  OS::TripleO::ComputeExtraConfigPre: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml

parameter_defaults:
  NodeDataLookup:
    {"C06C60C2-EC19-4237-B7EB-507F9E303EFD": {"nova::compute::cpu_dedicated_set": [ "1-4" ]}}

2.
Run a stack update

Actual results:
(heat-config) [INFO] b\\'\\'\\n[2021-02-03 23:55:58,781] (heat-config) [DEBUG] b\\'Traceback (most recent call last):\\\\n  File \"<string>\", line 5, in <module>\\\\n  File \"/usr/
lib64/python3.6/json/__init__.py\", line 354, in loads\\\\n    return _default_decoder.decode(s)\\\\n  File \"/usr/lib64/python3.6/json/decoder.py\", line 339, in decode\\\\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\\\n  File
\"/usr/lib64/python3.6/json/decoder.py\", line 355, in raw_decode\\\\n    obj, end = self.scan_once(s, idx)\\\\njson.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\\\\nTraceback (most r
ecent call last):\\\\n  File \"<string>\", line 5, in <module>\\\\n  File \"/usr/lib64/python3.6/json/__init__.py\", line 354, in loads\\\\n    return _default_decoder.decode(s)\\\\n  File \"/usr/lib64/python3.6/json/decoder.py\", line 339
, in decode\\\\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\\\n  File \"/usr/lib64/python3.6/json/decoder.py\", line 355, in raw_decode\\\\n    obj, end = self.scan_once(s, idx)\\\\njson.decoder.JSONDecodeError: Expecting proper
ty name enclosed in double quotes: line 1 column 2 (char 1)\\\\n\\'\\n

Expected results:
A file located at /etc/puppet/hieradata/<uuid>.json is created with the following content:
{"nova::compute::cpu_dedicated_set": [ "1-4" ]}}

Additional info:
It seems like the input to the script is coming in as a map/dictionary (using single quotes) rather than a proper json string, which causes json.loads to fail with the aforementioned decoding error.

  inputs:
    - name: node_lookup
      description:
      type: String
      value: |-
        {'C06C60C2-EC19-4237-B7EB-507F9E303EFD': {'nova::compute::cpu_dedicated_set': ['1-4']}}

Workaround:
Replacing json.loads(input) with ast.literal_eval(input) looks like it will properly create a dictionary object in python whether it's a json string or the single-quoted mapping style that heat is using.

Here's a patch demonstrating the changes:
--- a/usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml    2020-08-21 20:32:02.000000000 -0400
+++ b/usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml    2021-02-03 19:28:20.090743051 -0500
@@ -47,16 +47,18 @@
         echo $node_lookup | $(get_python) -c "
         import json
         import sys
+        import ast
         input = sys.stdin.readline() or '{}'
-        cnt = json.loads(input)
+        cnt = ast.literal_eval(input)
         print(json.dumps(cnt.get('${node_id}', {})))
         " > /etc/puppet/hieradata/${node_id}.json
         # handle upper case node id LP#1816652
         echo $node_lookup | $(get_python) -c "
         import json
         import sys
+        import ast
         input = sys.stdin.readline() or '{}'
-        cnt = json.loads(input)
+        cnt = ast.literal_eval(input)
         print(json.dumps(cnt.get('${node_id_upper}', {})))
         " > /etc/puppet/hieradata/${node_id_upper}.json

Comment 1 David Vallee Delisle 2021-02-06 00:24:30 UTC
Kellen,

Thanks for flagging this and to providing the patch.

[1] Reproduced this

I've submitted this patch [a] upstream and I will backport it to train to address this bug.

DVD

[a] https://review.opendev.org/774323

[1]
~~~
[2021-02-05 21:48:08,975] (heat-config) [INFO] deploy_action=CREATE\
[2021-02-05 21:48:08,975] (heat-config) [INFO] deploy_stack_id=overcloud-Compute-ylhk4xulanbt-0-iq4yrmavu2fr-ComputeExtraConfigPre-h3sri74r3k2n-NodeSpecificDeployment-ugiv4nr3g27o/697ed555-b51c-425d-ba0b-32ef1e149d87\
[2021-02-05 21:48:08,976] (heat-config) [INFO] deploy_resource_name=TripleOSoftwareDeployment\
[2021-02-05 21:48:08,976] (heat-config) [INFO] deploy_signal_transport=NO_SIGNAL\
[2021-02-05 21:48:08,976] (heat-config) [DEBUG] Running /var/lib/heat-config/heat-config-script/67fa688f-c518-49af-9835-8d8640948c05\
[2021-02-05 21:48:09,147] (heat-config) [INFO] b\\'\\'\
[2021-02-05 21:48:09,147] (heat-config) [DEBUG] b\\'Traceback (most recent call last):\\\
  File \"<string>\", line 5, in <module>\\\
  File \"/usr/lib64/python3.6/json/__init__.py\", line 354, in loads\\\
    return _default_decoder.decode(s)\\\
  File \"/usr/lib64/python3.6/json/decoder.py\", line 339, in decode\\\
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\\
  File \"/usr/lib64/python3.6/json/decoder.py\", line 355, in raw_decode\\\
    obj, end = self.scan_once(s, idx)\\\
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\\\
Traceback (most recent call last):\\\
  File \"<string>\", line 5, in <module>\\\
  File \"/usr/lib64/python3.6/json/__init__.py\", line 354, in loads\\\
    return _default_decoder.decode(s)\\\
  File \"/usr/lib64/python3.6/json/decoder.py\", line 339, in decode\\\
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\\
  File \"/usr/lib64/python3.6/json/decoder.py\", line 355, in raw_decode\\\
    obj, end = self.scan_once(s, idx)\\\
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\\\
\\'\
~~~

Comment 4 Rabi Mishra 2021-03-22 06:48:42 UTC
*** Bug 1941097 has been marked as a duplicate of this bug. ***

Comment 24 errata-xmlrpc 2021-05-26 13:50:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.6 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2097


Note You need to log in before you can comment on or make changes to this bug.