Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1924952

Summary: NodeDataLookup does not work due to invalid json input in puppet/extraconfig/pre_deploy/per_node.yaml
Product: Red Hat OpenStack Reporter: Kellen Gattis <kgattis>
Component: openstack-tripleo-heat-templatesAssignee: David Vallee Delisle <dvd>
Status: CLOSED ERRATA QA Contact: Jason Grosso <jgrosso>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: asalvati, bbowen, dhill, dvd, jamsmith, jgrosso, jzaher, mburns, nacurry, slinaber
Target Milestone: z6Keywords: Patch, Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-11.3.2-1.20210104205662.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-26 13:50:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: Train
Embargoed:

Description Kellen Gattis 2021-02-04 01:05:14 UTC
Description of problem:
Following the instructions in https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/advanced_overcloud_customization/chap-configuration_hooks#sect-Customizing_Hieradata_for_Individual_Nodes
leads to a json decoding error.


Steps to Reproduce:
1.
Create a template with the following:
resource_registry:
  OS::TripleO::ComputeExtraConfigPre: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml

parameter_defaults:
  NodeDataLookup:
    {"C06C60C2-EC19-4237-B7EB-507F9E303EFD": {"nova::compute::cpu_dedicated_set": [ "1-4" ]}}

2.
Run a stack update

Actual results:
(heat-config) [INFO] b\\'\\'\\n[2021-02-03 23:55:58,781] (heat-config) [DEBUG] b\\'Traceback (most recent call last):\\\\n  File \"<string>\", line 5, in <module>\\\\n  File \"/usr/
lib64/python3.6/json/__init__.py\", line 354, in loads\\\\n    return _default_decoder.decode(s)\\\\n  File \"/usr/lib64/python3.6/json/decoder.py\", line 339, in decode\\\\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\\\n  File
\"/usr/lib64/python3.6/json/decoder.py\", line 355, in raw_decode\\\\n    obj, end = self.scan_once(s, idx)\\\\njson.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\\\\nTraceback (most r
ecent call last):\\\\n  File \"<string>\", line 5, in <module>\\\\n  File \"/usr/lib64/python3.6/json/__init__.py\", line 354, in loads\\\\n    return _default_decoder.decode(s)\\\\n  File \"/usr/lib64/python3.6/json/decoder.py\", line 339
, in decode\\\\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\\\n  File \"/usr/lib64/python3.6/json/decoder.py\", line 355, in raw_decode\\\\n    obj, end = self.scan_once(s, idx)\\\\njson.decoder.JSONDecodeError: Expecting proper
ty name enclosed in double quotes: line 1 column 2 (char 1)\\\\n\\'\\n

Expected results:
A file located at /etc/puppet/hieradata/<uuid>.json is created with the following content:
{"nova::compute::cpu_dedicated_set": [ "1-4" ]}}

Additional info:
It seems like the input to the script is coming in as a map/dictionary (using single quotes) rather than a proper json string, which causes json.loads to fail with the aforementioned decoding error.

  inputs:
    - name: node_lookup
      description:
      type: String
      value: |-
        {'C06C60C2-EC19-4237-B7EB-507F9E303EFD': {'nova::compute::cpu_dedicated_set': ['1-4']}}

Workaround:
Replacing json.loads(input) with ast.literal_eval(input) looks like it will properly create a dictionary object in python whether it's a json string or the single-quoted mapping style that heat is using.

Here's a patch demonstrating the changes:
--- a/usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml    2020-08-21 20:32:02.000000000 -0400
+++ b/usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml    2021-02-03 19:28:20.090743051 -0500
@@ -47,16 +47,18 @@
         echo $node_lookup | $(get_python) -c "
         import json
         import sys
+        import ast
         input = sys.stdin.readline() or '{}'
-        cnt = json.loads(input)
+        cnt = ast.literal_eval(input)
         print(json.dumps(cnt.get('${node_id}', {})))
         " > /etc/puppet/hieradata/${node_id}.json
         # handle upper case node id LP#1816652
         echo $node_lookup | $(get_python) -c "
         import json
         import sys
+        import ast
         input = sys.stdin.readline() or '{}'
-        cnt = json.loads(input)
+        cnt = ast.literal_eval(input)
         print(json.dumps(cnt.get('${node_id_upper}', {})))
         " > /etc/puppet/hieradata/${node_id_upper}.json

Comment 1 David Vallee Delisle 2021-02-06 00:24:30 UTC
Kellen,

Thanks for flagging this and to providing the patch.

[1] Reproduced this

I've submitted this patch [a] upstream and I will backport it to train to address this bug.

DVD

[a] https://review.opendev.org/774323

[1]
~~~
[2021-02-05 21:48:08,975] (heat-config) [INFO] deploy_action=CREATE\
[2021-02-05 21:48:08,975] (heat-config) [INFO] deploy_stack_id=overcloud-Compute-ylhk4xulanbt-0-iq4yrmavu2fr-ComputeExtraConfigPre-h3sri74r3k2n-NodeSpecificDeployment-ugiv4nr3g27o/697ed555-b51c-425d-ba0b-32ef1e149d87\
[2021-02-05 21:48:08,976] (heat-config) [INFO] deploy_resource_name=TripleOSoftwareDeployment\
[2021-02-05 21:48:08,976] (heat-config) [INFO] deploy_signal_transport=NO_SIGNAL\
[2021-02-05 21:48:08,976] (heat-config) [DEBUG] Running /var/lib/heat-config/heat-config-script/67fa688f-c518-49af-9835-8d8640948c05\
[2021-02-05 21:48:09,147] (heat-config) [INFO] b\\'\\'\
[2021-02-05 21:48:09,147] (heat-config) [DEBUG] b\\'Traceback (most recent call last):\\\
  File \"<string>\", line 5, in <module>\\\
  File \"/usr/lib64/python3.6/json/__init__.py\", line 354, in loads\\\
    return _default_decoder.decode(s)\\\
  File \"/usr/lib64/python3.6/json/decoder.py\", line 339, in decode\\\
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\\
  File \"/usr/lib64/python3.6/json/decoder.py\", line 355, in raw_decode\\\
    obj, end = self.scan_once(s, idx)\\\
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\\\
Traceback (most recent call last):\\\
  File \"<string>\", line 5, in <module>\\\
  File \"/usr/lib64/python3.6/json/__init__.py\", line 354, in loads\\\
    return _default_decoder.decode(s)\\\
  File \"/usr/lib64/python3.6/json/decoder.py\", line 339, in decode\\\
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\\
  File \"/usr/lib64/python3.6/json/decoder.py\", line 355, in raw_decode\\\
    obj, end = self.scan_once(s, idx)\\\
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\\\
\\'\
~~~

Comment 4 Rabi Mishra 2021-03-22 06:48:42 UTC
*** Bug 1941097 has been marked as a duplicate of this bug. ***

Comment 24 errata-xmlrpc 2021-05-26 13:50:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.6 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2097