Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1730921

Summary: Failed to handle action completion [error=Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}], wf=tripleo.storage.v1.ceph-install, task=parse_node_data_lookup, action=std.noop]
Product: Red Hat OpenStack Reporter: Siggy Sigwald <ssigwald>
Component: openstack-tripleo-heat-templatesAssignee: John Fulton <johfulto>
Status: CLOSED NOTABUG QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: emacchi, johfulto, mburns, rcarrier, shdunne, tenobreg, tshefi
Target Milestone: ---Flags: tshefi: automate_bug-
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 13:59:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Siggy Sigwald 2019-07-17 21:41:54 UTC
Description of problem:

openstack stack failures list overcloud --long                                               overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:
  resource_type: OS::TripleO::WorkflowSteps
  physical_resource_id: 52bb4fde-9bf6-49b4-b64f-3bae8adac2a7
  status: CREATE_FAILED
  status_reason: |
    resources.WorkflowTasks_Step2_Execution: Failure caused by error in tasks: ceph_base_ansible_workflow

      ceph_base_ansible_workflow [task_ex_id=2ba99c86-1d95-4ce0-896a-7b152c091f5b] -> Failed to handle action completion [error=Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}], wf=tripleo.storage.v1.ceph-install, task=parse_node_data_lookup, action=std.noop]:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 110, in _on_action_complete
        task.on_action_complete(action_ex)
      File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
        result = f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 324, in on_action_complete
        self.complete(state, state_info)
      File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
        result = f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 191, in complete
        data_flow.publish_variables(self.task_ex, self.task_spec)
      File "/usr/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 215, in publish_variables
        task_ex.published = expr.evaluate_recursively(branch_vars, expr_ctx)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 100, in evaluate_recursively
        data[key] = _evaluate_item(data[key], context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
        return evaluate(item, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
        return evaluator.evaluate(expression, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 119, in evaluate
        cls).evaluate(trim_expr, data_context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 73, in evaluate
        ", data=%s]" % (expression, str(e), data_context)
    YaqlEvaluationException: Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}]

        [wf_ex_id=c68bb3e0-6ae4-42d0-8651-0a3a93e31fb4, idx=0]: Failed to handle action completion [error=Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}], wf=tripleo.storage.v1.ceph-install, task=parse_node_data_lookup, action=std.noop]:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 110, in _on_action_complete
        task.on_action_complete(action_ex)
      File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
        result = f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 324, in on_action_complete
        self.complete(state, state_info)
      File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
        result = f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 191, in complete
        data_flow.publish_variables(self.task_ex, self.task_spec)
      File "/usr/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 215, in publish_variables
        task_ex.published = expr.evaluate_recursively(branch_vars, expr_ctx)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 100, in evaluate_recursively
        data[key] = _evaluate_item(data[key], context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
        return evaluate(item, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
        return evaluator.evaluate(expression, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 119, in evaluate
        cls).evaluate(trim_expr, data_context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 73, in evaluate
        ", data=%s]" % (expression, str(e), data_context)
    YaqlEvaluationException: Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}]

The error looks similar to bug 1570050 in RHOSP13 

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-8.3.1-18.el7ost.noarch

Comment 6 John Fulton 2019-08-06 13:46:35 UTC
I see you are using node specific overrides as described in the following document:

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/deploying_an_overcloud_with_containerized_red_hat_ceph/configuring_ceph_storage_cluster_settings#map_disk_layout_non-homogen_ceph

As per the same document the JSON which is embedded in your YAML needs to be valid json and you can verify this with jq. In this scenario the ceph1-config.yaml is not valid JSON and that's why you're having this issue. 

Here's how I determined this as documented above.

1. remove the yaml from the file

[fultonj@skagra bz1730921]$ diff -u ceph1-config.yaml.bak  ceph1-config.yaml
--- ceph1-config.yaml.bak       2019-08-06 09:25:49.643880008 -0400
+++ ceph1-config.yaml   2019-08-06 09:26:24.044136893 -0400
@@ -1,16 +1,3 @@
-parameter_defaults:
-  CephConfigOverrides:
-    journal_size: 102400
-    max_open_files: 131072
-
-  CephAnsibleDisksConfig:
-    osd_scenario: non-collocated
-
-  CephPoolDefaultPgNum: 64
-  ManilaCephFSDataPoolPGNum: 64
-  ManilaCephFSMetadataPoolPGNum: 64 
-
-  NodeDataLookup: |
     {
     "39373638-3935-584d-5139-31383030574a": {
       "devices": [
[fultonj@skagra bz1730921]$ 


2. use jq to parse the JSON

[fultonj@skagra bz1730921]$ cat ceph1-config.yaml | jq 
parse error: Invalid numeric literal at line 24, column 32
[fultonj@skagra bz1730921]$ 

3. Change the " on line 24, column 32 to a ' and try again

[fultonj@skagra bz1730921]$ cat ceph1-config.yaml | jq 
{
  "39373638-3935-584d-5139-31383030574a": {
    "devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d557c2-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d557c3-lun-0"
    ],
    "dedicated_devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d557c4-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d557c4-lun-0"
    ]
  },
  "39373638-3935-584d-5139-313830305748": {
    "devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d65b82-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d65b83-lun-0"
    ],
    "dedicated_devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d65b84-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d65b84-lun-0"
    ]
  },
  "39373638-3935-584d-5139-313830305744": {
    "devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d654b2-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d654b3-lun-0"
    ],
    "dedicated_devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d654b4-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d654b4-lun-0"
    ]
  }
}
[fultonj@skagra bz1730921]$ 

4. Add the YAML which was removed in step1 back to the file

You should now be able to redeploy using the new ceph1-config.yaml

Comment 7 John Fulton 2019-08-06 13:59:23 UTC
Because this was a syntax issue for the program input (" vs ') I'm closing this as not a bug.

Comment 8 Tzach Shefi 2019-08-26 14:11:12 UTC
Closed NOTABUG, nothing to test/automate per close loop process.