Bug 1730921 - Failed to handle action completion [error=Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}], wf=tripleo.storage.v1.ceph-install, task=parse_node_data_lookup, action=std.noop]
Summary: Failed to handle action completion [error=Can not evaluate YAQL expression [e...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: ---
: ---
Assignee: John Fulton
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-17 21:41 UTC by Siggy Sigwald
Modified: 2019-09-20 16:12 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-06 13:59:23 UTC
Target Upstream Version:
tshefi: automate_bug-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1570050 0 urgent CLOSED tripleo.storage.v1.ceph-install fails with external ceph on yaql $.ansible_output.get('plays')[0].get('tasks')[0].get('h... 2021-02-22 00:41:40 UTC

Description Siggy Sigwald 2019-07-17 21:41:54 UTC
Description of problem:

openstack stack failures list overcloud --long                                               overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:
  resource_type: OS::TripleO::WorkflowSteps
  physical_resource_id: 52bb4fde-9bf6-49b4-b64f-3bae8adac2a7
  status: CREATE_FAILED
  status_reason: |
    resources.WorkflowTasks_Step2_Execution: Failure caused by error in tasks: ceph_base_ansible_workflow

      ceph_base_ansible_workflow [task_ex_id=2ba99c86-1d95-4ce0-896a-7b152c091f5b] -> Failed to handle action completion [error=Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}], wf=tripleo.storage.v1.ceph-install, task=parse_node_data_lookup, action=std.noop]:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 110, in _on_action_complete
        task.on_action_complete(action_ex)
      File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
        result = f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 324, in on_action_complete
        self.complete(state, state_info)
      File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
        result = f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 191, in complete
        data_flow.publish_variables(self.task_ex, self.task_spec)
      File "/usr/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 215, in publish_variables
        task_ex.published = expr.evaluate_recursively(branch_vars, expr_ctx)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 100, in evaluate_recursively
        data[key] = _evaluate_item(data[key], context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
        return evaluate(item, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
        return evaluator.evaluate(expression, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 119, in evaluate
        cls).evaluate(trim_expr, data_context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 73, in evaluate
        ", data=%s]" % (expression, str(e), data_context)
    YaqlEvaluationException: Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}]

        [wf_ex_id=c68bb3e0-6ae4-42d0-8651-0a3a93e31fb4, idx=0]: Failed to handle action completion [error=Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}], wf=tripleo.storage.v1.ceph-install, task=parse_node_data_lookup, action=std.noop]:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 110, in _on_action_complete
        task.on_action_complete(action_ex)
      File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
        result = f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 324, in on_action_complete
        self.complete(state, state_info)
      File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
        result = f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 191, in complete
        data_flow.publish_variables(self.task_ex, self.task_spec)
      File "/usr/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 215, in publish_variables
        task_ex.published = expr.evaluate_recursively(branch_vars, expr_ctx)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 100, in evaluate_recursively
        data[key] = _evaluate_item(data[key], context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
        return evaluate(item, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
        return evaluator.evaluate(expression, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 119, in evaluate
        cls).evaluate(trim_expr, data_context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 73, in evaluate
        ", data=%s]" % (expression, str(e), data_context)
    YaqlEvaluationException: Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=No JSON object could be decoded, data={}]

The error looks similar to bug 1570050 in RHOSP13 

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-8.3.1-18.el7ost.noarch

Comment 6 John Fulton 2019-08-06 13:46:35 UTC
I see you are using node specific overrides as described in the following document:

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/deploying_an_overcloud_with_containerized_red_hat_ceph/configuring_ceph_storage_cluster_settings#map_disk_layout_non-homogen_ceph

As per the same document the JSON which is embedded in your YAML needs to be valid json and you can verify this with jq. In this scenario the ceph1-config.yaml is not valid JSON and that's why you're having this issue. 

Here's how I determined this as documented above.

1. remove the yaml from the file

[fultonj@skagra bz1730921]$ diff -u ceph1-config.yaml.bak  ceph1-config.yaml
--- ceph1-config.yaml.bak       2019-08-06 09:25:49.643880008 -0400
+++ ceph1-config.yaml   2019-08-06 09:26:24.044136893 -0400
@@ -1,16 +1,3 @@
-parameter_defaults:
-  CephConfigOverrides:
-    journal_size: 102400
-    max_open_files: 131072
-
-  CephAnsibleDisksConfig:
-    osd_scenario: non-collocated
-
-  CephPoolDefaultPgNum: 64
-  ManilaCephFSDataPoolPGNum: 64
-  ManilaCephFSMetadataPoolPGNum: 64 
-
-  NodeDataLookup: |
     {
     "39373638-3935-584d-5139-31383030574a": {
       "devices": [
[fultonj@skagra bz1730921]$ 


2. use jq to parse the JSON

[fultonj@skagra bz1730921]$ cat ceph1-config.yaml | jq 
parse error: Invalid numeric literal at line 24, column 32
[fultonj@skagra bz1730921]$ 

3. Change the " on line 24, column 32 to a ' and try again

[fultonj@skagra bz1730921]$ cat ceph1-config.yaml | jq 
{
  "39373638-3935-584d-5139-31383030574a": {
    "devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d557c2-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d557c3-lun-0"
    ],
    "dedicated_devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d557c4-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d557c4-lun-0"
    ]
  },
  "39373638-3935-584d-5139-313830305748": {
    "devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d65b82-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d65b83-lun-0"
    ],
    "dedicated_devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d65b84-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d65b84-lun-0"
    ]
  },
  "39373638-3935-584d-5139-313830305744": {
    "devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d654b2-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d654b3-lun-0"
    ],
    "dedicated_devices": [
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d654b4-lun-0",
      "/dev/disk/by-path/pci-0000:5c:00.0-sas-0x31402ec012d654b4-lun-0"
    ]
  }
}
[fultonj@skagra bz1730921]$ 

4. Add the YAML which was removed in step1 back to the file

You should now be able to redeploy using the new ceph1-config.yaml

Comment 7 John Fulton 2019-08-06 13:59:23 UTC
Because this was a syntax issue for the program input (" vs ') I'm closing this as not a bug.

Comment 8 Tzach Shefi 2019-08-26 14:11:12 UTC
Closed NOTABUG, nothing to test/automate per close loop process.


Note You need to log in before you can comment on or make changes to this bug.