Description of problem: Upgrade from osp13 to osp14is failing with this error message: "ValueError: No JSON object could be decoded" This error message does not give any relevant information. When running the Upgrade command with the --debug flag we get more informative Error message: "fatal: [controller-0]: FAILED! => {"changed": false, "error": "Error: unable to get cib\\nError: unable to get cib\\n", "msg": "Failed, to set the resource rabbitmq-bundle to the state show", "output": "", "rc": 1}" We should give this full message in case of failure, As the upgrade procedure takes a lot of time and we do not want rerun it with the --debug flag every time Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. install osp13 (maybe with odl) 2. upgrade to osp14 3. Actual results: the error message is not informative Expected results: informative error message Additional info: The error message was found when upgrading osp13 with odl, please see BZ#1662575
This issue appears to be a high level orchestration error in tripleo/ansible. The upgrade logs point to a failure in opendaylight-ovs.yaml: u'TASK [Delete OVS groups and ports] *********************************************', u'Wednesday 09 January 2019 18:27:29 -0500 (0:00:00.716) 0:00:28.139 ***** ', u" [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather"] 2019-01-09 22:26:49.108 271452 ERROR openstack [-] No JSON object could be decoded: ValueError: No JSON object could be decoded 2019-01-09 22:26:49.108 271452 ERROR openstack Traceback (most recent call last): 2019-01-09 22:26:49.108 271452 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand 2019-01-09 22:26:49.108 271452 ERROR openstack result = cmd.run(parsed_args) 2019-01-09 22:26:49.108 271452 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run 2019-01-09 22:26:49.108 271452 ERROR openstack super(Command, self).run(parsed_args) 2019-01-09 22:26:49.108 271452 ERROR openstack File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run *** But *** Additional debug of mistral logs indicate that the opendaylight_ovs.yaml task "Delete OVS groups and ports" actually completed, indicating a failure in triple/ansible orchestration: 2019-01-10 09:55:04,961 p=23729 u=mistral | TASK [Delete OVS groups and ports] ********************************************* 2019-01-10 09:55:04,961 p=23729 u=mistral | Thursday 10 January 2019 09:55:04 -0500 (0:00:00.586) 0:00:17.928 ****** 2019-01-10 09:55:05,368 p=23729 u=mistral | [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo 2019-01-10 09:55:05,369 p=23729 u=mistral | changed: [controller-2] => {"changed": true, "cmd": "sudo ovs-ofctl -O Openflow13 del-groups br-int; for tun_port in $(sudo ovs-vs ctl list-ports br-int | grep tun); do sudo ovs-vsctl del-port br-int $tun_port; done", "delta": "0:00:00.075392", "end": "2019-01-10 14:55:05.338039", "rc": 0, "start": "2019- 01-10 14:55:05.262647", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-01-10 09:55:05,419 p=23729 u=mistral | changed: [controller-1] => {"changed": true, "cmd": "sudo ovs-ofctl -O Openflow13 del-groups br-int; for tun_port in $(sudo ovs-vs ctl list-ports br-int | grep tun); do sudo ovs-vsctl del-port br-int $tun_port; done", "delta": "0:00:00.071778", "end": "2019-01-10 14:55:05.395681", "rc": 0, "start": "2019- 01-10 14:55:05.323903", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-01-10 09:55:05,508 p=23729 u=mistral | changed: [controller-0] => {"changed": true, "cmd": "sudo ovs-ofctl -O Openflow13 del-groups br-int; for tun_port in $(sudo ovs-vs ctl list-ports br-int | grep tun); do sudo ovs-vsctl del-port br-int $tun_port; done", "delta": "0:00:00.069066", "end": "2019-01-10 14:55:05.476487", "rc": 0, "start": "2019- 01-10 14:55:05.407421", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-01-10 09:55:05,596 p=23729 u=mistral | TASK [Get docker Rabbitmq image] *********************************************** 2019-01-10 09:55:05,596 p=23729 u=mistral | Thursday 10 January 2019 09:55:05 -0500 (0:00:00.635) 0:00:18.563 ****** 2019-01-10 09:55:05,693 p=23729 u=mistral | ok: [controller-2] => {"ansible_facts": {"rabbitmq_docker_image_latest": "192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest" }, "changed": false} 2019-01-10 09:55:05,779 p=23729 u=mistral | ok: [controller-1] => {"ansible_facts": {"rabbitmq_docker_image_latest": "192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest" }, "changed": false} 2019-01-10 09:55:05,844 p=23729 u=mistral | ok: [controller-0] => {"ansible_facts": {"rabbitmq_docker_image_latest": "192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest" }, "changed": false} 2019-01-10 09:55:05,923 p=23729 u=mistral | TASK [Check for Rabbitmq Kolla configuration] **********************************
Created attachment 1519888 [details] mistral logs
The patch attached catch the --debug option and put the verbosity of ansible to -vvv
Pending the backport to stable/rocky
*** Bug 1662575 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1672