Bug 1663903
| Summary: | [UPGRADE] Upgrade error message is not informative "ValueError: No JSON object could be decoded" | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Raviv Bar-Tal <rbartal> | ||||
| Component: | python-tripleoclient | Assignee: | mathieu bultel <mbultel> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Raviv Bar-Tal <rbartal> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 14.0 (Rocky) | CC: | ccamacho, dbecker, emacchi, hbrock, jslagle, lbezdick, mbultel, mburns, morazi, nmanos, sgolovat, tvignaud, vpickard | ||||
| Target Milestone: | z3 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 14.0 (Rocky) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | python-tripleoclient-10.6.1-0.20190303235860.ab72340.el7ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-07-02 20:08:26 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1661804 | ||||||
| Attachments: |
|
||||||
This issue appears to be a high level orchestration error in tripleo/ansible.
The upgrade logs point to a failure in opendaylight-ovs.yaml:
u'TASK [Delete OVS groups and ports] *********************************************',
u'Wednesday 09 January 2019 18:27:29 -0500 (0:00:00.716) 0:00:28.139 ***** ',
u" [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather"]
2019-01-09 22:26:49.108 271452 ERROR openstack [-] No JSON object could be decoded: ValueError: No JSON object could be decoded
2019-01-09 22:26:49.108 271452 ERROR openstack Traceback (most recent call last):
2019-01-09 22:26:49.108 271452 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
2019-01-09 22:26:49.108 271452 ERROR openstack result = cmd.run(parsed_args)
2019-01-09 22:26:49.108 271452 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
2019-01-09 22:26:49.108 271452 ERROR openstack super(Command, self).run(parsed_args)
2019-01-09 22:26:49.108 271452 ERROR openstack File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
*** But ***
Additional debug of mistral logs indicate that the opendaylight_ovs.yaml task "Delete OVS groups and ports" actually completed, indicating a failure in triple/ansible orchestration:
2019-01-10 09:55:04,961 p=23729 u=mistral | TASK [Delete OVS groups and ports] *********************************************
2019-01-10 09:55:04,961 p=23729 u=mistral | Thursday 10 January 2019 09:55:04 -0500 (0:00:00.586) 0:00:17.928 ******
2019-01-10 09:55:05,368 p=23729 u=mistral | [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather
than running sudo
2019-01-10 09:55:05,369 p=23729 u=mistral | changed: [controller-2] => {"changed": true, "cmd": "sudo ovs-ofctl -O Openflow13 del-groups br-int; for tun_port in $(sudo ovs-vs
ctl list-ports br-int | grep tun); do sudo ovs-vsctl del-port br-int $tun_port; done", "delta": "0:00:00.075392", "end": "2019-01-10 14:55:05.338039", "rc": 0, "start": "2019-
01-10 14:55:05.262647", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2019-01-10 09:55:05,419 p=23729 u=mistral | changed: [controller-1] => {"changed": true, "cmd": "sudo ovs-ofctl -O Openflow13 del-groups br-int; for tun_port in $(sudo ovs-vs
ctl list-ports br-int | grep tun); do sudo ovs-vsctl del-port br-int $tun_port; done", "delta": "0:00:00.071778", "end": "2019-01-10 14:55:05.395681", "rc": 0, "start": "2019-
01-10 14:55:05.323903", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2019-01-10 09:55:05,508 p=23729 u=mistral | changed: [controller-0] => {"changed": true, "cmd": "sudo ovs-ofctl -O Openflow13 del-groups br-int; for tun_port in $(sudo ovs-vs
ctl list-ports br-int | grep tun); do sudo ovs-vsctl del-port br-int $tun_port; done", "delta": "0:00:00.069066", "end": "2019-01-10 14:55:05.476487", "rc": 0, "start": "2019-
01-10 14:55:05.407421", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2019-01-10 09:55:05,596 p=23729 u=mistral | TASK [Get docker Rabbitmq image] ***********************************************
2019-01-10 09:55:05,596 p=23729 u=mistral | Thursday 10 January 2019 09:55:05 -0500 (0:00:00.635) 0:00:18.563 ******
2019-01-10 09:55:05,693 p=23729 u=mistral | ok: [controller-2] => {"ansible_facts": {"rabbitmq_docker_image_latest": "192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest"
}, "changed": false}
2019-01-10 09:55:05,779 p=23729 u=mistral | ok: [controller-1] => {"ansible_facts": {"rabbitmq_docker_image_latest": "192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest"
}, "changed": false}
2019-01-10 09:55:05,844 p=23729 u=mistral | ok: [controller-0] => {"ansible_facts": {"rabbitmq_docker_image_latest": "192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest"
}, "changed": false}
2019-01-10 09:55:05,923 p=23729 u=mistral | TASK [Check for Rabbitmq Kolla configuration] **********************************
Created attachment 1519888 [details]
mistral logs
The patch attached catch the --debug option and put the verbosity of ansible to -vvv Pending the backport to stable/rocky *** Bug 1662575 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1672 |
Description of problem: Upgrade from osp13 to osp14is failing with this error message: "ValueError: No JSON object could be decoded" This error message does not give any relevant information. When running the Upgrade command with the --debug flag we get more informative Error message: "fatal: [controller-0]: FAILED! => {"changed": false, "error": "Error: unable to get cib\\nError: unable to get cib\\n", "msg": "Failed, to set the resource rabbitmq-bundle to the state show", "output": "", "rc": 1}" We should give this full message in case of failure, As the upgrade procedure takes a lot of time and we do not want rerun it with the --debug flag every time Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. install osp13 (maybe with odl) 2. upgrade to osp14 3. Actual results: the error message is not informative Expected results: informative error message Additional info: The error message was found when upgrading osp13 with odl, please see BZ#1662575