Bug 1662745

Summary: [UPGRADE] upgrade prepare command fails with error " No JSON object could be decoded"
Product: Red Hat OpenStack Reporter: Raviv Bar-Tal <rbartal>
Component: rhosp-directorAssignee: RHOS Maint <rhos-maint>
Status: CLOSED DUPLICATE QA Contact: Noam Manos <nmanos>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 14.0 (Rocky)CC: dbecker, mburns, morazi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-01 13:34:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Raviv Bar-Tal 2019-01-01 12:37:05 UTC
Description of problem:
Upgrade from osp13 to osp14 on ODL env fail running the command:
openstack overcloud  upgrade prepare --debug \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/config_heat.yaml \
-e /home/stack/virt/nodes_data.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml \
-e /home/stack/containers-prepare-parameter.yaml \
-e /home/stack/virt/log_levels/netvirt_genius_debug.yaml \
--log-file overcloud_update_prepare.log


the error message and traceback are:
WARNING: Following parameter(s) are defined but not used in plan. Could be possible that parameter is valid but currently not used.
  ComputeExtraConfig.nova::compute::consecutive_build_service_disable_threshold
  SwiftFetchDirPutTempurl
  CephAnsiblePlaybookVerbosity
  NeutronTunnelTypes
  NeutronExternalNetworkBridge
  SwiftFetchDirGetTempurl
2018-12-31 05:48:47.433 924885 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.ESC[00m
2018-12-31 09:43:31.114 924885 ERROR openstack [-] No JSON object could be decoded: ValueError: No JSON object could be decoded
2018-12-31 09:43:31.114 924885 ERROR openstack Traceback (most recent call last):
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
2018-12-31 09:43:31.114 924885 ERROR openstack     result = cmd.run(parsed_args)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
2018-12-31 09:43:31.114 924885 ERROR openstack     super(Command, self).run(parsed_args)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2018-12-31 09:43:31.114 924885 ERROR openstack     return super(Command, self).run(parsed_args)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2018-12-31 09:43:31.114 924885 ERROR openstack     return_code = self.take_action(parsed_args) or 0
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 68, in take_action
2018-12-31 09:43:31.114 924885 ERROR openstack     package_update.update(clients, container=stack_name)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 38, in update
2018-12-31 09:43:31.114 924885 ERROR openstack     for payload in base.wait_for_messages(workflow_client, ws, execution):
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages
2018-12-31 09:43:31.114 924885 ERROR openstack     for payload in websocket.wait_for_messages(timeout=timeout):
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages
2018-12-31 09:43:31.114 924885 ERROR openstack     message = self.recv()
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv
2018-12-31 09:43:31.114 924885 ERROR openstack     return json.loads(self._ws.recv())
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2018-12-31 09:43:31.114 924885 ERROR openstack     return _default_decoder.decode(s)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2018-12-31 09:43:31.114 924885 ERROR openstack     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2018-12-31 09:43:31.114 924885 ERROR openstack     raise ValueError("No JSON object could be decoded")
2018-12-31 09:43:31.114 924885 ERROR openstack ValueError: No JSON object could be decoded
2018-12-31 09:43:31.114 924885 ERROR openstack ESC[00m
2018-12-31 09:43:31.114 924885 ERROR openstack ESC[00m
2018-12-31 09:43:31.126 924885 ERROR osc_lib.shell [-] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 68, in take_action
    package_update.update(clients, container=stack_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 38, in update
    for payload in base.wait_for_messages(workflow_client, ws, execution):
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages
    for payload in websocket.wait_for_messages(timeout=timeout):
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages
    message = self.recv()
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv
    return json.loads(self._ws.recv())
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
ESC[00m
2018-12-31 09:43:31.127 924885 INFO osc_lib.shell [-] END return value: 1ESC[00m
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. install osp13 with ODL
2. start the upgrade procedure
3. the upgrade prepare command will fail

Actual results:


Expected results:


Additional info: