Bug 1662575

Summary:

OSP13 -> OSP14 major upgrade fail: No JSON object could be decoded

Product:

Red Hat OpenStack

Reporter:

Noam Manos <nmanos>

Component:

openstack-tripleo

Assignee:

mathieu bultel <mbultel>

Status:

CLOSED DUPLICATE

QA Contact:

Arik Chernetsky <achernet>

Severity:

high

Docs Contact:

Priority:

medium

Version:

14.0 (Rocky)

CC:

emacchi, lbezdick, mbultel, mburns, nmanos, rbartal, sgolovat, yprokule

Target Milestone:

---

Keywords:

Triaged, ZStream

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2019-05-03 09:14:22 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1661804

Attachments:

Description	Flags
OSP14 Upgrade logs and templates	none
mistral logs directory	none
upgrade run log (debug)	none

Description Noam Manos 2018-12-30 09:38:33 UTC

Created attachment 1517440 [details]
OSP14 Upgrade logs and templates

Description of problem:
OSP13 -> OSP14 major upgrade failed after 4 hours, during "openstack overcloud upgrade prepare":

2018-12-27 10:22:38.322 105231 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.
2018-12-27 14:17:38.111 105231 ERROR openstack [-] No JSON object could be decoded
2018-12-27 14:17:38.125 105231 INFO osc_lib.shell [-] END return value: 1

Version-Release number of selected component (if applicable):
OSP13 2018-12-13.4  -->  OSP14 2018-12-20.4

How reproducible:
Sometimes

Steps to Reproduce:
1. Install OSP13 + ODL
2. Follow OSP14 Upgrade guide.


Actual results:

. stackrc

sudo yum localinstall -y http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/rhos-release-latest.noarch.rpm
sudo rhos-release -x
sudo rhos-release -P 14 -p 2018-12-20.4
sudo yum update -y python-tripleoclient

openstack tripleo container image prepare default --local-push-destination --output-env-file /home/stack/containers-prepare-parameter.yaml

openstack undercloud upgrade -y |& tee osp14_undercloud_upgrade.log


for i in $( openstack server list -f value -c Networks | grep -Eo '[[:digit:].]+' )
do
    ssh -o StrictHostKeyChecking=no heat-admin@${i} \
        "sudo yum localinstall -y http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/rhos-release-latest.noarch.rpm; \
        sudo rhos-release -x; \
        sudo rhos-release -P 14 -p 2018-12-20.4"
done

python /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py \
  --tht-dir /usr/share/openstack-tripleo-heat-templates \
  --role-name Controller \
  --discard-comments yes \
  --template /home/stack/virt/network/three-nics-vlans/controller.yaml

python /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py \
  --tht-dir /usr/share/openstack-tripleo-heat-templates \
  --role-name Compute \
  --discard-comments yes \
  --template /home/stack/virt/network/three-nics-vlans/compute.yaml


(undercloud) [stack@undercloud-0 ~]$ openstack overcloud upgrade prepare --stack overcloud \
>   --timeout 100 \
>   --templates /usr/share/openstack-tripleo-heat-templates \
>   --stack overcloud \
>   --libvirt-type kvm \
>   --ntp-server clock.redhat.com \
>   -e /home/stack/virt/config_lvm.yaml \
>   -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
>   -e /home/stack/virt/network/network-environment.yaml \
>   -e /home/stack/virt/inject-trust-anchor.yaml \
>   -e /home/stack/virt/hostnames.yml \
>   -e /home/stack/virt/debug.yaml \
>   -e /home/stack/virt/config_heat.yaml \
>   -e /home/stack/virt/nodes_data.yaml \
>   -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml \
>   -e /home/stack/containers-prepare-parameter.yaml \
>   -e /home/stack/virt/log_levels/netvirt_genius_debug.yaml |& tee osp14_upgrade_prepare.log

2018-12-27 10:17:37.255 105231 INFO osc_lib.shell [-] command: overcloud upgrade prepare -> tripleoclient.v1.overcloud_upgrade.UpgradePrepare (auth=True)
2018-12-27 10:17:37.257 105231 INFO osc_lib.clientmanager [-] Using auth plugin: password
2018-12-27 10:17:37.257 105231 DEBUG osc_lib.clientmanager [-] Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'https://192.168.24.2:13000/', 'password': '***', 'project_domain_name': 'Default'} setup_auth /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:157
2018-12-27 10:17:37.261 105231 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:201
2018-12-27 10:17:43.477 105231 INFO tripleoclient.v1.overcloud_upgrade.MajorUpgradePrepare [-] Stack found, will be doing a stack update
Removing the current plan files
Uploading new plan files
Temporary Swift GET/PUT URL parameters have successfully been updated.
Temporary Swift GET/PUT URL parameters have successfully been updated.
The backup of the ceph-ansible fetch directory did not need to be renamed
Plan updated.
Processing templates in the directory /tmp/tripleoclient-Yso3zz/tripleo-heat-templates
WARNING: Following parameter(s) are deprecated and still defined. Deprecated parameters will be removed soon!
  OvercloudControlFlavor
WARNING: Following parameter(s) are defined but not used in plan. Could be possible that parameter is valid but currently not used.
  ComputeExtraConfig.nova::compute::consecutive_build_service_disable_threshold
  SwiftFetchDirPutTempurl
  CephAnsiblePlaybookVerbosity
  NeutronTunnelTypes
  NeutronExternalNetworkBridge
  SwiftFetchDirGetTempurl
2018-12-27 10:22:38.322 105231 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.
2018-12-27 14:17:38.111 105231 ERROR openstack [-] No JSON object could be decoded
2018-12-27 14:17:38.125 105231 INFO osc_lib.shell [-] END return value: 1
(undercloud) [stack@undercloud-0 ~]$


Expected results:
Upgrade to pass.

Additional info:

There's an U/s bug on similar issue: https://bugs.launchpad.net/tripleo/+bug/1690850

it was closed as not reproduced. in any case that bug mentions a zero file size for "/etc/os-net-config/config.json", which is not seen on my env:


(undercloud) [stack@undercloud-0 ~]$ cat /etc/os-net-config/config.json                                                                                                                                             
 {"network_config": [{"addresses": [{"ip_netmask": "192.168.24.1/24"}], "dns_servers": [], "members": [{"mtu": 1500, "name": "eth0", "primary": true, "type": "interface"}], "name": "br-ctlplane", "ovs_extra": ["br-set-external-id br-ctlplane bridge-id br-ctlplane"], "routes": [], "type": "ovs_bridge", "use_dhcp": false}]}

Comment 1 Noam Manos 2019-01-01 07:38:08 UTC

Trying to run OSP14 upgrade prepare again, has failed after 4 hours on the same error:

openstack overcloud upgrade prepare --debug \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
-e /home/stack/virt/config_lvm.yaml \ 
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \   
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/config_heat.yaml \
-e /home/stack/virt/nodes_data.yaml \ 
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml \
-e /home/stack/containers-prepare-parameter.yaml \
-e /home/stack/virt/log_levels/netvirt_genius_debug.yaml \
--log-file overcloud_update_prepare.log


(undercloud) [stack@undercloud-0 ~]$ ./overcloud_upgrade_prepare.sh |& tee prepare.log
START with options: [u'overcloud', u'upgrade', u'prepare', u'--debug', u'--templates', u'/usr/share/openstack-tripleo-heat-templates', u'--stack', u'overcloud', u'-e', u'/home/stack/virt/config_lvm.yaml', u'-e', u'/usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml', u'-e', u'/home/stack/virt/network/network-environment.yaml', u'-e', u'/home/stack/virt/inject-trust-anchor.yaml', u'-e', u'/home/stack/virt/hostnames.yml', u'-e', u'/home/stack/virt/debug.yaml', u'-e', u'/home/stack/virt/config_heat.yaml', u'-e', u'/home/stack/virt/nodes_data.yaml', u'--environment-file', u'/usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml', u'-e', u'/home/stack/containers-prepare-parameter.yaml', u'-e', u'/home/stack/virt/log_levels/netvirt_genius_debug.yaml', u'--log-file', u'overcloud_update_prepare.log']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_type='password', auth_url='https://192.168.24.2:13000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=u'overcloud_update_prepare.log', openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.46', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', u'status': u'active', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u'interface': None, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloud_update_prepare.log', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, u'interface': None, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': u'1', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', u'baremetal_status_code_retries': 5, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, u'interface': None, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': u'1', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', u'baremetal_status_code_retries': 5, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'username': 'admin', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'block_storage_api_version': u'2', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', u'container_api_version': u'1', 'log_file': u'overcloud_update_prepare.log', u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
baremetal API version 1.46, cmd group openstack.baremetal.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
metric API version 1, cmd group openstack.metric.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
orchestration API version 1, cmd group openstack.orchestration.v1
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
messaging API version 2, cmd group openstack.messaging.v2
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
dns API version 2, cmd group openstack.dns.v2
database API version 1, cmd group openstack.database.v1
container_infra API version 1, cmd group openstack.container_infra.v1
key_manager API version 1, cmd group openstack.key_manager.v1
alarming API version 2, cmd group openstack.alarming.v2
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
data_processing API version 1.1, cmd group openstack.data_processing.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', u'status': u'active', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u'interface': None, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloud_update_prepare.log', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}}
2018-12-31 05:43:30.443 924885 INFO osc_lib.shell [-] command: overcloud upgrade prepare -> tripleoclient.v1.overcloud_upgrade.UpgradePrepare (auth=True)
2018-12-31 05:43:30.446 924885 INFO osc_lib.clientmanager [-] Using auth plugin: password
2018-12-31 05:43:30.447 924885 DEBUG osc_lib.clientmanager [-] Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'https://192.168.24.2:13000/', 'password': '***', 'project_domain_name': 'Default'} setup_auth /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:157
2018-12-31 05:43:30.454 924885 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:201
2018-12-31 05:43:37.327 924885 INFO tripleoclient.v1.overcloud_upgrade.MajorUpgradePrepare [-] Stack found, will be doing a stack update
Removing the current plan files 
Uploading new plan files
Temporary Swift GET/PUT URL parameters have successfully been updated.
Temporary Swift GET/PUT URL parameters have successfully been updated.
The backup of the ceph-ansible fetch directory did not need to be renamed
Plan updated.
Processing templates in the directory /tmp/tripleoclient-XLFG21/tripleo-heat-templates
WARNING: Following parameter(s) are deprecated and still defined. Deprecated parameters will be removed soon!
  OvercloudControlFlavor
WARNING: Following parameter(s) are defined but not used in plan. Could be possible that parameter is valid but currently not used.
  ComputeExtraConfig.nova::compute::consecutive_build_service_disable_threshold
  SwiftFetchDirPutTempurl
  CephAnsiblePlaybookVerbosity
  NeutronTunnelTypes
  NeutronExternalNetworkBridge
  SwiftFetchDirGetTempurl
2018-12-31 05:48:47.433 924885 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.
2018-12-31 09:43:31.114 924885 ERROR openstack [-] No JSON object could be decoded: ValueError: No JSON object could be decoded
2018-12-31 09:43:31.114 924885 ERROR openstack Traceback (most recent call last):
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
2018-12-31 09:43:31.114 924885 ERROR openstack     result = cmd.run(parsed_args)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
2018-12-31 09:43:31.114 924885 ERROR openstack     super(Command, self).run(parsed_args)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2018-12-31 09:43:31.114 924885 ERROR openstack     return super(Command, self).run(parsed_args)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2018-12-31 09:43:31.114 924885 ERROR openstack     return_code = self.take_action(parsed_args) or 0
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 68, in take_action
2018-12-31 09:43:31.114 924885 ERROR openstack     package_update.update(clients, container=stack_name)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 38, in update
2018-12-31 09:43:31.114 924885 ERROR openstack     for payload in base.wait_for_messages(workflow_client, ws, execution):
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages
2018-12-31 09:43:31.114 924885 ERROR openstack     for payload in websocket.wait_for_messages(timeout=timeout):
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages
2018-12-31 09:43:31.114 924885 ERROR openstack     message = self.recv()
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv
2018-12-31 09:43:31.114 924885 ERROR openstack     return json.loads(self._ws.recv())
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2018-12-31 09:43:31.114 924885 ERROR openstack     return _default_decoder.decode(s)
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2018-12-31 09:43:31.114 924885 ERROR openstack     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2018-12-31 09:43:31.114 924885 ERROR openstack   File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2018-12-31 09:43:31.114 924885 ERROR openstack     raise ValueError("No JSON object could be decoded")
2018-12-31 09:43:31.114 924885 ERROR openstack ValueError: No JSON object could be decoded
2018-12-31 09:43:31.114 924885 ERROR openstack
2018-12-31 09:43:31.126 924885 ERROR osc_lib.shell [-] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 68, in take_action
    package_update.update(clients, container=stack_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 38, in update
    for payload in base.wait_for_messages(workflow_client, ws, execution):
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages
    for payload in websocket.wait_for_messages(timeout=timeout):
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages
    message = self.recv()
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv
    return json.loads(self._ws.recv())
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s) 
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

2018-12-31 09:43:31.127 924885 INFO osc_lib.shell [-] END return value: 1

Comment 2 Noam Manos 2019-01-01 13:34:29 UTC

*** Bug 1662745 has been marked as a duplicate of this bug. ***

Comment 5 Noam Manos 2019-01-02 09:17:30 UTC

Created attachment 1517892 [details]
mistral logs directory

Comment 6 Noam Manos 2019-01-06 07:28:52 UTC

On another attempt to run overcloud upgrade prepare it has passed, however it failed later on overcloud upgrade run: 
fatal: [controller-2]: FAILED! => ValueError: No JSON object could be decoded

------------

openstack overcloud upgrade prepare --debug \
--stack overcloud   \
--timeout 100   \
--templates /usr/share/openstack-tripleo-heat-templates   \
--stack overcloud   \
--libvirt-type kvm   \
--ntp-server clock.redhat.com   \
-e /home/stack/virt/config_lvm.yaml   \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml   \
-e /home/stack/virt/network/network-environment.yaml   \
-e /home/stack/virt/inject-trust-anchor.yaml   \
-e /home/stack/virt/hostnames.yml   \
-e /home/stack/virt/debug.yaml   \
-e /home/stack/virt/config_heat.yaml   \
-e /home/stack/virt/nodes_data.yaml   \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml   \
-e /home/stack/containers-prepare-parameter.yaml   \
-e /home/stack/virt/log_levels/netvirt_genius_debug.yaml


openstack overcloud external-upgrade run --stack overcloud --tags container_image_prepare 

openstack overcloud upgrade run --debug --stack overcloud --roles Controller --playbook all 
  
fatal: [controller-2]: FAILED! => 
{"changed": true, "cmd": ["systemctl", "is-enabled", "--quiet", "openstack-ceilometer-notification"], "delta": "0:00:00.007509", 
"end": "2019-01-03 13:47:43.306157", "msg": "non-zero return code", "rc": 1, 
"start": "2019-01-03 13:47:43.298648", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}']
2019-01-03 12:47:00.104 1044241 ERROR openstack [-] No JSON object could be decoded: ValueError: No JSON object could be decoded
2019-01-03 12:47:00.104 1044241 ERROR openstack Traceback (most recent call last):
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
2019-01-03 12:47:00.104 1044241 ERROR openstack     result = cmd.run(parsed_args)
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
2019-01-03 12:47:00.104 1044241 ERROR openstack     super(Command, self).run(parsed_args)
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2019-01-03 12:47:00.104 1044241 ERROR openstack     return super(Command, self).run(parsed_args)
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2019-01-03 12:47:00.104 1044241 ERROR openstack     return_code = self.take_action(parsed_args) or 0
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 209, in take_action
2019-01-03 12:47:00.104 1044241 ERROR openstack     skip_tags=skip_tags)
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 1040, in run_update_ansible_action
2019-01-03 12:47:00.104 1044241 ERROR openstack     skip_tags=skip_tags)
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 91, in update_ansible
2019-01-03 12:47:00.104 1044241 ERROR openstack     execution):
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages
2019-01-03 12:47:00.104 1044241 ERROR openstack     for payload in websocket.wait_for_messages(timeout=timeout):
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages
2019-01-03 12:47:00.104 1044241 ERROR openstack     message = self.recv()
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv
2019-01-03 12:47:00.104 1044241 ERROR openstack     return json.loads(self._ws.recv())
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2019-01-03 12:47:00.104 1044241 ERROR openstack     return _default_decoder.decode(s)
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2019-01-03 12:47:00.104 1044241 ERROR openstack     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-01-03 12:47:00.104 1044241 ERROR openstack   File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2019-01-03 12:47:00.104 1044241 ERROR openstack     raise ValueError("No JSON object could be decoded")
2019-01-03 12:47:00.104 1044241 ERROR openstack ValueError: No JSON object could be decoded
2019-01-03 12:47:00.104 1044241 ERROR openstack
2019-01-03 12:47:00.112 1044241 ERROR osc_lib.shell [-] Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
  ret_val = super(OpenStackShell, self).run(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run
  result = self.run_subcommand(remainder)
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand
  ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
  result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
  super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
  return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
  return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 209, in take_action
  skip_tags=skip_tags)
File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 1040, in run_update_ansible_action
  skip_tags=skip_tags)
File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 91, in update_ansible
  execution):
File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages
  for payload in websocket.wait_for_messages(timeout=timeout):
File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages
  message = self.recv()
File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv
  return json.loads(self._ws.recv())
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
  return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
  obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
  raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

2019-01-03 12:47:00.113 1044241 INFO osc_lib.shell [-] END return value: 1

Comment 7 Noam Manos 2019-01-06 07:53:27 UTC

Created attachment 1518745 [details]
upgrade run log (debug)

Apparently "ValueError: No JSON object could be decoded" is just a symptom.

When using "--debug" I see in log:

fatal: [controller-0]: FAILED! => {"changed": false, "error": "Error: unable to get cib\\nError: unable to get cib\\n", "msg": "Failed, to set the resource rabbitmq-bundle to the state show", "output": "", "rc": 1}

I will reattempt an upgrade on a new environment of OSP13.

Since "No JSON object could be decoded" error messages, on two different overcloud commands, is not too informative, we better have a Doc-release flag, to add "--debug" on all overcloud upgrade/update commands, so we can easily see the real root cause, should such error occur.

Comment 10 mathieu bultel 2019-01-17 15:25:12 UTC

Can you try with those patches: https://review.openstack.org/#/c/630986/ & https://review.openstack.org/#/c/631152/
I think that could fix the handling of the error.

NB: for the 1st one you will need to update the workbook manually while applying the patch:
mistral workbook-update tripleo-common/workbooks/package_update.yaml

Comment 11 Noam Manos 2019-01-20 05:52:33 UTC

The environment is not available for ODL update/upgrade tests currently. Changing priority to medium.

Comment 12 Lukas Bezdicka 2019-05-03 09:14:22 UTC


*** This bug has been marked as a duplicate of bug 1663903 ***