Hide Forgot
Created attachment 1517440 [details] OSP14 Upgrade logs and templates Description of problem: OSP13 -> OSP14 major upgrade failed after 4 hours, during "openstack overcloud upgrade prepare": 2018-12-27 10:22:38.322 105231 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout. 2018-12-27 14:17:38.111 105231 ERROR openstack [-] No JSON object could be decoded 2018-12-27 14:17:38.125 105231 INFO osc_lib.shell [-] END return value: 1 Version-Release number of selected component (if applicable): OSP13 2018-12-13.4 --> OSP14 2018-12-20.4 How reproducible: Sometimes Steps to Reproduce: 1. Install OSP13 + ODL 2. Follow OSP14 Upgrade guide. Actual results: . stackrc sudo yum localinstall -y http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/rhos-release-latest.noarch.rpm sudo rhos-release -x sudo rhos-release -P 14 -p 2018-12-20.4 sudo yum update -y python-tripleoclient openstack tripleo container image prepare default --local-push-destination --output-env-file /home/stack/containers-prepare-parameter.yaml openstack undercloud upgrade -y |& tee osp14_undercloud_upgrade.log for i in $( openstack server list -f value -c Networks | grep -Eo '[[:digit:].]+' ) do ssh -o StrictHostKeyChecking=no heat-admin@${i} \ "sudo yum localinstall -y http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/rhos-release-latest.noarch.rpm; \ sudo rhos-release -x; \ sudo rhos-release -P 14 -p 2018-12-20.4" done python /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py \ --tht-dir /usr/share/openstack-tripleo-heat-templates \ --role-name Controller \ --discard-comments yes \ --template /home/stack/virt/network/three-nics-vlans/controller.yaml python /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py \ --tht-dir /usr/share/openstack-tripleo-heat-templates \ --role-name Compute \ --discard-comments yes \ --template /home/stack/virt/network/three-nics-vlans/compute.yaml (undercloud) [stack@undercloud-0 ~]$ openstack overcloud upgrade prepare --stack overcloud \ > --timeout 100 \ > --templates /usr/share/openstack-tripleo-heat-templates \ > --stack overcloud \ > --libvirt-type kvm \ > --ntp-server clock.redhat.com \ > -e /home/stack/virt/config_lvm.yaml \ > -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ > -e /home/stack/virt/network/network-environment.yaml \ > -e /home/stack/virt/inject-trust-anchor.yaml \ > -e /home/stack/virt/hostnames.yml \ > -e /home/stack/virt/debug.yaml \ > -e /home/stack/virt/config_heat.yaml \ > -e /home/stack/virt/nodes_data.yaml \ > -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml \ > -e /home/stack/containers-prepare-parameter.yaml \ > -e /home/stack/virt/log_levels/netvirt_genius_debug.yaml |& tee osp14_upgrade_prepare.log 2018-12-27 10:17:37.255 105231 INFO osc_lib.shell [-] command: overcloud upgrade prepare -> tripleoclient.v1.overcloud_upgrade.UpgradePrepare (auth=True) 2018-12-27 10:17:37.257 105231 INFO osc_lib.clientmanager [-] Using auth plugin: password 2018-12-27 10:17:37.257 105231 DEBUG osc_lib.clientmanager [-] Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'https://192.168.24.2:13000/', 'password': '***', 'project_domain_name': 'Default'} setup_auth /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:157 2018-12-27 10:17:37.261 105231 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:201 2018-12-27 10:17:43.477 105231 INFO tripleoclient.v1.overcloud_upgrade.MajorUpgradePrepare [-] Stack found, will be doing a stack update Removing the current plan files Uploading new plan files Temporary Swift GET/PUT URL parameters have successfully been updated. Temporary Swift GET/PUT URL parameters have successfully been updated. The backup of the ceph-ansible fetch directory did not need to be renamed Plan updated. Processing templates in the directory /tmp/tripleoclient-Yso3zz/tripleo-heat-templates WARNING: Following parameter(s) are deprecated and still defined. Deprecated parameters will be removed soon! OvercloudControlFlavor WARNING: Following parameter(s) are defined but not used in plan. Could be possible that parameter is valid but currently not used. ComputeExtraConfig.nova::compute::consecutive_build_service_disable_threshold SwiftFetchDirPutTempurl CephAnsiblePlaybookVerbosity NeutronTunnelTypes NeutronExternalNetworkBridge SwiftFetchDirGetTempurl 2018-12-27 10:22:38.322 105231 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout. 2018-12-27 14:17:38.111 105231 ERROR openstack [-] No JSON object could be decoded 2018-12-27 14:17:38.125 105231 INFO osc_lib.shell [-] END return value: 1 (undercloud) [stack@undercloud-0 ~]$ Expected results: Upgrade to pass. Additional info: There's an U/s bug on similar issue: https://bugs.launchpad.net/tripleo/+bug/1690850 it was closed as not reproduced. in any case that bug mentions a zero file size for "/etc/os-net-config/config.json", which is not seen on my env: (undercloud) [stack@undercloud-0 ~]$ cat /etc/os-net-config/config.json {"network_config": [{"addresses": [{"ip_netmask": "192.168.24.1/24"}], "dns_servers": [], "members": [{"mtu": 1500, "name": "eth0", "primary": true, "type": "interface"}], "name": "br-ctlplane", "ovs_extra": ["br-set-external-id br-ctlplane bridge-id br-ctlplane"], "routes": [], "type": "ovs_bridge", "use_dhcp": false}]}
Trying to run OSP14 upgrade prepare again, has failed after 4 hours on the same error: openstack overcloud upgrade prepare --debug \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/config_heat.yaml \ -e /home/stack/virt/nodes_data.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml \ -e /home/stack/containers-prepare-parameter.yaml \ -e /home/stack/virt/log_levels/netvirt_genius_debug.yaml \ --log-file overcloud_update_prepare.log (undercloud) [stack@undercloud-0 ~]$ ./overcloud_upgrade_prepare.sh |& tee prepare.log START with options: [u'overcloud', u'upgrade', u'prepare', u'--debug', u'--templates', u'/usr/share/openstack-tripleo-heat-templates', u'--stack', u'overcloud', u'-e', u'/home/stack/virt/config_lvm.yaml', u'-e', u'/usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml', u'-e', u'/home/stack/virt/network/network-environment.yaml', u'-e', u'/home/stack/virt/inject-trust-anchor.yaml', u'-e', u'/home/stack/virt/hostnames.yml', u'-e', u'/home/stack/virt/debug.yaml', u'-e', u'/home/stack/virt/config_heat.yaml', u'-e', u'/home/stack/virt/nodes_data.yaml', u'--environment-file', u'/usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml', u'-e', u'/home/stack/containers-prepare-parameter.yaml', u'-e', u'/home/stack/virt/log_levels/netvirt_genius_debug.yaml', u'--log-file', u'overcloud_update_prepare.log'] options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_type='password', auth_url='https://192.168.24.2:13000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=u'overcloud_update_prepare.log', openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.46', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'auth_type': 'password', u'status': u'active', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u'interface': None, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloud_update_prepare.log', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}} defaults: {u'auth_type': 'password', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, u'interface': None, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': u'1', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', u'baremetal_status_code_retries': 5, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}} defaults: {u'auth_type': 'password', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, u'interface': None, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': u'1', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', u'baremetal_status_code_retries': 5, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}} cloud cfg: {'auth_type': 'password', 'username': 'admin', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', u'status': u'active', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'block_storage_api_version': u'2', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', u'container_api_version': u'1', 'log_file': u'overcloud_update_prepare.log', u'interface': None, u'disable_vendor_agent': {}} compute API version 2, cmd group openstack.compute.v2 network API version 2, cmd group openstack.network.v2 image API version 2, cmd group openstack.image.v2 volume API version 2, cmd group openstack.volume.v2 identity API version 3, cmd group openstack.identity.v3 object_store API version 1, cmd group openstack.object_store.v1 baremetal API version 1.46, cmd group openstack.baremetal.v1 tripleoclient API version 1, cmd group openstack.tripleoclient.v1 metric API version 1, cmd group openstack.metric.v1 neutronclient API version 2, cmd group openstack.neutronclient.v2 orchestration API version 1, cmd group openstack.orchestration.v1 workflow_engine API version 2, cmd group openstack.workflow_engine.v2 messaging API version 2, cmd group openstack.messaging.v2 baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1 dns API version 2, cmd group openstack.dns.v2 database API version 1, cmd group openstack.database.v1 container_infra API version 1, cmd group openstack.container_infra.v1 key_manager API version 1, cmd group openstack.key_manager.v1 alarming API version 2, cmd group openstack.alarming.v2 load_balancer API version 2.0, cmd group openstack.load_balancer.v2 data_processing API version 1.1, cmd group openstack.data_processing.v1 Auth plugin password selected auth_config_hook(): {'auth_type': 'password', u'status': u'active', 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', u'clustering_api_version': u'1', 'verify': True, 'timing': False, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.46', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', u'interface': None, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', 'beta_command': False, u'metering_api_version': u'2', u'baremetal_status_code_retries': '5', u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'deferred_help': False, 'cert': None, u'secgroup_source': u'neutron', 'alarming_api_version': '2', 'debug': True, 'log_file': u'overcloud_update_prepare.log', u'block_storage_api_version': u'2', u'disable_vendor_agent': {}} 2018-12-31 05:43:30.443 924885 INFO osc_lib.shell [-] command: overcloud upgrade prepare -> tripleoclient.v1.overcloud_upgrade.UpgradePrepare (auth=True) 2018-12-31 05:43:30.446 924885 INFO osc_lib.clientmanager [-] Using auth plugin: password 2018-12-31 05:43:30.447 924885 DEBUG osc_lib.clientmanager [-] Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'https://192.168.24.2:13000/', 'password': '***', 'project_domain_name': 'Default'} setup_auth /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:157 2018-12-31 05:43:30.454 924885 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:201 2018-12-31 05:43:37.327 924885 INFO tripleoclient.v1.overcloud_upgrade.MajorUpgradePrepare [-] Stack found, will be doing a stack update Removing the current plan files Uploading new plan files Temporary Swift GET/PUT URL parameters have successfully been updated. Temporary Swift GET/PUT URL parameters have successfully been updated. The backup of the ceph-ansible fetch directory did not need to be renamed Plan updated. Processing templates in the directory /tmp/tripleoclient-XLFG21/tripleo-heat-templates WARNING: Following parameter(s) are deprecated and still defined. Deprecated parameters will be removed soon! OvercloudControlFlavor WARNING: Following parameter(s) are defined but not used in plan. Could be possible that parameter is valid but currently not used. ComputeExtraConfig.nova::compute::consecutive_build_service_disable_threshold SwiftFetchDirPutTempurl CephAnsiblePlaybookVerbosity NeutronTunnelTypes NeutronExternalNetworkBridge SwiftFetchDirGetTempurl 2018-12-31 05:48:47.433 924885 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout. 2018-12-31 09:43:31.114 924885 ERROR openstack [-] No JSON object could be decoded: ValueError: No JSON object could be decoded 2018-12-31 09:43:31.114 924885 ERROR openstack Traceback (most recent call last): 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand 2018-12-31 09:43:31.114 924885 ERROR openstack result = cmd.run(parsed_args) 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run 2018-12-31 09:43:31.114 924885 ERROR openstack super(Command, self).run(parsed_args) 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run 2018-12-31 09:43:31.114 924885 ERROR openstack return super(Command, self).run(parsed_args) 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run 2018-12-31 09:43:31.114 924885 ERROR openstack return_code = self.take_action(parsed_args) or 0 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 68, in take_action 2018-12-31 09:43:31.114 924885 ERROR openstack package_update.update(clients, container=stack_name) 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 38, in update 2018-12-31 09:43:31.114 924885 ERROR openstack for payload in base.wait_for_messages(workflow_client, ws, execution): 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages 2018-12-31 09:43:31.114 924885 ERROR openstack for payload in websocket.wait_for_messages(timeout=timeout): 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages 2018-12-31 09:43:31.114 924885 ERROR openstack message = self.recv() 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv 2018-12-31 09:43:31.114 924885 ERROR openstack return json.loads(self._ws.recv()) 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads 2018-12-31 09:43:31.114 924885 ERROR openstack return _default_decoder.decode(s) 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode 2018-12-31 09:43:31.114 924885 ERROR openstack obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 2018-12-31 09:43:31.114 924885 ERROR openstack File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode 2018-12-31 09:43:31.114 924885 ERROR openstack raise ValueError("No JSON object could be decoded") 2018-12-31 09:43:31.114 924885 ERROR openstack ValueError: No JSON object could be decoded 2018-12-31 09:43:31.114 924885 ERROR openstack 2018-12-31 09:43:31.126 924885 ERROR osc_lib.shell [-] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 68, in take_action package_update.update(clients, container=stack_name) File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 38, in update for payload in base.wait_for_messages(workflow_client, ws, execution): File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages for payload in websocket.wait_for_messages(timeout=timeout): File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages message = self.recv() File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv return json.loads(self._ws.recv()) File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded 2018-12-31 09:43:31.127 924885 INFO osc_lib.shell [-] END return value: 1
*** Bug 1662745 has been marked as a duplicate of this bug. ***
Created attachment 1517892 [details] mistral logs directory
On another attempt to run overcloud upgrade prepare it has passed, however it failed later on overcloud upgrade run: fatal: [controller-2]: FAILED! => ValueError: No JSON object could be decoded ------------ openstack overcloud upgrade prepare --debug \ --stack overcloud \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/config_heat.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-opendaylight.yaml \ -e /home/stack/containers-prepare-parameter.yaml \ -e /home/stack/virt/log_levels/netvirt_genius_debug.yaml openstack overcloud external-upgrade run --stack overcloud --tags container_image_prepare openstack overcloud upgrade run --debug --stack overcloud --roles Controller --playbook all fatal: [controller-2]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "--quiet", "openstack-ceilometer-notification"], "delta": "0:00:00.007509", "end": "2019-01-03 13:47:43.306157", "msg": "non-zero return code", "rc": 1, "start": "2019-01-03 13:47:43.298648", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}'] 2019-01-03 12:47:00.104 1044241 ERROR openstack [-] No JSON object could be decoded: ValueError: No JSON object could be decoded 2019-01-03 12:47:00.104 1044241 ERROR openstack Traceback (most recent call last): 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand 2019-01-03 12:47:00.104 1044241 ERROR openstack result = cmd.run(parsed_args) 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run 2019-01-03 12:47:00.104 1044241 ERROR openstack super(Command, self).run(parsed_args) 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run 2019-01-03 12:47:00.104 1044241 ERROR openstack return super(Command, self).run(parsed_args) 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run 2019-01-03 12:47:00.104 1044241 ERROR openstack return_code = self.take_action(parsed_args) or 0 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 209, in take_action 2019-01-03 12:47:00.104 1044241 ERROR openstack skip_tags=skip_tags) 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 1040, in run_update_ansible_action 2019-01-03 12:47:00.104 1044241 ERROR openstack skip_tags=skip_tags) 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 91, in update_ansible 2019-01-03 12:47:00.104 1044241 ERROR openstack execution): 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages 2019-01-03 12:47:00.104 1044241 ERROR openstack for payload in websocket.wait_for_messages(timeout=timeout): 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages 2019-01-03 12:47:00.104 1044241 ERROR openstack message = self.recv() 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv 2019-01-03 12:47:00.104 1044241 ERROR openstack return json.loads(self._ws.recv()) 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads 2019-01-03 12:47:00.104 1044241 ERROR openstack return _default_decoder.decode(s) 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode 2019-01-03 12:47:00.104 1044241 ERROR openstack obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 2019-01-03 12:47:00.104 1044241 ERROR openstack File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode 2019-01-03 12:47:00.104 1044241 ERROR openstack raise ValueError("No JSON object could be decoded") 2019-01-03 12:47:00.104 1044241 ERROR openstack ValueError: No JSON object could be decoded 2019-01-03 12:47:00.104 1044241 ERROR openstack 2019-01-03 12:47:00.112 1044241 ERROR osc_lib.shell [-] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 209, in take_action skip_tags=skip_tags) File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 1040, in run_update_ansible_action skip_tags=skip_tags) File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 91, in update_ansible execution): File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages for payload in websocket.wait_for_messages(timeout=timeout): File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages message = self.recv() File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv return json.loads(self._ws.recv()) File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded 2019-01-03 12:47:00.113 1044241 INFO osc_lib.shell [-] END return value: 1
Created attachment 1518745 [details] upgrade run log (debug) Apparently "ValueError: No JSON object could be decoded" is just a symptom. When using "--debug" I see in log: fatal: [controller-0]: FAILED! => {"changed": false, "error": "Error: unable to get cib\\nError: unable to get cib\\n", "msg": "Failed, to set the resource rabbitmq-bundle to the state show", "output": "", "rc": 1} I will reattempt an upgrade on a new environment of OSP13. Since "No JSON object could be decoded" error messages, on two different overcloud commands, is not too informative, we better have a Doc-release flag, to add "--debug" on all overcloud upgrade/update commands, so we can easily see the real root cause, should such error occur.
Can you try with those patches: https://review.openstack.org/#/c/630986/ & https://review.openstack.org/#/c/631152/ I think that could fix the handling of the error. NB: for the 1st one you will need to update the workbook manually while applying the patch: mistral workbook-update tripleo-common/workbooks/package_update.yaml
The environment is not available for ODL update/upgrade tests currently. Changing priority to medium.
*** This bug has been marked as a duplicate of bug 1663903 ***