Bug 1722738 - OSP12 to 13 upgrade - fail to upgrade controllers.
Summary: OSP12 to 13 upgrade - fail to upgrade controllers.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z7
: 13.0 (Queens)
Assignee: Rabi Mishra
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks: 1688098
TreeView+ depends on / blocked
 
Reported: 2019-06-21 06:55 UTC by Tzach Shefi
Modified: 2019-07-10 13:01 UTC (History)
6 users (show)

Fixed In Version: openstack-tripleo-common-8.6.8-11.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-10 13:01:47 UTC
Target Upstream Version:


Attachments (Terms of Use)
logs (2.90 MB, application/gzip)
2019-07-02 14:00 UTC, Tzach Shefi
no flags Details


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 667134 None MERGED Fix regression in validation action 2020-04-28 16:57:59 UTC
Red Hat Product Errata RHSA-2019:1742 None None None 2019-07-10 13:01:54 UTC

Description Tzach Shefi 2019-06-21 06:55:08 UTC
Description of problem: During an upgrade from osp12 to osp13, I'd hit bz1651136, after updating repos and trying again the undercloud managed to upgrade. Yet i got stuck on controller upgrade.

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud upgrade run --nodes Controller
Started Mistral Workflow tripleo.package_update.v1.update_nodes. Execution ID: 8d7a597e-5d31-4e85-935d-ab5e75064569
Waiting for messages on queue 'tripleo' with no timeout.
Ansible failed, check log at /var/lib/mistral/8d7a597e-5d31-4e85-935d-ab5e75064569/ansible.log.
Update failed with: Ansible failed, check log at /var/lib/mistral/8d7a597e-5d31-4e85-935d-ab5e75064569/ansible.log.
(undercloud) [stack@undercloud-0 ~]$ vi /var/lib/mistral/8d7a597e-5d31-4e85-935d-ab5e75064569/ansible.log


Version-Release number of selected component (if applicable):
ansible-tripleo-ipsec-8.1.1-0.20190513184007.7eb892c.el7ost.noarch
openstack-tripleo-common-8.6.8-9.el7ost.noarch
openstack-tripleo-image-elements-8.0.2-1.el7ost.noarch
openstack-tripleo-ui-8.3.2-3.el7ost.noarch
openstack-tripleo-heat-templates-compat-7.0.12-17.1.el7ost.noarch
openstack-tripleo-validations-8.4.5-1.el7ost.noarch
openstack-tripleo-common-containers-8.6.8-9.el7ost.noarch
openstack-tripleo-puppet-elements-8.0.2-2.el7ost.noarch
puppet-tripleo-8.4.1-12.el7ost.noarch
openstack-tripleo-heat-templates-8.3.1-48.el7ost.noarch
python-tripleoclient-9.2.7-9.el7ost.noarch


How reproducible:
Unsure first time I hit this. 

Steps to Reproduce:
1. install osp12
2. upgrade undercloud
3. Start upgrading controller nodes.

Actual results:
(undercloud) [stack@undercloud-0 ~]$ openstack overcloud upgrade run --nodes Controller
Started Mistral Workflow tripleo.package_update.v1.update_nodes. Execution ID: 8d7a597e-5d31-4e85-935d-ab5e75064569
Waiting for messages on queue 'tripleo' with no timeout.
Ansible failed, check log at /var/lib/mistral/8d7a597e-5d31-4e85-935d-ab5e75064569/ansible.log.
Update failed with: Ansible failed, check log at /var/lib/mistral/8d7a597e-5d31-4e85-935d-ab5e75064569/ansible.log.
(undercloud) [stack@undercloud-0 ~]$ vi /var/lib/mistral/8d7a597e-5d31-4e85-935d-ab5e75064569/ansible.log


There is no /var/lib/mistral/8d7a597e-5d31-4e85-935d-ab5e75064569/ folder argo no ansible log to add else I'd attach it. 

Expected results:
Upgrade should complete. 

Additional info:
I had problems with low disk space on the undercloud 0% free, instead of caching the docker images on local registry I removed them which cleared some 11G. And  switched to RH registry without local cache. After this I started the controller upgrade which failed. If this is related or not I'm not sure.

Comment 3 Jose Luis Franco 2019-06-24 14:43:37 UTC
So, the problem seems to be in the "upgrade prepare command", it didn't success correctly and when I tried to run it again I saw this error:

2019-06-21 12:25:07.460 1959 INFO mistral.executors.executor_server [req-79efc84f-5d11-4a77-bf1a-32fc34d183b8 e05127232d9c444a82598fa4db89e38c 65950edd3cfe4e89b7215cd28f5cda77 - default default] Received RPC request 'run_action'[action_ex_id=8d18e5a2-e1a2-4c1a-810a-f0e811d41d60, action_cls_str=tripleo_common.actions.validations.CheckNodesCountAction, action_cls_attrs={}, params={statistics: {u'count': 4, u'vcpus_used': 22, u'local_gb_used': 98, u'manager': {u'api': {u'serve..., timeout=None]
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor [req-79efc84f-5d11-4a77-bf1a-32fc34d183b8 e05127232d9c444a82598fa4db89e38c 65950edd3cfe4e89b7215cd28f5cda77 - default default] Failed to run action [action_ex_id=8d18e5a2-e1a2-4c1a-810a-f0e811d41d60, action_cls='<class 'mistral.actions.action_factory.CheckNodesCountAction'>', attributes='{}', params='{u'statistics': {u'count': 4, u'vcpus_used': 22, u'local_gb_used': 98, u'manager': {u'api': {u'server_groups': None, u'keypairs': None, u'servers': None, u'hypervisors': None, u'server_migrations': None, u'agents': None, u'instance_action': None, u'glance': None, u'hypervisor_stats': None, u'flavors': None, u'availability_zones': None, u'user_id': None, u'list_extensions': None, u'os_cache': False, u'quotas': None, u'migrations': None, u'usage': None, u'logger': None, u'project_id': None, u'neutron': None, u'quota_classes': None, u'project_name': None, u'aggregates': None, u'flavor_access': None, u'services': None, u'limits': None, u'versions': None, u'cells': None, u'client': None, u'server_external_events': None, u'volumes': None, u'assisted_volume_snapshots': None}}, u'x_openstack_request_ids': [u'req-158c1b78-ce68-4699-a5df-c5dd442f9aa4'], u'memory_mb': 104448, u'current_workload': 0, u'vcpus': 26, u'running_vms': 4, u'free_disk_gb': 8, u'disk_available_least': 0, u'_info': {u'count': 4, u'vcpus_used': 22, u'local_gb_used': 98, u'memory_mb': 104448, u'current_workload': 0, u'vcpus': 26, u'running_vms': 4, u'free_disk_gb': 8, u'disk_available_least': 0, u'local_gb': 106, u'free_ram_mb': 0, u'memory_mb_used': 104448}, u'local_gb': 106, u'free_ram_mb': 0, u'memory_mb_used': 104448, u'_loaded': True}, u'parameters': {u'DeployIdentifier': u'1561125802', u'StackAction': u'UPDATE', u'UpdateIdentifier': u''}, u'default_role_counts': {u'ComputeCount': 1, u'ControllerCount': 1, u'ObjectStorageCount': 0, u'CephStorageCount': 0, u'BlockStorageCount': 0}, u'available_nodes': [], u'associated_nodes': [{u'instance_uuid': u'33f55535-2089-4f72-b488-1df2da4bc10f', u'name': u'compute-0', u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'rel': u'bookmark'}], u'uuid': u'7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'manager': {u'api': {u'os_ironic_api_version': u'1.34', u'endpoint': u'http://192.168.24.1:6385', u'endpoint_trimmed': u'http://192.168.24.1:6385', u'conflict_retry_interval': 2, u'auth_token': u'gAAAAABdDQSt8Rrzuu_TiIGx3BmbeDFliDgRYx9vj4jP1_b2t9LcxdCGlOaKpc8AsHKNJs4qo-0IH1q-abd7F2X4gU5WW_t2XJ_mnfXMbYGUyJjrDST0B2CEjem_rEVpMZwZT4C3t_CP3WH3ZGWlcms9mlBECNPagcfSNiY0J0HbkZSjt49DBWk', u'auth_ref': None, u'conflict_max_retries': 5, u'session': None, u'api_version_select_state': u'user'}}, u'maintenance': False, u'provision_state': u'active', u'_info': {u'instance_uuid': u'33f55535-2089-4f72-b488-1df2da4bc10f', u'maintenance': False, u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'rel': u'bookmark'}], u'uuid': u'7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'power_state': u'power on', u'provision_state': u'active', u'name': u'compute-0'}, u'power_state': u'power on', u'_loaded': True}, {u'instance_uuid': u'7a0d4b2a-6d16-4cb7-bca9-94e4d1238dfa', u'name': u'controller-0', u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'rel': u'bookmark'}], u'uuid': u'0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'manager': {u'api': {u'os_ironic_api_version': u'1.34', u'endpoint': u'http://192.168.24.1:6385', u'endpoint_trimmed': u'http://192.168.24.1:6385', u'conflict_retry_interval': 2, u'auth_token': u'gAAAAABdDQSt8Rrzuu_TiIGx3BmbeDFliDgRYx9vj4jP1_b2t9LcxdCGlOaKpc8AsHKNJs4qo-0IH1q-abd7F2X4gU5WW_t2XJ_mnfXMbYGUyJjrDST0B2CEjem_rEVpMZwZT4C3t_CP3WH3ZGWlcms9mlBECNPagcfSNiY0J0HbkZSjt49DBWk', u'auth_ref': None, u'conflict_max_retries': 5, u'session': None, u'api_version_select_state': u'user'}}, u'maintenance': False, u'provision_state': u'active', u'_info': {u'instance_uuid': u'7a0d4b2a-6d16-4cb7-bca9-94e4d1238dfa', u'maintenance': False, u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'rel': u'bookmark'}], u'uuid': u'0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'power_state': u'power on', u'provision_state': u'active', u'name': u'controller-0'}, u'power_state': u'power on', u'_loaded': True}, {u'instance_uuid': u'93ca39fe-b246-4dd8-a131-1fa60f790009', u'name': u'controller-1', u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/38ae95e5-5e9e-4127-a3be-56892d33ff32', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/38ae95e5-5e9e-4127-a3be-56892d33ff32', u'rel': u'bookmark'}], u'uuid': u'38ae95e5-5e9e-4127-a3be-56892d33ff32', u'manager': {u'api': {u'os_ironic_api_version': u'1.34', u'endpoint': u'http://192.168.24.1:6385', u'endpoint_trimmed': u'http://192.168.24.1:6385', u'conflict_retry_interval': 2, u'auth_token': u'gAAAAABdDQSt8Rrzuu_TiIGx3BmbeDFliDgRYx9vj4jP1_b2t9LcxdCGlOaKpc8AsHKNJs4qo-0IH1q-abd7F2X4gU5WW_t2XJ_mnfXMbYGUyJjrDST0B2CEjem_rEVpMZwZT4C3t_CP3WH3ZGWlcms9mlBECNPagcfSNiY0J0HbkZSjt49DBWk', u'auth_ref': None, u'conflict_max_retries': 5, u'session': None, u'api_version_select_state': u'user'}}, u'maintenance': False, u'provision_state': u'active', u'_info': {u'instance_uuid': u'93ca39fe-b246-4dd8-a131-1fa60f790009', u'maintenance': False, u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/38ae95e5-5e9e-4127-a3be-56892d33ff32', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/38ae95e5-5e9e-4127-a3be-56892d33ff32', u'rel': u'bookmark'}], u'uuid': u'38ae95e5-5e9e-4127-a3be-56892d33ff32', u'power_state': u'power on', u'provision_state': u'active', u'name': u'controller-1'}, u'power_state': u'power on', u'_loaded': True}, {u'instance_uuid': u'67e17e23-9646-4c2e-a4b7-45391dc539bc', u'name': u'controller-2', u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'rel': u'bookmark'}], u'uuid': u'6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'manager': {u'api': {u'os_ironic_api_version': u'1.34', u'endpoint': u'http://192.168.24.1:6385', u'endpoint_trimmed': u'http://192.168.24.1:6385', u'conflict_retry_interval': 2, u'auth_token': u'gAAAAABdDQSt8Rrzuu_TiIGx3BmbeDFliDgRYx9vj4jP1_b2t9LcxdCGlOaKpc8AsHKNJs4qo-0IH1q-abd7F2X4gU5WW_t2XJ_mnfXMbYGUyJjrDST0B2CEjem_rEVpMZwZT4C3t_CP3WH3ZGWlcms9mlBECNPagcfSNiY0J0HbkZSjt49DBWk', u'auth_ref': None, u'conflict_max_retries': 5, u'session': None, u'api_version_select_state': u'user'}}, u'maintenance': False, u'provision_state': u'active', u'_info': {u'instance_uuid': u'67e17e23-9646-4c2e-a4b7-45391dc539bc', u'maintenance': False, u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'rel': u'bookmark'}], u'uuid': u'6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'power_state': u'power on', u'provision_state': u'active', u'name': u'controller-2'}, u'power_state': u'power on', u'_loaded': True}], u'stack': {u'links': [{u'href': u'http://192.168.24.1:8004/v1/65950edd3cfe4e89b7215cd28f5cda77/stacks/overcloud/c7b5ae04-0e2c-4d80-a4c0-930bf353af62', u'rel': u'self'}], u'stack_name': u'overcloud', u'creation_time': u'2019-06-21T14:04:34Z', u'notification_topics': [], u'manager': {u'client': {u'allow_version_hack': None, u'global_request_id': None, u'region_name': None, u'additional_headers': {}, u'service_name': None, u'min_version': None, u'auth': None, u'interface': None, u'version': None, u'session': None, u'connect_retries': None, u'allow': {}, u'endpoint_override': u'http://192.168.24.1:8004/v1/65950edd3cfe4e89b7215cd28f5cda77', u'service_type': u'orchestration', u'max_version': None, u'logger': None, u'default_microversion': None, u'user_agent': None}}, u'timeout_mins': 100, u'id': u'c7b5ae04-0e2c-4d80-a4c0-930bf353af62', u'parameters': {u'ControllerParameters': u'{}', u'OS::project_id': u'65950edd3cfe4e89b7215cd28f5cda77', u'ControllerCount': u'3', u'ExtraConfig': u'{}', u'BlockStorageParameters': u'{}', u'ServerMetadata': u'{}', u'CloudName': u'overcloud.localdomain', u'PublicVirtualFixedIPs': u'[]', u'StorageMgmtVirtualFixedIPs': u'[]', u'InternalApiVirtualFixedIPs': u'[]', u'NeutronPublicInterface': u'nic1', u'BlockStorageRemovalPolicies': u'[]', u'CloudNameInternal': u'overcloud.internalapi.localdomain', u'CephStorageSchedulerHints': u'{}', u'ObjectStorageHostnameFormat': u'swift-%index%', u'ControllerHostnameFormat': u'controller-%index%', u'CloudDomain': u'localdomain', u'BlockStorageServices': u'OS::TripleO::Services::AuditD,OS::TripleO::Services::BlockStorageCinderVolume,OS::TripleO::Services::CACerts,OS::TripleO::Services::CertmongerUser,OS::TripleO::Services::CinderBackendVRTSHyperScale,OS::TripleO::Services::Collectd,OS::TripleO::Services::Docker,OS::TripleO::Services::FluentdClient,OS::TripleO::Services::Iscsid,OS::TripleO::Services::Kernel,OS::TripleO::Services::MySQLClient,OS::TripleO::Services::Ntp,OS::TripleO::Services::ContainersLogrotateCrond,OS::TripleO::Services::Securetty,OS::TripleO::Services::SensuClient,OS::TripleO::Services::Snmp,OS::TripleO::Services::Sshd,OS::TripleO::Services::Timezone,OS::TripleO::Services::TripleoFirewall,OS::TripleO::Services::TripleoPackages,OS::TripleO::Services::Tuned', u'NovaComputeExtraConfig': u'{}', u'ControllerRemovalPolicies': u'[]', u'ComputeCount': u'1', u'ControllerExtraConfig': u'{}', u'HypervisorNeutronPhysicalBridge': u'br-ex', u'NeutronControlPlaneID': u'ctlplane', u'BlockStorageExtraConfig': u'{}', u'NodeCreateBatchSize': u'30', u'CephStorageServices': u'OS::TripleO::Services::AuditD,OS::TripleO::Services::CACerts,OS::TripleO::Services::CephOSD,OS::TripleO::Services::CertmongerUser,OS::TripleO::Services::Collectd,OS::TripleO::Services::Docker,OS::TripleO::Services::FluentdClient,OS::TripleO::Services::Kernel,OS::TripleO::Services::MySQLClient,OS::TripleO::Services::Ntp,OS::TripleO::Services::ContainersLogrotateCrond,OS::TripleO::Services::Securetty,OS::TripleO::Services::SensuClient,OS::TripleO::Services::Snmp,OS::TripleO::Services::Sshd,OS::TripleO::Services::Timezone,OS::TripleO::Services::TripleoFirewall,OS::TripleO::Services::TripleoPackages,OS::TripleO::Services::Tuned', u'ComputeHostnameFormat': u'compute-%index%', u'controllerExtraConfig': u'{}', u'RedisVirtualFixedIPs': u'[]', u'CloudNameStorage': u'overcloud.storage.localdomain', u'HypervisorNeutronPublicInterface': u'nic1', u'ObjectStorageCount': u'0', u'OS::stack_name': u'overcloud', u'ObjectStorageParameters': u'{}', u'CloudNameCtlplane': u'overcloud.ctlplane.localdomain', u'RabbitCookieSalt': u'unset', u'CephStorageRemovalPolicies': u'[]', u'AddVipsToEtcHosts': u'True', u'CloudNameStorageManagement': u'overcloud.storagemgmt.localdomain', u'DeployIdentifier': u'1561125802', u'ComputeServices': u'OS::TripleO::Services::AuditD,OS::TripleO::Services::CACerts,OS::TripleO::Services::CephClient,OS::TripleO::Services::CephExternal,OS::TripleO::Services::CertmongerUser,OS::TripleO::Services::Collectd,OS::TripleO::Services::ComputeCeilometerAgent,OS::TripleO::Services::ComputeNeutronCorePlugin,OS::TripleO::Services::ComputeNeutronL3Agent,OS::TripleO::Services::ComputeNeutronMetadataAgent,OS::TripleO::Services::ComputeNeutronOvsAgent,OS::TripleO::Services::Docker,OS::TripleO::Services::FluentdClient,OS::TripleO::Services::Iscsid,OS::TripleO::Services::Kernel,OS::TripleO::Services::MySQLClient,OS::TripleO::Services::NeutronLinuxbridgeAgent,OS::TripleO::Services::NeutronSriovAgent,OS::TripleO::Services::NeutronSriovHostConfig,OS::TripleO::Services::NeutronVppAgent,OS::TripleO::Services::NovaCompute,OS::TripleO::Services::NovaLibvirt,OS::TripleO::Services::NovaMigrationTarget,OS::TripleO::Services::Ntp,OS::TripleO::Services::ContainersLogrotateCrond,OS::TripleO::Services::OpenDaylightOvs,OS::TripleO::Services::Securetty,OS::TripleO::Services::SensuClient,OS::TripleO::Services::Snmp,OS::TripleO::Services::Sshd,OS::TripleO::Services::Timezone,OS::TripleO::Services::TripleoFirewall,OS::TripleO::Services::TripleoPackages,OS::TripleO::Services::Tuned,OS::TripleO::Services::Vpp,OS::TripleO::Services::OVNController', u'BlockStorageCount': u'0', u'BlockStorageSchedulerHints': u'{}', u'ControlFixedIPs': u'[]', u'StorageVirtualFixedIPs': u'[]', u'CephStorageExtraConfig': u'{}', u'ObjectStorageSchedulerHints': u'{}', u'BlockStorageHostnameFormat': u'%stackname%-blockstorage-%index%', u'DeploymentServerBlacklist': u'', u'OS::stack_id': u'c7b5ae04-0e2c-4d80-a4c0-930bf353af62', u'CephStorageParameters': u'{}', u'ObjectStorageServices': u'OS::TripleO::Services::AuditD,OS::TripleO::Services::CACerts,OS::TripleO::Services::CertmongerUser,OS::TripleO::Services::Collectd,OS::TripleO::Services::Docker,OS::TripleO::Services::FluentdClient,OS::TripleO::Services::Kernel,OS::TripleO::Services::MySQLClient,OS::TripleO::Services::Ntp,OS::TripleO::Services::ContainersLogrotateCrond,OS::TripleO::Services::Securetty,OS::TripleO::Services::SensuClient,OS::TripleO::Services::Snmp,OS::TripleO::Services::Sshd,OS::TripleO::Services::SwiftRingBuilder,OS::TripleO::Services::SwiftStorage,OS::TripleO::Services::Timezone,OS::TripleO::Services::TripleoFirewall,OS::TripleO::Services::TripleoPackages,OS::TripleO::Services::Tuned', u'ControllerServices': u'OS::TripleO::Services::AodhApi,OS::TripleO::Services::AodhEvaluator,OS::TripleO::Services::AodhListener,OS::TripleO::Services::AodhNotifier,OS::TripleO::Services::AuditD,OS::TripleO::Services::BarbicanApi,OS::TripleO::Services::CACerts,OS::TripleO::Services::CeilometerAgentCentral,OS::TripleO::Services::CeilometerAgentNotification,OS::TripleO::Services::CeilometerApi,OS::TripleO::Services::CeilometerCollector,OS::TripleO::Services::CeilometerExpirer,OS::TripleO::Services::CephExternal,OS::TripleO::Services::CephMds,OS::TripleO::Services:.....
 'dict' object has no attribute 'status': AttributeError: 'dict' object has no attribute 'status'
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor.py", line 114, in run_action
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/validations.py", line 432, in run
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor     if self.stack and self.stack.status != 'FAILED':
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor AttributeError: 'dict' object has no attribute 'status'
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor
2019-06-21 12:25:18.454 1959 INFO mistral.executors.executor_server [req-0a31b0f0-21b9-4795-90aa-5acb8199834e e05127232d9c444a82598fa4db89e38c 65950edd3cfe4e89b7215cd28f5cda77 - default default] Received RPC request 'run_action'[action_ex_id=fcf7d33a-e176-4779-8e66-3f7e8737bf79, action_cls_str=mistral.actions.std_actions.NoOpAction, action_cls_attrs={}, params={}, timeout=None]


This issue seems to be caused by this patch which merged fairly "recently": https://review.opendev.org/#/c/656379/, so I'll redirect it to DFG:DF.

It would be great if you could keep the environment for further debugging.

Comment 14 Tzach Shefi 2019-07-02 14:00:34 UTC
Created attachment 1586725 [details]
logs

Comment 19 Tzach Shefi 2019-07-03 15:36:03 UTC
Verified on:
openstack-tripleo-common-8.6.8-11.el7ost.noarch

Took a while and two deployments plus some messing around with docker image source and mirror problems. 

But I was finally able to upgrade an OSP12 to OSP13. 
Tested on two separate systems, the none HA completed the upgrade without issues. 

The HA system still waiting for it to finish, I'll update should it fail or if I hit any more issues. 

From the none HA system:

2019-07-03 15:28:14Z [AllNodesDeploySteps]: UPDATE_COMPLETE  state changed
2019-07-03 15:28:19Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully

 Stack overcloud UPDATE_COMPLETE 

Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: f41ae804-9aed-420c-b5f0-bddcf15593d9
Overcloud Endpoint: http://10.0.0.107:5000/
Overcloud Horizon Dashboard URL: http://10.0.0.107:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed
Completed Overcloud Upgrade Converge for stack overcloud


Yay... Okay to verify.

Comment 21 errata-xmlrpc 2019-07-10 13:01:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1742


Note You need to log in before you can comment on or make changes to this bug.