Bug 1722738

So, the problem seems to be in the "upgrade prepare command", it didn't success correctly and when I tried to run it again I saw this error:

2019-06-21 12:25:07.460 1959 INFO mistral.executors.executor_server [req-79efc84f-5d11-4a77-bf1a-32fc34d183b8 e05127232d9c444a82598fa4db89e38c 65950edd3cfe4e89b7215cd28f5cda77 - default default] Received RPC request 'run_action'[action_ex_id=8d18e5a2-e1a2-4c1a-810a-f0e811d41d60, action_cls_str=tripleo_common.actions.validations.CheckNodesCountAction, action_cls_attrs={}, params={statistics: {u'count': 4, u'vcpus_used': 22, u'local_gb_used': 98, u'manager': {u'api': {u'serve..., timeout=None]
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor [req-79efc84f-5d11-4a77-bf1a-32fc34d183b8 e05127232d9c444a82598fa4db89e38c 65950edd3cfe4e89b7215cd28f5cda77 - default default] Failed to run action [action_ex_id=8d18e5a2-e1a2-4c1a-810a-f0e811d41d60, action_cls='<class 'mistral.actions.action_factory.CheckNodesCountAction'>', attributes='{}', params='{u'statistics': {u'count': 4, u'vcpus_used': 22, u'local_gb_used': 98, u'manager': {u'api': {u'server_groups': None, u'keypairs': None, u'servers': None, u'hypervisors': None, u'server_migrations': None, u'agents': None, u'instance_action': None, u'glance': None, u'hypervisor_stats': None, u'flavors': None, u'availability_zones': None, u'user_id': None, u'list_extensions': None, u'os_cache': False, u'quotas': None, u'migrations': None, u'usage': None, u'logger': None, u'project_id': None, u'neutron': None, u'quota_classes': None, u'project_name': None, u'aggregates': None, u'flavor_access': None, u'services': None, u'limits': None, u'versions': None, u'cells': None, u'client': None, u'server_external_events': None, u'volumes': None, u'assisted_volume_snapshots': None}}, u'x_openstack_request_ids': [u'req-158c1b78-ce68-4699-a5df-c5dd442f9aa4'], u'memory_mb': 104448, u'current_workload': 0, u'vcpus': 26, u'running_vms': 4, u'free_disk_gb': 8, u'disk_available_least': 0, u'_info': {u'count': 4, u'vcpus_used': 22, u'local_gb_used': 98, u'memory_mb': 104448, u'current_workload': 0, u'vcpus': 26, u'running_vms': 4, u'free_disk_gb': 8, u'disk_available_least': 0, u'local_gb': 106, u'free_ram_mb': 0, u'memory_mb_used': 104448}, u'local_gb': 106, u'free_ram_mb': 0, u'memory_mb_used': 104448, u'_loaded': True}, u'parameters': {u'DeployIdentifier': u'1561125802', u'StackAction': u'UPDATE', u'UpdateIdentifier': u''}, u'default_role_counts': {u'ComputeCount': 1, u'ControllerCount': 1, u'ObjectStorageCount': 0, u'CephStorageCount': 0, u'BlockStorageCount': 0}, u'available_nodes': [], u'associated_nodes': [{u'instance_uuid': u'33f55535-2089-4f72-b488-1df2da4bc10f', u'name': u'compute-0', u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'rel': u'bookmark'}], u'uuid': u'7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'manager': {u'api': {u'os_ironic_api_version': u'1.34', u'endpoint': u'http://192.168.24.1:6385', u'endpoint_trimmed': u'http://192.168.24.1:6385', u'conflict_retry_interval': 2, u'auth_token': u'gAAAAABdDQSt8Rrzuu_TiIGx3BmbeDFliDgRYx9vj4jP1_b2t9LcxdCGlOaKpc8AsHKNJs4qo-0IH1q-abd7F2X4gU5WW_t2XJ_mnfXMbYGUyJjrDST0B2CEjem_rEVpMZwZT4C3t_CP3WH3ZGWlcms9mlBECNPagcfSNiY0J0HbkZSjt49DBWk', u'auth_ref': None, u'conflict_max_retries': 5, u'session': None, u'api_version_select_state': u'user'}}, u'maintenance': False, u'provision_state': u'active', u'_info': {u'instance_uuid': u'33f55535-2089-4f72-b488-1df2da4bc10f', u'maintenance': False, u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'rel': u'bookmark'}], u'uuid': u'7bf13c1c-69ea-44f5-8979-69b6d098eb2c', u'power_state': u'power on', u'provision_state': u'active', u'name': u'compute-0'}, u'power_state': u'power on', u'_loaded': True}, {u'instance_uuid': u'7a0d4b2a-6d16-4cb7-bca9-94e4d1238dfa', u'name': u'controller-0', u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'rel': u'bookmark'}], u'uuid': u'0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'manager': {u'api': {u'os_ironic_api_version': u'1.34', u'endpoint': u'http://192.168.24.1:6385', u'endpoint_trimmed': u'http://192.168.24.1:6385', u'conflict_retry_interval': 2, u'auth_token': u'gAAAAABdDQSt8Rrzuu_TiIGx3BmbeDFliDgRYx9vj4jP1_b2t9LcxdCGlOaKpc8AsHKNJs4qo-0IH1q-abd7F2X4gU5WW_t2XJ_mnfXMbYGUyJjrDST0B2CEjem_rEVpMZwZT4C3t_CP3WH3ZGWlcms9mlBECNPagcfSNiY0J0HbkZSjt49DBWk', u'auth_ref': None, u'conflict_max_retries': 5, u'session': None, u'api_version_select_state': u'user'}}, u'maintenance': False, u'provision_state': u'active', u'_info': {u'instance_uuid': u'7a0d4b2a-6d16-4cb7-bca9-94e4d1238dfa', u'maintenance': False, u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'rel': u'bookmark'}], u'uuid': u'0a12cbab-1cca-43eb-9ad4-08e6d99044cd', u'power_state': u'power on', u'provision_state': u'active', u'name': u'controller-0'}, u'power_state': u'power on', u'_loaded': True}, {u'instance_uuid': u'93ca39fe-b246-4dd8-a131-1fa60f790009', u'name': u'controller-1', u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/38ae95e5-5e9e-4127-a3be-56892d33ff32', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/38ae95e5-5e9e-4127-a3be-56892d33ff32', u'rel': u'bookmark'}], u'uuid': u'38ae95e5-5e9e-4127-a3be-56892d33ff32', u'manager': {u'api': {u'os_ironic_api_version': u'1.34', u'endpoint': u'http://192.168.24.1:6385', u'endpoint_trimmed': u'http://192.168.24.1:6385', u'conflict_retry_interval': 2, u'auth_token': u'gAAAAABdDQSt8Rrzuu_TiIGx3BmbeDFliDgRYx9vj4jP1_b2t9LcxdCGlOaKpc8AsHKNJs4qo-0IH1q-abd7F2X4gU5WW_t2XJ_mnfXMbYGUyJjrDST0B2CEjem_rEVpMZwZT4C3t_CP3WH3ZGWlcms9mlBECNPagcfSNiY0J0HbkZSjt49DBWk', u'auth_ref': None, u'conflict_max_retries': 5, u'session': None, u'api_version_select_state': u'user'}}, u'maintenance': False, u'provision_state': u'active', u'_info': {u'instance_uuid': u'93ca39fe-b246-4dd8-a131-1fa60f790009', u'maintenance': False, u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/38ae95e5-5e9e-4127-a3be-56892d33ff32', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/38ae95e5-5e9e-4127-a3be-56892d33ff32', u'rel': u'bookmark'}], u'uuid': u'38ae95e5-5e9e-4127-a3be-56892d33ff32', u'power_state': u'power on', u'provision_state': u'active', u'name': u'controller-1'}, u'power_state': u'power on', u'_loaded': True}, {u'instance_uuid': u'67e17e23-9646-4c2e-a4b7-45391dc539bc', u'name': u'controller-2', u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'rel': u'bookmark'}], u'uuid': u'6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'manager': {u'api': {u'os_ironic_api_version': u'1.34', u'endpoint': u'http://192.168.24.1:6385', u'endpoint_trimmed': u'http://192.168.24.1:6385', u'conflict_retry_interval': 2, u'auth_token': u'gAAAAABdDQSt8Rrzuu_TiIGx3BmbeDFliDgRYx9vj4jP1_b2t9LcxdCGlOaKpc8AsHKNJs4qo-0IH1q-abd7F2X4gU5WW_t2XJ_mnfXMbYGUyJjrDST0B2CEjem_rEVpMZwZT4C3t_CP3WH3ZGWlcms9mlBECNPagcfSNiY0J0HbkZSjt49DBWk', u'auth_ref': None, u'conflict_max_retries': 5, u'session': None, u'api_version_select_state': u'user'}}, u'maintenance': False, u'provision_state': u'active', u'_info': {u'instance_uuid': u'67e17e23-9646-4c2e-a4b7-45391dc539bc', u'maintenance': False, u'links': [{u'href': u'http://192.168.24.1:6385/v1/nodes/6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'rel': u'self'}, {u'href': u'http://192.168.24.1:6385/nodes/6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'rel': u'bookmark'}], u'uuid': u'6d04dcfc-be04-4a7c-9d1c-08cea1f1d7e0', u'power_state': u'power on', u'provision_state': u'active', u'name': u'controller-2'}, u'power_state': u'power on', u'_loaded': True}], u'stack': {u'links': [{u'href': u'http://192.168.24.1:8004/v1/65950edd3cfe4e89b7215cd28f5cda77/stacks/overcloud/c7b5ae04-0e2c-4d80-a4c0-930bf353af62', u'rel': u'self'}], u'stack_name': u'overcloud', u'creation_time': u'2019-06-21T14:04:34Z', u'notification_topics': [], u'manager': {u'client': {u'allow_version_hack': None, u'global_request_id': None, u'region_name': None, u'additional_headers': {}, u'service_name': None, u'min_version': None, u'auth': None, u'interface': None, u'version': None, u'session': None, u'connect_retries': None, u'allow': {}, u'endpoint_override': u'http://192.168.24.1:8004/v1/65950edd3cfe4e89b7215cd28f5cda77', u'service_type': u'orchestration', u'max_version': None, u'logger': None, u'default_microversion': None, u'user_agent': None}}, u'timeout_mins': 100, u'id': u'c7b5ae04-0e2c-4d80-a4c0-930bf353af62', u'parameters': {u'ControllerParameters': u'{}', u'OS::project_id': u'65950edd3cfe4e89b7215cd28f5cda77', u'ControllerCount': u'3', u'ExtraConfig': u'{}', u'BlockStorageParameters': u'{}', u'ServerMetadata': u'{}', u'CloudName': u'overcloud.localdomain', u'PublicVirtualFixedIPs': u'[]', u'StorageMgmtVirtualFixedIPs': u'[]', u'InternalApiVirtualFixedIPs': u'[]', u'NeutronPublicInterface': u'nic1', u'BlockStorageRemovalPolicies': u'[]', u'CloudNameInternal': u'overcloud.internalapi.localdomain', u'CephStorageSchedulerHints': u'{}', u'ObjectStorageHostnameFormat': u'swift-%index%', u'ControllerHostnameFormat': u'controller-%index%', u'CloudDomain': u'localdomain', u'BlockStorageServices': u'OS::TripleO::Services::AuditD,OS::TripleO::Services::BlockStorageCinderVolume,OS::TripleO::Services::CACerts,OS::TripleO::Services::CertmongerUser,OS::TripleO::Services::CinderBackendVRTSHyperScale,OS::TripleO::Services::Collectd,OS::TripleO::Services::Docker,OS::TripleO::Services::FluentdClient,OS::TripleO::Services::Iscsid,OS::TripleO::Services::Kernel,OS::TripleO::Services::MySQLClient,OS::TripleO::Services::Ntp,OS::TripleO::Services::ContainersLogrotateCrond,OS::TripleO::Services::Securetty,OS::TripleO::Services::SensuClient,OS::TripleO::Services::Snmp,OS::TripleO::Services::Sshd,OS::TripleO::Services::Timezone,OS::TripleO::Services::TripleoFirewall,OS::TripleO::Services::TripleoPackages,OS::TripleO::Services::Tuned', u'NovaComputeExtraConfig': u'{}', u'ControllerRemovalPolicies': u'[]', u'ComputeCount': u'1', u'ControllerExtraConfig': u'{}', u'HypervisorNeutronPhysicalBridge': u'br-ex', u'NeutronControlPlaneID': u'ctlplane', u'BlockStorageExtraConfig': u'{}', u'NodeCreateBatchSize': u'30', u'CephStorageServices': u'OS::TripleO::Services::AuditD,OS::TripleO::Services::CACerts,OS::TripleO::Services::CephOSD,OS::TripleO::Services::CertmongerUser,OS::TripleO::Services::Collectd,OS::TripleO::Services::Docker,OS::TripleO::Services::FluentdClient,OS::TripleO::Services::Kernel,OS::TripleO::Services::MySQLClient,OS::TripleO::Services::Ntp,OS::TripleO::Services::ContainersLogrotateCrond,OS::TripleO::Services::Securetty,OS::TripleO::Services::SensuClient,OS::TripleO::Services::Snmp,OS::TripleO::Services::Sshd,OS::TripleO::Services::Timezone,OS::TripleO::Services::TripleoFirewall,OS::TripleO::Services::TripleoPackages,OS::TripleO::Services::Tuned', u'ComputeHostnameFormat': u'compute-%index%', u'controllerExtraConfig': u'{}', u'RedisVirtualFixedIPs': u'[]', u'CloudNameStorage': u'overcloud.storage.localdomain', u'HypervisorNeutronPublicInterface': u'nic1', u'ObjectStorageCount': u'0', u'OS::stack_name': u'overcloud', u'ObjectStorageParameters': u'{}', u'CloudNameCtlplane': u'overcloud.ctlplane.localdomain', u'RabbitCookieSalt': u'unset', u'CephStorageRemovalPolicies': u'[]', u'AddVipsToEtcHosts': u'True', u'CloudNameStorageManagement': u'overcloud.storagemgmt.localdomain', u'DeployIdentifier': u'1561125802', u'ComputeServices': u'OS::TripleO::Services::AuditD,OS::TripleO::Services::CACerts,OS::TripleO::Services::CephClient,OS::TripleO::Services::CephExternal,OS::TripleO::Services::CertmongerUser,OS::TripleO::Services::Collectd,OS::TripleO::Services::ComputeCeilometerAgent,OS::TripleO::Services::ComputeNeutronCorePlugin,OS::TripleO::Services::ComputeNeutronL3Agent,OS::TripleO::Services::ComputeNeutronMetadataAgent,OS::TripleO::Services::ComputeNeutronOvsAgent,OS::TripleO::Services::Docker,OS::TripleO::Services::FluentdClient,OS::TripleO::Services::Iscsid,OS::TripleO::Services::Kernel,OS::TripleO::Services::MySQLClient,OS::TripleO::Services::NeutronLinuxbridgeAgent,OS::TripleO::Services::NeutronSriovAgent,OS::TripleO::Services::NeutronSriovHostConfig,OS::TripleO::Services::NeutronVppAgent,OS::TripleO::Services::NovaCompute,OS::TripleO::Services::NovaLibvirt,OS::TripleO::Services::NovaMigrationTarget,OS::TripleO::Services::Ntp,OS::TripleO::Services::ContainersLogrotateCrond,OS::TripleO::Services::OpenDaylightOvs,OS::TripleO::Services::Securetty,OS::TripleO::Services::SensuClient,OS::TripleO::Services::Snmp,OS::TripleO::Services::Sshd,OS::TripleO::Services::Timezone,OS::TripleO::Services::TripleoFirewall,OS::TripleO::Services::TripleoPackages,OS::TripleO::Services::Tuned,OS::TripleO::Services::Vpp,OS::TripleO::Services::OVNController', u'BlockStorageCount': u'0', u'BlockStorageSchedulerHints': u'{}', u'ControlFixedIPs': u'[]', u'StorageVirtualFixedIPs': u'[]', u'CephStorageExtraConfig': u'{}', u'ObjectStorageSchedulerHints': u'{}', u'BlockStorageHostnameFormat': u'%stackname%-blockstorage-%index%', u'DeploymentServerBlacklist': u'', u'OS::stack_id': u'c7b5ae04-0e2c-4d80-a4c0-930bf353af62', u'CephStorageParameters': u'{}', u'ObjectStorageServices': u'OS::TripleO::Services::AuditD,OS::TripleO::Services::CACerts,OS::TripleO::Services::CertmongerUser,OS::TripleO::Services::Collectd,OS::TripleO::Services::Docker,OS::TripleO::Services::FluentdClient,OS::TripleO::Services::Kernel,OS::TripleO::Services::MySQLClient,OS::TripleO::Services::Ntp,OS::TripleO::Services::ContainersLogrotateCrond,OS::TripleO::Services::Securetty,OS::TripleO::Services::SensuClient,OS::TripleO::Services::Snmp,OS::TripleO::Services::Sshd,OS::TripleO::Services::SwiftRingBuilder,OS::TripleO::Services::SwiftStorage,OS::TripleO::Services::Timezone,OS::TripleO::Services::TripleoFirewall,OS::TripleO::Services::TripleoPackages,OS::TripleO::Services::Tuned', u'ControllerServices': u'OS::TripleO::Services::AodhApi,OS::TripleO::Services::AodhEvaluator,OS::TripleO::Services::AodhListener,OS::TripleO::Services::AodhNotifier,OS::TripleO::Services::AuditD,OS::TripleO::Services::BarbicanApi,OS::TripleO::Services::CACerts,OS::TripleO::Services::CeilometerAgentCentral,OS::TripleO::Services::CeilometerAgentNotification,OS::TripleO::Services::CeilometerApi,OS::TripleO::Services::CeilometerCollector,OS::TripleO::Services::CeilometerExpirer,OS::TripleO::Services::CephExternal,OS::TripleO::Services::CephMds,OS::TripleO::Services:.....
 'dict' object has no attribute 'status': AttributeError: 'dict' object has no attribute 'status'
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor.py", line 114, in run_action
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/validations.py", line 432, in run
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor     if self.stack and self.stack.status != 'FAILED':
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor AttributeError: 'dict' object has no attribute 'status'
2019-06-21 12:25:07.519 1959 ERROR mistral.executors.default_executor
2019-06-21 12:25:18.454 1959 INFO mistral.executors.executor_server [req-0a31b0f0-21b9-4795-90aa-5acb8199834e e05127232d9c444a82598fa4db89e38c 65950edd3cfe4e89b7215cd28f5cda77 - default default] Received RPC request 'run_action'[action_ex_id=fcf7d33a-e176-4779-8e66-3f7e8737bf79, action_cls_str=mistral.actions.std_actions.NoOpAction, action_cls_attrs={}, params={}, timeout=None]


This issue seems to be caused by this patch which merged fairly "recently": https://review.opendev.org/#/c/656379/, so I'll redirect it to DFG:DF.

It would be great if you could keep the environment for further debugging.

Created attachment 1586725 [details]
logs

Verified on:
openstack-tripleo-common-8.6.8-11.el7ost.noarch

Took a while and two deployments plus some messing around with docker image source and mirror problems. 

But I was finally able to upgrade an OSP12 to OSP13. 
Tested on two separate systems, the none HA completed the upgrade without issues. 

The HA system still waiting for it to finish, I'll update should it fail or if I hit any more issues. 

From the none HA system:

2019-07-03 15:28:14Z [AllNodesDeploySteps]: UPDATE_COMPLETE  state changed
2019-07-03 15:28:19Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully

 Stack overcloud UPDATE_COMPLETE 

Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: f41ae804-9aed-420c-b5f0-bddcf15593d9
Overcloud Endpoint: http://10.0.0.107:5000/
Overcloud Horizon Dashboard URL: http://10.0.0.107:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed
Completed Overcloud Upgrade Converge for stack overcloud


Yay... Okay to verify.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1742