Bug 1579504
| Summary: | FFU: overcloud ffwd-upgrade prepare times out; heat-engine reports several 'The Referenced Attribute is incorrect' messages | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> |
| Component: | rhosp-director | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | Amit Ugol <augol> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 13.0 (Queens) | CC: | dbecker, mburns, morazi |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-05-17 22:00:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 1561548 *** |
Description of problem: FFU: overcloud ffwd-upgrade prepare times out and gets stuck. heat-engine reports several The Referenced Attribute is incorrect messages Version-Release number of selected component (if applicable): openstack-tripleo-common-8.6.1-11.el7ost.noarch openstack-heat-api-10.0.1-0.20180411125639.825731d.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploys OSP10 with 3 controllers + 2 compute + 3 ceph osd nodes with predictable ips and ComputePostConfig customization: openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \ -e ~/openstack_deployment/environments/enable-cpu-pinning.yaml \ -e ~/openstack_deployment/environments/nodes.yaml \ -e ~/openstack_deployment/environments/network-environment.yaml \ -e ~/openstack_deployment/environments/disk-layout.yaml \ -e ~/openstack_deployment/environments/public_vip.yaml \ -e ~/openstack_deployment/environments/enable-tls.yaml \ -e ~/openstack_deployment/environments/inject-trust-anchor.yaml \ -e ~/openstack_deployment/environments/scheduler_hints_env.yaml \ -e ~/openstack_deployment/environments/ips-from-pool-all.yaml \ -e ~/openstack_deployment/environments/neutron-settings.yaml \ -e ~/openstack_deployment/environments/custom_hiera.yaml \ --log-file overcloud_deployment.log &> overcloud_install.log 2. Upgrade undercloud to OSP11/12/13 3. Run openstack overcloud ffwd-upgrade prepare : openstack overcloud ffwd-upgrade prepare --yes \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \ -e ~/openstack_deployment/environments/enable-cpu-pinning.yaml \ -e ~/openstack_deployment/environments/nodes.yaml \ -e ~/openstack_deployment/environments/network-environment.yaml \ -e ~/openstack_deployment/environments/disk-layout.yaml \ -e ~/openstack_deployment/environments/public_vip.yaml \ -e ~/openstack_deployment/environments/enable-tls.yaml \ -e ~/openstack_deployment/environments/inject-trust-anchor.yaml \ -e ~/openstack_deployment/environments/scheduler_hints_env.yaml \ -e ~/openstack_deployment/environments/ips-from-pool-all.yaml \ -e ~/openstack_deployment/environments/neutron-settings.yaml \ -e ~/openstack_deployment/environments/custom_hiera.yaml \ -e /home/stack/ffu_repos.yaml \ --container-registry-file /home/stack/virt/docker-images.yaml \ Actual results: Times out, gets stuck on: (undercloud) [stack@undercloud-0 ~]$ bash overcloud_ffu_prepare.sh Warning! The TripleO Fast Forward Upgrade workflow is currently considered under development. In particular invocations of the ffwd-upgrade cli should be initially limited to development/test environments. Once and if you decide to use ff wd-upgrade in production, ensure you are adequately prepared with valid backup of your current deployment state. Continuing fast forward upgrade Started Mistral Workflow tripleo.deployment.v1.deploy_on_servers. Execution ID: 1dea5ba0-86a4-44b1-9ffc-31b65092348d Waiting for messages on queue 'ffwdupgrade' with no timeout. Success Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: 3dc2284a-b87d-4fed-ae5d-9773f478f71d Waiting for messages on queue 'tripleo' with no timeout. Removing the current plan files Uploading new plan files Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: f0f40071-0d8b-4f48-87a7-e280d3ba2216 Plan updated. Processing templates in the directory /tmp/tripleoclient-tQIQgM/tripleo-heat-templates Started Mistral Workflow tripleo.plan_management.v1.get_deprecated_parameters. Execution ID: 21ab1cef-2e72-4d52-98f2-df6c8b4a1021 WARNING: Following parameters are deprecated and still defined. Deprecated parameters will be removed soon! OvercloudControlFlavor NovaComputeSchedulerHints NovaComputeExtraConfig NovaComputeIPs WARNING: Following parameters are defined but not used in plan. Could be possible that parameter is valid but currently not used. StorageMgmtNetworkVip StorageAllocationPools ControlPlaneDefaultRoute ManagementInterfaceDefaultRoute SSLRootCertificate CephAnsiblePlaybook StorageMgmtNetworkVlanID ExternalAllocationPools TenantNetCidr SSLIntermediateCertificate InternalApiNetworkVlanID InternalApiNetCidr InternalApiNetworkVip ExternalInterfaceDefaultRoute ExternalNetworkVlanID DnsServers InternalApiAllocationPools ServiceVips compute_isol_cpu ExternalNetCidr CinderBackupBackend StorageMgmtAllocationPools ManagementNetCidr StorageNetCidr StorageMgmtNetCidr compute_root_disk ManagementAllocationPools EC2MetadataIp ExternalNetworkVip StorageNetworkVip SSLKey StorageNetworkVlanID TenantAllocationPools Started Mistral Workflow tripleo.package_update.v1.package_update_plan. Execution ID: b6f1b2df-793b-43da-98e0-e1ef949ba50c Waiting for messages on queue 'tripleo' with no timeout. 2018-05-17 19:40:26Z [HorizonSecret]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:26Z [HorizonSecret]: UPDATE_COMPLETE state changed 2018-05-17 19:40:27Z [DeploymentServerBlacklistDict]: CREATE_IN_PROGRESS state changed 2018-05-17 19:40:27Z [DeploymentServerBlacklistDict]: CREATE_COMPLETE state changed 2018-05-17 19:40:28Z [HeatAuthEncryptionKey]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:28Z [HeatAuthEncryptionKey]: UPDATE_COMPLETE state changed 2018-05-17 19:40:29Z [MysqlRootPassword]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:29Z [MysqlRootPassword]: UPDATE_COMPLETE state changed 2018-05-17 19:40:30Z [RabbitCookie]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:30Z [RabbitCookie]: UPDATE_COMPLETE state changed 2018-05-17 19:40:31Z [PcsdPassword]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:31Z [PcsdPassword]: UPDATE_COMPLETE state changed 2018-05-17 19:40:32Z [DefaultPasswords]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:32Z [Networks]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:33Z [overcloud-Networks-k7dpkgw4zhb6]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-05-17 19:40:33Z [overcloud-Networks-k7dpkgw4zhb6.StorageNetwork]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:33Z [DefaultPasswords]: UPDATE_COMPLETE state changed 2018-05-17 19:40:33Z [ServiceNetMap]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:34Z [overcloud-Networks-k7dpkgw4zhb6-StorageNetwork-ad6psc4uslss]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-05-17 19:40:34Z [overcloud-ServiceNetMap-dwu6cywxcqn7]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-05-17 19:40:34Z [overcloud-Networks-k7dpkgw4zhb6-StorageNetwork-ad6psc4uslss.StorageSubnet]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:34Z [overcloud-ServiceNetMap-dwu6cywxcqn7.ServiceNetMapValue]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:34Z [overcloud-ServiceNetMap-dwu6cywxcqn7.ServiceNetMapValue]: UPDATE_COMPLETE state changed 2018-05-17 19:40:34Z [overcloud-Networks-k7dpkgw4zhb6.TenantNetwork]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:34Z [overcloud-ServiceNetMap-dwu6cywxcqn7]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6-TenantNetwork-nx3aqib647cz]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-05-17 19:40:35Z [ServiceNetMap]: UPDATE_COMPLETE state changed 2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6-StorageNetwork-ad6psc4uslss.StorageSubnet]: UPDATE_COMPLETE state changed 2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6-TenantNetwork-nx3aqib647cz.TenantSubnet]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6-StorageNetwork-ad6psc4uslss]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6.InternalNetwork]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6.StorageNetwork]: UPDATE_COMPLETE state changed 2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6-TenantNetwork-nx3aqib647cz.TenantSubnet]: UPDATE_COMPLETE state changed 2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6.ExternalNetwork]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6-InternalNetwork-lejsgtuaqsbd]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6-TenantNetwork-nx3aqib647cz]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6-InternalNetwork-lejsgtuaqsbd.InternalApiSubnet]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:37Z [overcloud-Networks-k7dpkgw4zhb6.TenantNetwork]: UPDATE_COMPLETE state changed 2018-05-17 19:40:37Z [overcloud-Networks-k7dpkgw4zhb6-ExternalNetwork-gphk5tswlht5]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-05-17 19:40:37Z [overcloud-Networks-k7dpkgw4zhb6-InternalNetwork-lejsgtuaqsbd.InternalApiSubnet]: UPDATE_COMPLETE state changed 2018-05-17 19:40:37Z [overcloud-Networks-k7dpkgw4zhb6-InternalNetwork-lejsgtuaqsbd]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-05-17 19:40:38Z [overcloud-Networks-k7dpkgw4zhb6.StorageMgmtNetwork]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:38Z [overcloud-Networks-k7dpkgw4zhb6.InternalNetwork]: UPDATE_COMPLETE state changed 2018-05-17 19:40:39Z [overcloud-Networks-k7dpkgw4zhb6-StorageMgmtNetwork-ukdxmtjlgsa4]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-05-17 19:40:39Z [overcloud-Networks-k7dpkgw4zhb6-StorageMgmtNetwork-ukdxmtjlgsa4.StorageMgmtSubnet]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:39Z [overcloud-Networks-k7dpkgw4zhb6.ManagementNetwork]: UPDATE_IN_PROGRESS state changed 2018-05-17 19:40:40Z [overcloud-Networks-k7dpkgw4zhb6-ManagementNetwork-zal6p43y7hyw]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-05-17 19:40:40Z [overcloud-Networks-k7dpkgw4zhb6-StorageMgmtNetwork-ukdxmtjlgsa4.StorageMgmtSubnet]: UPDATE_COMPLETE state changed 2018-05-17 19:40:40Z [overcloud-Networks-k7dpkgw4zhb6-StorageMgmtNetwork-ukdxmtjlgsa4]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-05-17 19:40:41Z [overcloud-Networks-k7dpkgw4zhb6.StorageMgmtNetwork]: UPDATE_COMPLETE state changed 2018-05-17 19:40:41Z [overcloud-Networks-k7dpkgw4zhb6-ManagementNetwork-zal6p43y7hyw]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-05-17 19:40:42Z [overcloud-Networks-k7dpkgw4zhb6.ManagementNetwork]: UPDATE_COMPLETE state changed Expected results: overcloud ffwd-upgrade prepare times out completes fine. Additional info: Attaching sosreport and templates used for deployment.