1579504 – FFU: overcloud ffwd-upgrade prepare times out; heat-engine reports several 'The Referenced Attribute is incorrect' messages

Bug 1579504 - FFU: overcloud ffwd-upgrade prepare times out; heat-engine reports several 'The Referenced Attribute is incorrect' messages

Summary: FFU: overcloud ffwd-upgrade prepare times out; heat-engine reports several 'T...

Keywords:
Status:	CLOSED DUPLICATE of bug 1561548
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	rhosp-director
Sub Component:
Version:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	---
Assignee:	RHOS Maint
QA Contact:	Amit Ugol
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-05-17 19:50 UTC by Marius Cornea
Modified:	2018-05-17 22:00 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-05-17 22:00:16 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Marius Cornea 2018-05-17 19:50:44 UTC

Description of problem:
FFU: overcloud ffwd-upgrade prepare times out and gets stuck. heat-engine reports several The Referenced Attribute is incorrect messages

Version-Release number of selected component (if applicable):
openstack-tripleo-common-8.6.1-11.el7ost.noarch
openstack-heat-api-10.0.1-0.20180411125639.825731d.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:

1. Deploys OSP10 with 3 controllers + 2 compute + 3 ceph osd nodes with predictable ips and ComputePostConfig customization:

openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/enable-cpu-pinning.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/scheduler_hints_env.yaml \
-e ~/openstack_deployment/environments/ips-from-pool-all.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
-e ~/openstack_deployment/environments/custom_hiera.yaml \
--log-file overcloud_deployment.log &> overcloud_install.log

2. Upgrade undercloud to OSP11/12/13

3. Run openstack overcloud ffwd-upgrade prepare :

openstack overcloud ffwd-upgrade prepare --yes \
            --templates /usr/share/openstack-tripleo-heat-templates \
            --stack overcloud \
            -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
            -e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \
            -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
            -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
            -e ~/openstack_deployment/environments/enable-cpu-pinning.yaml \
            -e ~/openstack_deployment/environments/nodes.yaml \
            -e ~/openstack_deployment/environments/network-environment.yaml \
            -e ~/openstack_deployment/environments/disk-layout.yaml \
            -e ~/openstack_deployment/environments/public_vip.yaml \
            -e ~/openstack_deployment/environments/enable-tls.yaml \
            -e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
            -e ~/openstack_deployment/environments/scheduler_hints_env.yaml \
            -e ~/openstack_deployment/environments/ips-from-pool-all.yaml \
            -e ~/openstack_deployment/environments/neutron-settings.yaml \
            -e ~/openstack_deployment/environments/custom_hiera.yaml \
            -e /home/stack/ffu_repos.yaml \
            --container-registry-file /home/stack/virt/docker-images.yaml \

Actual results:
Times out, gets stuck on:

(undercloud) [stack@undercloud-0 ~]$ bash overcloud_ffu_prepare.sh 
Warning! The TripleO Fast Forward Upgrade workflow is currently considered under development. In particular invocations of the ffwd-upgrade cli should be initially limited to development/test environments. Once and if you decide to use ff
wd-upgrade in production, ensure you are adequately prepared with valid backup of your current deployment state.
Continuing fast forward upgrade
Started Mistral Workflow tripleo.deployment.v1.deploy_on_servers. Execution ID: 1dea5ba0-86a4-44b1-9ffc-31b65092348d
Waiting for messages on queue 'ffwdupgrade' with no timeout.
Success
Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: 3dc2284a-b87d-4fed-ae5d-9773f478f71d
Waiting for messages on queue 'tripleo' with no timeout.
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: f0f40071-0d8b-4f48-87a7-e280d3ba2216
Plan updated.
Processing templates in the directory /tmp/tripleoclient-tQIQgM/tripleo-heat-templates
Started Mistral Workflow tripleo.plan_management.v1.get_deprecated_parameters. Execution ID: 21ab1cef-2e72-4d52-98f2-df6c8b4a1021
WARNING: Following parameters are deprecated and still defined. Deprecated parameters will be removed soon!
  OvercloudControlFlavor
  NovaComputeSchedulerHints
  NovaComputeExtraConfig
  NovaComputeIPs
WARNING: Following parameters are defined but not used in plan. Could be possible that parameter is valid but currently not used.
  StorageMgmtNetworkVip
  StorageAllocationPools
  ControlPlaneDefaultRoute
  ManagementInterfaceDefaultRoute
  SSLRootCertificate
  CephAnsiblePlaybook
  StorageMgmtNetworkVlanID
  ExternalAllocationPools
  TenantNetCidr
  SSLIntermediateCertificate
  InternalApiNetworkVlanID
  InternalApiNetCidr
  InternalApiNetworkVip
  ExternalInterfaceDefaultRoute
  ExternalNetworkVlanID
  DnsServers
  InternalApiAllocationPools
  ServiceVips
  compute_isol_cpu
  ExternalNetCidr
  CinderBackupBackend
  StorageMgmtAllocationPools
  ManagementNetCidr
  StorageNetCidr
  StorageMgmtNetCidr
  compute_root_disk
  ManagementAllocationPools
  EC2MetadataIp
  ExternalNetworkVip
  StorageNetworkVip
  SSLKey
  StorageNetworkVlanID
  TenantAllocationPools
Started Mistral Workflow tripleo.package_update.v1.package_update_plan. Execution ID: b6f1b2df-793b-43da-98e0-e1ef949ba50c
Waiting for messages on queue 'tripleo' with no timeout.
2018-05-17 19:40:26Z [HorizonSecret]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:26Z [HorizonSecret]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:27Z [DeploymentServerBlacklistDict]: CREATE_IN_PROGRESS  state changed
2018-05-17 19:40:27Z [DeploymentServerBlacklistDict]: CREATE_COMPLETE  state changed
2018-05-17 19:40:28Z [HeatAuthEncryptionKey]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:28Z [HeatAuthEncryptionKey]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:29Z [MysqlRootPassword]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:29Z [MysqlRootPassword]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:30Z [RabbitCookie]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:30Z [RabbitCookie]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:31Z [PcsdPassword]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:31Z [PcsdPassword]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:32Z [DefaultPasswords]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:32Z [Networks]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:33Z [overcloud-Networks-k7dpkgw4zhb6]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-05-17 19:40:33Z [overcloud-Networks-k7dpkgw4zhb6.StorageNetwork]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:33Z [DefaultPasswords]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:33Z [ServiceNetMap]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:34Z [overcloud-Networks-k7dpkgw4zhb6-StorageNetwork-ad6psc4uslss]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-05-17 19:40:34Z [overcloud-ServiceNetMap-dwu6cywxcqn7]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-05-17 19:40:34Z [overcloud-Networks-k7dpkgw4zhb6-StorageNetwork-ad6psc4uslss.StorageSubnet]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:34Z [overcloud-ServiceNetMap-dwu6cywxcqn7.ServiceNetMapValue]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:34Z [overcloud-ServiceNetMap-dwu6cywxcqn7.ServiceNetMapValue]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:34Z [overcloud-Networks-k7dpkgw4zhb6.TenantNetwork]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:34Z [overcloud-ServiceNetMap-dwu6cywxcqn7]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6-TenantNetwork-nx3aqib647cz]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-05-17 19:40:35Z [ServiceNetMap]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6-StorageNetwork-ad6psc4uslss.StorageSubnet]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6-TenantNetwork-nx3aqib647cz.TenantSubnet]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6-StorageNetwork-ad6psc4uslss]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-05-17 19:40:35Z [overcloud-Networks-k7dpkgw4zhb6.InternalNetwork]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6.StorageNetwork]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6-TenantNetwork-nx3aqib647cz.TenantSubnet]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6.ExternalNetwork]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6-InternalNetwork-lejsgtuaqsbd]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6-TenantNetwork-nx3aqib647cz]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-05-17 19:40:36Z [overcloud-Networks-k7dpkgw4zhb6-InternalNetwork-lejsgtuaqsbd.InternalApiSubnet]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:37Z [overcloud-Networks-k7dpkgw4zhb6.TenantNetwork]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:37Z [overcloud-Networks-k7dpkgw4zhb6-ExternalNetwork-gphk5tswlht5]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-05-17 19:40:37Z [overcloud-Networks-k7dpkgw4zhb6-InternalNetwork-lejsgtuaqsbd.InternalApiSubnet]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:37Z [overcloud-Networks-k7dpkgw4zhb6-InternalNetwork-lejsgtuaqsbd]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-05-17 19:40:38Z [overcloud-Networks-k7dpkgw4zhb6.StorageMgmtNetwork]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:38Z [overcloud-Networks-k7dpkgw4zhb6.InternalNetwork]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:39Z [overcloud-Networks-k7dpkgw4zhb6-StorageMgmtNetwork-ukdxmtjlgsa4]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-05-17 19:40:39Z [overcloud-Networks-k7dpkgw4zhb6-StorageMgmtNetwork-ukdxmtjlgsa4.StorageMgmtSubnet]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:39Z [overcloud-Networks-k7dpkgw4zhb6.ManagementNetwork]: UPDATE_IN_PROGRESS  state changed
2018-05-17 19:40:40Z [overcloud-Networks-k7dpkgw4zhb6-ManagementNetwork-zal6p43y7hyw]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-05-17 19:40:40Z [overcloud-Networks-k7dpkgw4zhb6-StorageMgmtNetwork-ukdxmtjlgsa4.StorageMgmtSubnet]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:40Z [overcloud-Networks-k7dpkgw4zhb6-StorageMgmtNetwork-ukdxmtjlgsa4]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-05-17 19:40:41Z [overcloud-Networks-k7dpkgw4zhb6.StorageMgmtNetwork]: UPDATE_COMPLETE  state changed
2018-05-17 19:40:41Z [overcloud-Networks-k7dpkgw4zhb6-ManagementNetwork-zal6p43y7hyw]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-05-17 19:40:42Z [overcloud-Networks-k7dpkgw4zhb6.ManagementNetwork]: UPDATE_COMPLETE  state changed


Expected results:
overcloud ffwd-upgrade prepare times out completes fine.


Additional info:
Attaching sosreport and templates used for deployment.

Comment 1 Marius Cornea 2018-05-17 22:00:16 UTC


*** This bug has been marked as a duplicate of bug 1561548 ***

Note You need to log in before you can comment on or make changes to this bug.