rhos-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded from OSP9: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]


Environment:
openstack-tripleo-heat-templates-6.0.0-5.el7ost.noarch
instack-undercloud-6.0.0-6.el7ost.noarch
openstack-tripleo-heat-templates-compat-5.2.0-7.el7ost.noarch
openstack-puppet-modules-10.0.0-1.el7ost.noarch
python-oslo-messaging-5.17.1-2.el7ost.noarch

Steps to reproduce:
1) Deploy OSP9 with:
 Deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server 10.5.26.10 --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml


2) Upgrade the setup to OSP10

3) Attempt to upgrade to OSP11

Result:



2017-04-19 17:44:19Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps.ControllerDeployment_Step3]: CREATE_FAILED  Error: resources.ControllerDeployment_Step3.resources[0]
: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:19Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerDeployment_Step3.resources[0]: D
eployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:20Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps]: CREATE_FAILED  Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.reso
urces[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:20Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu]: UPDATE_FAILED  Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to s
erver failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:21Z [AllNodesDeploySteps]: UPDATE_FAILED  resources.AllNodesDeploySteps: Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deploym
ent to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:21Z [overcloud]: UPDATE_FAILED  resources.AllNodesDeploySteps: Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to ser
ver failed: deploy_status_code: Deployment exited with non-zero status code: 6

 Stack overcloud UPDATE_FAILED

Heat Stack update failed.


[stack@undercloud72 ~]$ openstack stack failures list overcloud                                                                                                                               
overcloud.AllNodesDeploySteps.AllNodesPostUpgradeSteps.ControllerDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: cc651233-9c9e-4e1a-8666-45f25c7a5958
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_user[placement]/ensure: created
    Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_user_role[placement@service]/ensure: created
    Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_service[placement::placement]/ensure: created
    Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_endpoint[regionOne/placement::placement]/ensure: created
    Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_user[panko]/ensure: created
    Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_user_role[panko@service]/ensure: created
    Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_service[panko::event]/ensure: created
    Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_endpoint[regionOne/panko::event]/ensure: created
    Notice: Applied catalog in 542.22 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    ...
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: Scope(Oslo::Messaging::Rabbit[keystone_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[glance_api_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[glance_registry_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[neutron_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[ceilometer_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[aodh_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
    Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: Failed to call refresh: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
    Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
    (truncated, view all with --long)