rhos-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded from OSP9: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
DescriptionAlexander Chuzhoy
2017-04-19 17:59:57 UTC
rhos-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded from OSP9: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
Environment:
openstack-tripleo-heat-templates-6.0.0-5.el7ost.noarch
instack-undercloud-6.0.0-6.el7ost.noarch
openstack-tripleo-heat-templates-compat-5.2.0-7.el7ost.noarch
openstack-puppet-modules-10.0.0-1.el7ost.noarch
python-oslo-messaging-5.17.1-2.el7ost.noarch
Steps to reproduce:
1) Deploy OSP9 with:
Deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server 10.5.26.10 --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml
2) Upgrade the setup to OSP10
3) Attempt to upgrade to OSP11
Result:
2017-04-19 17:44:19Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps.ControllerDeployment_Step3]: CREATE_FAILED Error: resources.ControllerDeployment_Step3.resources[0]
: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:19Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerDeployment_Step3.resources[0]: D
eployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:20Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps]: CREATE_FAILED Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.reso
urces[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:20Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu]: UPDATE_FAILED Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to s
erver failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:21Z [AllNodesDeploySteps]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deploym
ent to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:21Z [overcloud]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to ser
ver failed: deploy_status_code: Deployment exited with non-zero status code: 6
Stack overcloud UPDATE_FAILED
Heat Stack update failed.
[stack@undercloud72 ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.AllNodesPostUpgradeSteps.ControllerDeployment_Step3.0:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: cc651233-9c9e-4e1a-8666-45f25c7a5958
status: CREATE_FAILED
status_reason: |
Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_user[placement]/ensure: created
Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_user_role[placement@service]/ensure: created
Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_service[placement::placement]/ensure: created
Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_endpoint[regionOne/placement::placement]/ensure: created
Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_user[panko]/ensure: created
Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_user_role[panko@service]/ensure: created
Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_service[panko::event]/ensure: created
Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_endpoint[regionOne/panko::event]/ensure: created
Notice: Applied catalog in 542.22 seconds
(truncated, view all with --long)
deploy_stderr: |
...
(at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
Warning: Scope(Oslo::Messaging::Rabbit[keystone_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
Warning: Scope(Oslo::Messaging::Rabbit[glance_api_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
Warning: Scope(Oslo::Messaging::Rabbit[glance_registry_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
Warning: Scope(Oslo::Messaging::Rabbit[neutron_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
Warning: Scope(Oslo::Messaging::Rabbit[ceilometer_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
Warning: Scope(Oslo::Messaging::Rabbit[aodh_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: Failed to call refresh: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
(truncated, view all with --long)
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2017:1245