Bug 1443686 - rhos-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded from OSP9: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
Summary: rhos-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded from ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-gnocchi
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 11.0 (Ocata)
Assignee: Pradeep Kilambi
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On: 1443735
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-19 17:59 UTC by Alexander Chuzhoy
Modified: 2017-05-17 20:23 UTC (History)
15 users (show)

Fixed In Version: openstack-gnocchi-3.1.2-5.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1443735 (view as bug list)
Environment:
Last Closed: 2017-05-17 20:23:05 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC

Description Alexander Chuzhoy 2017-04-19 17:59:57 UTC
rhos-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded from OSP9: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]


Environment:
openstack-tripleo-heat-templates-6.0.0-5.el7ost.noarch
instack-undercloud-6.0.0-6.el7ost.noarch
openstack-tripleo-heat-templates-compat-5.2.0-7.el7ost.noarch
openstack-puppet-modules-10.0.0-1.el7ost.noarch
python-oslo-messaging-5.17.1-2.el7ost.noarch

Steps to reproduce:
1) Deploy OSP9 with:
 Deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server 10.5.26.10 --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml


2) Upgrade the setup to OSP10

3) Attempt to upgrade to OSP11

Result:



2017-04-19 17:44:19Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps.ControllerDeployment_Step3]: CREATE_FAILED  Error: resources.ControllerDeployment_Step3.resources[0]
: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:19Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerDeployment_Step3.resources[0]: D
eployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:20Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu.AllNodesPostUpgradeSteps]: CREATE_FAILED  Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.reso
urces[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:20Z [overcloud-AllNodesDeploySteps-j5zdjt3ghxuu]: UPDATE_FAILED  Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to s
erver failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:21Z [AllNodesDeploySteps]: UPDATE_FAILED  resources.AllNodesDeploySteps: Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deploym
ent to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-04-19 17:44:21Z [overcloud]: UPDATE_FAILED  resources.AllNodesDeploySteps: Error: resources.AllNodesPostUpgradeSteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to ser
ver failed: deploy_status_code: Deployment exited with non-zero status code: 6

 Stack overcloud UPDATE_FAILED

Heat Stack update failed.


[stack@undercloud72 ~]$ openstack stack failures list overcloud                                                                                                                               
overcloud.AllNodesDeploySteps.AllNodesPostUpgradeSteps.ControllerDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: cc651233-9c9e-4e1a-8666-45f25c7a5958
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_user[placement]/ensure: created
    Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_user_role[placement@service]/ensure: created
    Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_service[placement::placement]/ensure: created
    Notice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_endpoint[regionOne/placement::placement]/ensure: created
    Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_user[panko]/ensure: created
    Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_user_role[panko@service]/ensure: created
    Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_service[panko::event]/ensure: created
    Notice: /Stage[main]/Panko::Keystone::Auth/Keystone::Resource::Service_identity[panko]/Keystone_endpoint[regionOne/panko::event]/ensure: created
    Notice: Applied catalog in 542.22 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    ...
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: Scope(Oslo::Messaging::Rabbit[keystone_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[glance_api_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[glance_registry_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[neutron_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[ceilometer_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[aodh_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
    Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: Failed to call refresh: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
    Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage returned 1 instead of one of [0]
    (truncated, view all with --long)

Comment 1 Pradeep Kilambi 2017-04-19 18:16:23 UTC
The issue is due to bad alembic migration

http://pastebin.test.redhat.com/476597

Comment 3 Sofer Athlan-Guyot 2017-04-21 07:02:00 UTC
This is a (for the moment ?) downstream only patch and hasn't been ported back to upstream stable/ocata

Comment 9 errata-xmlrpc 2017-05-17 20:23:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.