Bug 1438608

Summary: gnocchi-upgrade fails on deploy
Product: Red Hat OpenStack Reporter: Yariv <yrachman>
Component: puppet-tripleoAssignee: RHOS Maint <rhos-maint>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: high    
Version: 11.0 (Ocata)CC: achernet, aschultz, atelang, augol, edannon, jbiao, jdanjou, jjoyce, jschluet, mbabushk, mfuruta, mihai, mlopes, mnadeem, pgsousa, pkilambi, rhel-osp-director-maint, rscarazz, sathlang, slinaber, tvignaud, vchundur, wliu, yrachman
Target Milestone: z4Keywords: Reopened, Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: puppet-tripleo-6.5.4-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-13 16:31:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1539616, 1544802    
Bug Blocks:    
Attachments:
Description Flags
RHOS-11-deploy-files
none
sos-report-controller-0 none

Description Yariv 2017-04-03 23:12:49 UTC 
Created attachment 1268488 [details]
RHOS-11-deploy-files

Description of problem:

Deploying OVS+DPDK with Linux 

Version-Release number of selected component (if applicable):

RHOS 11 
http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/2017-03-30.4/

How reproducible:
Permanent

Steps to Reproduce:
1. use the attached yaml 
2.
3.

Actual results:

overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: acdafd2a-96a3-4f36-bb4b-c244be6b0b37
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.



Expected results:

Over cloud should pass with success as previous puddels

Additional info:
Comment 1 Yariv 2017-04-03 23:41:50 UTC 
Created attachment 1268490 [details]
sos-report-controller-0
Comment 2 Saravanan KR 2017-04-04 06:27:26 UTC 
I did a quick check on the environment. Controller is failing on Step5 of puppet apply:

overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
    Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]
    Error: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]
    Error: gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf returned 1 instead of one of [0]
    Error: /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[run gnocchi upgrade with storage]/returns: change from notrun to 0 failed: gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf 



ceilometer-upgrade is failing. This is the recent backport for this upgrade code addition - https://review.openstack.org/#/c/447735.



ceilometer-upgrade.log:
-----------------------
2017-04-03 21:58:14.376 84782 INFO ceilometer.cmd.storage [-] Skipping metering database upgrade
2017-04-03 21:58:16.830 84782 CRITICAL ceilometer [-] ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
 (HTTP 500)
2017-04-03 21:58:16.830 84782 ERROR ceilometer Traceback (most recent call last):
2017-04-03 21:58:16.830 84782 ERROR ceilometer   File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2017-04-03 21:58:16.830 84782 ERROR ceilometer     sys.exit(upgrade())
2017-04-03 21:58:16.830 84782 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py", line 53, in upgrade
2017-04-03 21:58:16.830 84782 ERROR ceilometer     gnocchi_client.upgrade_resource_types(conf)
2017-04-03 21:58:16.830 84782 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/gnocchi_client.py", line 113, in upgrade_resource_types
2017-04-03 21:58:16.830 84782 ERROR ceilometer     gnocchi.resource_type.get(name=name)
2017-04-03 21:58:16.830 84782 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/resource_type.py", line 44, in get
2017-04-03 21:58:16.830 84782 ERROR ceilometer     headers={'Content-Type': "application/json"}).json()
2017-04-03 21:58:16.830 84782 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line 37, in _get
2017-04-03 21:58:16.830 84782 ERROR ceilometer     return self.client.api.get(*args, **kwargs)
2017-04-03 21:58:16.830 84782 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 217, in get
2017-04-03 21:58:16.830 84782 ERROR ceilometer     return self.request(url, 'GET', **kwargs)
2017-04-03 21:58:16.830 84782 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/gnocchiclient/client.py", line 38, in request
2017-04-03 21:58:16.830 84782 ERROR ceilometer     raise exceptions.from_response(resp, method)
Comment 3 Sofer Athlan-Guyot 2017-04-04 07:22:41 UTC 
Hi,

Did a quick pass on this one.

This is a one controller node deployment, so it doesn't involve complicated orchestration issue.

From the sos report, we can see that the db and haproxy are currently working properly, so it's not this kind of error.
Comment 4 Pradeep Kilambi 2017-04-04 15:14:32 UTC 
Can you provide more info on this. This passes upstream stable/ocata CI and we deployed the latest puddle and its working fine. Is there anything different with ovs +dpdk tripleo deployment?
Comment 5 Pradeep Kilambi 2017-04-04 15:21:32 UTC 
Looking at the sos report, I dont see any gnocchi config files or logs.

$ ls sosreport-YarivRachmani-20170403230530/etc/gnocchi*
ls: cannot access 'sosreport-YarivRachmani-20170403230530/etc/gnocchi*': No such file or directory

$ ls sosreport-YarivRachmani-20170403230530/var/log/gnocchi*
ls: cannot access 'sosreport-YarivRachmani-20170403230530/var/log/gnocchi*': No such file or directory

gnocchi api should have been configured in step 4. So something is weird in your deploy. Which explains why ceilometer upgrade is failing. Are you not configuring gnocchi/aodh/panko in your deployment?
Comment 6 Yariv 2017-04-04 20:07:32 UTC 
(In reply to Pradeep Kilambi from comment #5)
> Looking at the sos report, I dont see any gnocchi config files or logs.
> 
> $ ls sosreport-YarivRachmani-20170403230530/etc/gnocchi*
> ls: cannot access 'sosreport-YarivRachmani-20170403230530/etc/gnocchi*': No
> such file or directory
> 
> $ ls sosreport-YarivRachmani-20170403230530/var/log/gnocchi*
> ls: cannot access 'sosreport-YarivRachmani-20170403230530/var/log/gnocchi*':
> No such file or directory
> 
> gnocchi api should have been configured in step 4. So something is weird in
> your deploy. Which explains why ceilometer upgrade is failing. Are you not
> configuring gnocchi/aodh/panko in your deployment?

No.. i had THT files from RHOS 10.. is there something new that i am not aware of regards gnocchi/aodh/panko

In case i do not want to use it.. is there some settings to disable in network-environment.yaml?
Comment 7 Pradeep Kilambi 2017-04-04 20:41:07 UTC 
(In reply to Yariv from comment #6)
> (In reply to Pradeep Kilambi from comment #5)
> > Looking at the sos report, I dont see any gnocchi config files or logs.
> > 
> > $ ls sosreport-YarivRachmani-20170403230530/etc/gnocchi*
> > ls: cannot access 'sosreport-YarivRachmani-20170403230530/etc/gnocchi*': No
> > such file or directory
> > 
> > $ ls sosreport-YarivRachmani-20170403230530/var/log/gnocchi*
> > ls: cannot access 'sosreport-YarivRachmani-20170403230530/var/log/gnocchi*':
> > No such file or directory
> > 
> > gnocchi api should have been configured in step 4. So something is weird in
> > your deploy. Which explains why ceilometer upgrade is failing. Are you not
> > configuring gnocchi/aodh/panko in your deployment?
> 
> No.. i had THT files from RHOS 10.. is there something new that i am not
> aware of regards gnocchi/aodh/panko

There are a lot of new changes that went into OSP11. So its better you use OSP 11 heat templates.

> 
> In case i do not want to use it.. is there some settings to disable in
> network-environment.yaml?

If you dont want to deploy any of the Telemetry Services, you can use composable roles and create an env file and disable resource registry for the services you want to skip by setting OS::Heat::None.
Comment 8 Yariv 2017-04-13 12:02:15 UTC 
It seems related to Director deployment with ControlPlane LinuxBond
It disappeared.. during new puddle release http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/2017-04-04.3/
Comment 9 Raoul Scarazzini 2017-05-25 12:12:10 UTC 
I hit this today, while testing OSP 11 (Puddle 2017-05-09.2) on a composable roles deployment and took sosreports [1]. The machine that hit the error is overcloud-controller-0, so the sosreport to take a look at is sosreport-controller-0.localdomain-20170525102419.tar.xz.

This issue is a race, I deployed several times the same exact environment without hitting the issue, so I can't say how this is reproducible.

[1] http://file.rdu.redhat.com/~rscarazz/BZ1437554/
Comment 10 Raoul Scarazzini 2017-05-26 13:02:41 UTC 
Increasing priority and severity because I'm hitting this more frequently, 1 every 3 deployments.
Comment 11 Pradeep Kilambi 2017-05-26 14:30:22 UTC 
Can you provide more info on what kind of deployment you are doing? Are you deploying a separate Telemetry node as composable roles? Whats your deployment configuration? Can you add your roles definition. Also i see no gnocchi logs in your sos report so hard to say whats going on.

The upstream CI ocata jobs are deploying without issues from what i see:

http://logs.openstack.org/37/467137/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/3774c89/logs/overcloud-controller-0/var/log/gnocchi/app.log.txt.gz
Comment 12 Raoul Scarazzini 2017-05-26 15:40:38 UTC 
Sure, I took new sosreports with full logs and uploaded everything here [1].

About the environment, it is a 10 nodes composable deployment in which I have configured in addition to the 3 controllers, 3 nodes for messaging and 3 nodes for galera. The last node is a ceph one. The customization about the roles are inside the "composable_roles" directory at the link.

[1] http://file.rdu.redhat.com/~rscarazz/BZ1438608/
Comment 13 Pradeep Kilambi 2017-05-26 18:12:02 UTC 
Well I still dont see gnocchi logs in your sosreport. var/log/gnocchi/* should be there, which i dont see. 

$ ls sosreport-controller-0.localdomain-20170526151227/var/log/gnocchi
ls: cannot access 'sosreport-controller-0.localdomain-20170526151227/var/log/gnocchi': No such file or directory

$ ls sosreport-controller-1.localdomain-20170526151355/var/log/gnocchi
ls: cannot access 'sosreport-controller-1.localdomain-20170526151355/var/log/gnocchi': No such file or directory

$ ls sosreport-controller-2.localdomain-20170526151524/var/log/gnocchi
ls: cannot access 'sosreport-controller-2.localdomain-20170526151524/var/log/gnocchi': No such file or directory


I did look at the heat debug logs on bootstrap node and i dont see error with gnocchi-upgrade:

2017-05-26 11:30:11 +0000 Exec[gnocchi-db-sync](provider=posix) (debug): Executing 'gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage'
2017-05-26 11:30:11 +0000 Puppet (debug): Executing with uid=gnocchi: 'gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage'
2017-05-26 11:30:12 +0000 /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync] (notice): Triggered 'refresh' from 1 events
2017-05-26 11:30:12 +0000 /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync] (info): Scheduling refresh of Anchor[gnocchi::dbsync::end]
2017-05-26 11:30:12 +0000 /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync] (debug): The container Class[Gnocchi::Db::Sync] will propagate my refresh event


2017-05-26 11:29:31 +0000 Exec[run gnocchi upgrade with storage](provider=posix) (debug): Executing 'gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf'
2017-05-26 11:29:31 +0000 Puppet (debug): Executing: 'gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf'
2017-05-26 11:29:32 +0000 /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[run gnocchi upgrade with storage]/returns (notice): executed successfully
2017-05-26 11:29:32 +0000 /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[run gnocchi upgrade with storage] (debug): The container Class[Tripleo::Profile::Base::Gnocchi::Api] will propagate my refresh event
2017-05-26 11:29:32 +0000 Class[Tripleo::Profile::Base::Gnocchi::Api] (debug): The container Stage[main] will propagate my refresh event
2017-05-26 11:29:32 +

The above indicates the gnocchi-upgrade went fine. 

At what point during the deployment do you see the error. What specific error are you seeing? can past me the error indicating gnocchi failure during your deployment. Past em the deplyment failure output. Also var/log/gnocchi/* should help in this case.
Comment 14 Raoul Scarazzini 2017-05-29 10:12:29 UTC 
For some reason the sosreport didn't include the gnocchi log files, but luckily environment was still available so I was able to take those logs and put them in the same place [1] of the others.
The point in which we see the error in the deployment is Step5:

...
...
2017-05-26 11:28:46Z [1]: SIGNAL_IN_PROGRESS  Signal: deployment 71296b30-5265-463e-a354-b279531b80e1 succeeded
2017-05-26 11:28:47Z [1]: CREATE_COMPLETE  state changed
2017-05-26 11:28:53Z [0]: SIGNAL_IN_PROGRESS  SHeat Stack create failed.
ignal: deployment 6629a82e-3742-43e8-9e4f-030dce14bafa succeeded
2017-05-26 11:28:54Z [0]: CREATE_COMPLETE  state changed
2017-05-26 11:29:03Z [0]: SIGNAL_IN_PROGRESS  Signal: deployment 94c4c696-668b-4460-b8d3-7fb8dc9e2c13 succeeded
2017-05-26 11:29:03Z [0]: CREATE_COMPLETE  state changed
2017-05-26 11:29:05Z [2]: SIGNAL_IN_PROGRESS  Signal: deployment a6de4ffb-ee54-41e1-8002-4a62e329a732 succeeded
2017-05-26 11:29:05Z [2]: CREATE_COMPLETE  state changed
2017-05-26 11:29:07Z [0]: SIGNAL_IN_PROGRESS  Signal: deployment 3d854d52-3ffb-47e8-a5ea-03b931d0f6d4 succeeded
2017-05-26 11:29:07Z [0]: CREATE_COMPLETE  state changed
2017-05-26 11:29:07Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE  Stack CREATE completed successfully
2017-05-26 11:29:08Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE  state changed
2017-05-26 11:29:11Z [1]: SIGNAL_IN_PROGRESS  Signal: deployment 9f8a9784-c7ff-4974-95d5-b19c82358acb succeeded
2017-05-26 11:29:12Z [1]: CREATE_COMPLETE  state changed
2017-05-26 11:29:12Z [overcloud.AllNodesDeploySteps.MessagingDeployment_Step5]: CREATE_COMPLETE  Stack CREATE completed successfully
2017-05-26 11:29:12Z [overcloud.AllNodesDeploySteps.MessagingDeployment_Step5]: CREATE_COMPLETE  state changed
2017-05-26 11:29:18Z [2]: SIGNAL_IN_PROGRESS  Signal: deployment 2a7dbe4d-89af-466e-b71a-8792acef94f2 succeeded
2017-05-26 11:29:18Z [2]: CREATE_COMPLETE  state changed
2017-05-26 11:29:19Z [overcloud.AllNodesDeploySteps.GaleraDeployment_Step5]: CREATE_COMPLETE  Stack CREATE completed successfully
2017-05-26 11:29:19Z [overcloud.AllNodesDeploySteps.GaleraDeployment_Step5]: CREATE_COMPLETE  state changed
2017-05-26 11:31:08Z [1]: SIGNAL_IN_PROGRESS  Signal: deployment 3f6dbad6-f15e-43a0-a3e9-9e11126b309f succeeded
2017-05-26 11:31:09Z [1]: CREATE_COMPLETE  state changed
2017-05-26 11:31:21Z [2]: SIGNAL_IN_PROGRESS  Signal: deployment 275d838d-c671-4105-a8c8-5b7d776c958b succeeded
2017-05-26 11:31:22Z [2]: CREATE_COMPLETE  state changed
2017-05-26 11:32:44Z [0]: SIGNAL_IN_PROGRESS  Signal: deployment a9ef26dd-c29f-4dfc-a4fd-d183efcb9706 failed (6)
2017-05-26 11:32:44Z [0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2017-05-26 11:32:44Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2017-05-26 11:32:45Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED  Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-05-26 11:32:45Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-05-26 11:32:46Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-05-26 11:32:46Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6

 Stack overcloud CREATE_FAILED 

The error I see inside controller-0 is this one:

Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]

Let me know if you need something else.

[1] http://file.rdu.redhat.com/~rscarazz/BZ1438608/
Comment 15 James Biao 2017-06-14 00:46:38 UTC 
Hello,

I'm hitting this issue on a support case. Customer deployed 1 controller node and 2 compute nodes. Retried the deployment once and still the same issue.

[stack@director10 ~]$ openstack overcloud deploy --templates templates/openstack-tripleo-heat-templates/ -e templates/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e templates/network-environment.yaml  --control-scale 1 --compute-scale 2 --control-flavor control --compute-flavor compute   --ntp-server 192.0.3.1 

2017-06-12 09:10:43Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5]: CREATE_COMPLETE  state changed
2017-06-12 09:13:57Z [0]: SIGNAL_IN_PROGRESS  Signal: deployment 4ac95325-4614-4b95-98fd-576a064ec66f failed (6)
2017-06-12 09:13:57Z [0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2017-06-12 09:13:57Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2017-06-12 09:13:57Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED  Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-06-12 09:13:58Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-06-12 09:13:58Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-06-12 09:13:58Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6

[stack@director10 ~]$ heat deployment-show  4ac95325-4614-4b95-98fd-576a064ec66f
oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\\transport_url. Please use oslo::messaging::default::transport_url instead.\u001b[0m\n\u001b[1;33mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;33mWarning: [norpm] Attempting to uninstall openstack-nova-migration but it will not be removed\u001b[0m\n\u001b[1;31m
    Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31m
    Error: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31m
    Error: gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[run gnocchi upgrade with storage]/returns: change from notrun to 0 failed: gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf returned 1 instead of one of [0]\u001b[0m\n", 
    "deploy_status_code": 6
  },

gnocchi error log

2017-06-13 06:53:42.441 934382 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://192.168.32.16:5000/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:87
2017-06-13 06:53:42.605 934382 CRITICAL gnocchi [-] ConnectionError: ('Connection aborted.', BadStatusLine("''",))
2017-06-13 06:53:42.605 934382 ERROR gnocchi Traceback (most recent call last):
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/bin/gnocchi-upgrade", line 10, in <module>
2017-06-13 06:53:42.605 934382 ERROR gnocchi     sys.exit(upgrade())
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 70, in upgrade
2017-06-13 06:53:42.605 934382 ERROR gnocchi     s = storage.get_driver(conf)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 144, in get_driver
2017-06-13 06:53:42.605 934382 ERROR gnocchi     conf.incoming)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/incoming/swift.py", line 35, in __init__
2017-06-13 06:53:42.605 934382 ERROR gnocchi     self.swift.put_container(self.MEASURE_PREFIX)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1755, in put_container
2017-06-13 06:53:42.605 934382 ERROR gnocchi     query_string=query_string)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1673, in _retry
2017-06-13 06:53:42.605 934382 ERROR gnocchi     service_token=self.service_token, **kwargs)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1014, in put_container
2017-06-13 06:53:42.605 934382 ERROR gnocchi     conn.request(method, path, '', headers)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 436, in request
2017-06-13 06:53:42.605 934382 ERROR gnocchi     files=files, **self.requests_args)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 419, in _request
2017-06-13 06:53:42.605 934382 ERROR gnocchi     return self.request_session.request(*arg, **kwarg)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
2017-06-13 06:53:42.605 934382 ERROR gnocchi     resp = self.send(prep, **send_kwargs)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 596, in send
2017-06-13 06:53:42.605 934382 ERROR gnocchi     r = adapter.send(request, **kwargs)
2017-06-13 06:53:42.605 934382 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
2017-06-13 06:53:42.605 934382 ERROR gnocchi     raise ConnectionError(err, request=request)
2017-06-13 06:53:42.605 934382 ERROR gnocchi ConnectionError: ('Connection aborted.', BadStatusLine("''",))
2017-06-13 06:53:42.605 934382 ERROR gnocchi
Comment 26 Md Nadeem 2017-08-07 10:10:11 UTC 
IHAC, who is facing the same issue (case #01905549). he claims the overcloud deployment completed successfully with 1 controller & 1 compute , however it gets failed with 3 controller .

mError: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]\u001b[0m\n



var/log/ceilometer/./collector.log:13:2017-08-06 02:30:25.051 150843 ERROR ceilometer.dispatcher.gnocchi [-] <html><body><h1>504 Gateway Time-out</h1>
var/log/ceilometer/./collector.log:17:2017-08-06 02:40:26.026 150843 ERROR ceilometer.dispatcher.gnocchi [-] <html><body><h1>504 Gateway Time-out</h1>
var/log/ceilometer/./collector.log:21:2017-08-06 02:50:26.165 150843 ERROR ceilometer.dispatcher.gnocchi [-] <html><body><h1>504 Gateway Time-out</h1>
var/log/ceilometer/./collector.log:25:2017-08-06 03:00:26.139 150843 ERROR ceilometer.dispatcher.gnocchi [-] <html><body><h1>504 Gateway Time-out</h1>
var/log/ceilometer/./ceilometer-upgrade.log:11:<title>500 Internal Server Error</title>
var/log/ceilometer/./ceilometer-upgrade.log:13:<h1>Internal Server Error</h1>


I have asked to re-run the test, 3 times failed with same error.
Executing command "ceilometer-upgrade --skip-metering-database" manually also re-produces the same error.
Comment 27 pgsousa 2017-08-08 15:17:45 UTC 
Hi,

FYI I'm having this error when I launch the overcloud deployment with invalid ssl certifcates:

/var/log/gnocchi/gnocchi-upgrade.log:

ERROR gnocchi SSLError: hostname '172.29.64.103' doesn't match cloud.ccom.xxx.xx'

Thanks
Comment 35 Sasha Smolyak 2018-02-08 16:13:39 UTC 
Tested on swift as well now. Looks ok
Comment 38 errata-xmlrpc 2018-02-13 16:31:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0310
Comment 39 Mihai 2018-05-04 21:50:07 UTC 
Hi,

I know this bug looks nice and closed...but I have the same behavior and same Internal Server Error 500 message in /var/log/ceilometer-upgrade.log

I am using:
RHEL 7.4
Ocata / OSP 11

Your fix is since: puppet-tripleo-6.5.4-1.el7ost
My RHEL Undercloud/Director has: 
puppet-tripleo-6.5.5-2.el7ost.noarch

I get the feeling that a regression managed to make its way into this version.
Would it be possible to check or can I help somehow?
This issue seems to be a reoccurring one.

Thanks!