Created attachment 1268488 [details] RHOS-11-deploy-files Description of problem: Deploying OVS+DPDK with Linux Version-Release number of selected component (if applicable): RHOS 11 http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/2017-03-30.4/ How reproducible: Permanent Steps to Reproduce: 1. use the attached yaml 2. 3. Actual results: overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: acdafd2a-96a3-4f36-bb4b-c244be6b0b37 status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 deploy_stdout: | Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked. Expected results: Over cloud should pass with success as previous puddels Additional info:
Created attachment 1268490 [details] sos-report-controller-0
I did a quick check on the environment. Controller is failing on Step5 of puppet apply: overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0: resource_type: OS::Heat::StructuredDeployment Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0] Error: gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[run gnocchi upgrade with storage]/returns: change from notrun to 0 failed: gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf ceilometer-upgrade is failing. This is the recent backport for this upgrade code addition - https://review.openstack.org/#/c/447735. ceilometer-upgrade.log: ----------------------- 2017-04-03 21:58:14.376 84782 INFO ceilometer.cmd.storage [-] Skipping metering database upgrade 2017-04-03 21:58:16.830 84782 CRITICAL ceilometer [-] ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> (HTTP 500) 2017-04-03 21:58:16.830 84782 ERROR ceilometer Traceback (most recent call last): 2017-04-03 21:58:16.830 84782 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module> 2017-04-03 21:58:16.830 84782 ERROR ceilometer sys.exit(upgrade()) 2017-04-03 21:58:16.830 84782 ERROR ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py", line 53, in upgrade 2017-04-03 21:58:16.830 84782 ERROR ceilometer gnocchi_client.upgrade_resource_types(conf) 2017-04-03 21:58:16.830 84782 ERROR ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/gnocchi_client.py", line 113, in upgrade_resource_types 2017-04-03 21:58:16.830 84782 ERROR ceilometer gnocchi.resource_type.get(name=name) 2017-04-03 21:58:16.830 84782 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/resource_type.py", line 44, in get 2017-04-03 21:58:16.830 84782 ERROR ceilometer headers={'Content-Type': "application/json"}).json() 2017-04-03 21:58:16.830 84782 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line 37, in _get 2017-04-03 21:58:16.830 84782 ERROR ceilometer return self.client.api.get(*args, **kwargs) 2017-04-03 21:58:16.830 84782 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 217, in get 2017-04-03 21:58:16.830 84782 ERROR ceilometer return self.request(url, 'GET', **kwargs) 2017-04-03 21:58:16.830 84782 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/client.py", line 38, in request 2017-04-03 21:58:16.830 84782 ERROR ceilometer raise exceptions.from_response(resp, method)
Hi, Did a quick pass on this one. This is a one controller node deployment, so it doesn't involve complicated orchestration issue. From the sos report, we can see that the db and haproxy are currently working properly, so it's not this kind of error.
Can you provide more info on this. This passes upstream stable/ocata CI and we deployed the latest puddle and its working fine. Is there anything different with ovs +dpdk tripleo deployment?
Looking at the sos report, I dont see any gnocchi config files or logs. $ ls sosreport-YarivRachmani-20170403230530/etc/gnocchi* ls: cannot access 'sosreport-YarivRachmani-20170403230530/etc/gnocchi*': No such file or directory $ ls sosreport-YarivRachmani-20170403230530/var/log/gnocchi* ls: cannot access 'sosreport-YarivRachmani-20170403230530/var/log/gnocchi*': No such file or directory gnocchi api should have been configured in step 4. So something is weird in your deploy. Which explains why ceilometer upgrade is failing. Are you not configuring gnocchi/aodh/panko in your deployment?
(In reply to Pradeep Kilambi from comment #5) > Looking at the sos report, I dont see any gnocchi config files or logs. > > $ ls sosreport-YarivRachmani-20170403230530/etc/gnocchi* > ls: cannot access 'sosreport-YarivRachmani-20170403230530/etc/gnocchi*': No > such file or directory > > $ ls sosreport-YarivRachmani-20170403230530/var/log/gnocchi* > ls: cannot access 'sosreport-YarivRachmani-20170403230530/var/log/gnocchi*': > No such file or directory > > gnocchi api should have been configured in step 4. So something is weird in > your deploy. Which explains why ceilometer upgrade is failing. Are you not > configuring gnocchi/aodh/panko in your deployment? No.. i had THT files from RHOS 10.. is there something new that i am not aware of regards gnocchi/aodh/panko In case i do not want to use it.. is there some settings to disable in network-environment.yaml?
(In reply to Yariv from comment #6) > (In reply to Pradeep Kilambi from comment #5) > > Looking at the sos report, I dont see any gnocchi config files or logs. > > > > $ ls sosreport-YarivRachmani-20170403230530/etc/gnocchi* > > ls: cannot access 'sosreport-YarivRachmani-20170403230530/etc/gnocchi*': No > > such file or directory > > > > $ ls sosreport-YarivRachmani-20170403230530/var/log/gnocchi* > > ls: cannot access 'sosreport-YarivRachmani-20170403230530/var/log/gnocchi*': > > No such file or directory > > > > gnocchi api should have been configured in step 4. So something is weird in > > your deploy. Which explains why ceilometer upgrade is failing. Are you not > > configuring gnocchi/aodh/panko in your deployment? > > No.. i had THT files from RHOS 10.. is there something new that i am not > aware of regards gnocchi/aodh/panko There are a lot of new changes that went into OSP11. So its better you use OSP 11 heat templates. > > In case i do not want to use it.. is there some settings to disable in > network-environment.yaml? If you dont want to deploy any of the Telemetry Services, you can use composable roles and create an env file and disable resource registry for the services you want to skip by setting OS::Heat::None.
It seems related to Director deployment with ControlPlane LinuxBond It disappeared.. during new puddle release http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/2017-04-04.3/
I hit this today, while testing OSP 11 (Puddle 2017-05-09.2) on a composable roles deployment and took sosreports [1]. The machine that hit the error is overcloud-controller-0, so the sosreport to take a look at is sosreport-controller-0.localdomain-20170525102419.tar.xz. This issue is a race, I deployed several times the same exact environment without hitting the issue, so I can't say how this is reproducible. [1] http://file.rdu.redhat.com/~rscarazz/BZ1437554/
Increasing priority and severity because I'm hitting this more frequently, 1 every 3 deployments.
Can you provide more info on what kind of deployment you are doing? Are you deploying a separate Telemetry node as composable roles? Whats your deployment configuration? Can you add your roles definition. Also i see no gnocchi logs in your sos report so hard to say whats going on. The upstream CI ocata jobs are deploying without issues from what i see: http://logs.openstack.org/37/467137/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/3774c89/logs/overcloud-controller-0/var/log/gnocchi/app.log.txt.gz
Sure, I took new sosreports with full logs and uploaded everything here [1]. About the environment, it is a 10 nodes composable deployment in which I have configured in addition to the 3 controllers, 3 nodes for messaging and 3 nodes for galera. The last node is a ceph one. The customization about the roles are inside the "composable_roles" directory at the link. [1] http://file.rdu.redhat.com/~rscarazz/BZ1438608/
Well I still dont see gnocchi logs in your sosreport. var/log/gnocchi/* should be there, which i dont see. $ ls sosreport-controller-0.localdomain-20170526151227/var/log/gnocchi ls: cannot access 'sosreport-controller-0.localdomain-20170526151227/var/log/gnocchi': No such file or directory $ ls sosreport-controller-1.localdomain-20170526151355/var/log/gnocchi ls: cannot access 'sosreport-controller-1.localdomain-20170526151355/var/log/gnocchi': No such file or directory $ ls sosreport-controller-2.localdomain-20170526151524/var/log/gnocchi ls: cannot access 'sosreport-controller-2.localdomain-20170526151524/var/log/gnocchi': No such file or directory I did look at the heat debug logs on bootstrap node and i dont see error with gnocchi-upgrade: 2017-05-26 11:30:11 +0000 Exec[gnocchi-db-sync](provider=posix) (debug): Executing 'gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage' 2017-05-26 11:30:11 +0000 Puppet (debug): Executing with uid=gnocchi: 'gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage' 2017-05-26 11:30:12 +0000 /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync] (notice): Triggered 'refresh' from 1 events 2017-05-26 11:30:12 +0000 /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync] (info): Scheduling refresh of Anchor[gnocchi::dbsync::end] 2017-05-26 11:30:12 +0000 /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync] (debug): The container Class[Gnocchi::Db::Sync] will propagate my refresh event 2017-05-26 11:29:31 +0000 Exec[run gnocchi upgrade with storage](provider=posix) (debug): Executing 'gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf' 2017-05-26 11:29:31 +0000 Puppet (debug): Executing: 'gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf' 2017-05-26 11:29:32 +0000 /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[run gnocchi upgrade with storage]/returns (notice): executed successfully 2017-05-26 11:29:32 +0000 /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[run gnocchi upgrade with storage] (debug): The container Class[Tripleo::Profile::Base::Gnocchi::Api] will propagate my refresh event 2017-05-26 11:29:32 +0000 Class[Tripleo::Profile::Base::Gnocchi::Api] (debug): The container Stage[main] will propagate my refresh event 2017-05-26 11:29:32 + The above indicates the gnocchi-upgrade went fine. At what point during the deployment do you see the error. What specific error are you seeing? can past me the error indicating gnocchi failure during your deployment. Past em the deplyment failure output. Also var/log/gnocchi/* should help in this case.
For some reason the sosreport didn't include the gnocchi log files, but luckily environment was still available so I was able to take those logs and put them in the same place [1] of the others. The point in which we see the error in the deployment is Step5: ... ... 2017-05-26 11:28:46Z [1]: SIGNAL_IN_PROGRESS Signal: deployment 71296b30-5265-463e-a354-b279531b80e1 succeeded 2017-05-26 11:28:47Z [1]: CREATE_COMPLETE state changed 2017-05-26 11:28:53Z [0]: SIGNAL_IN_PROGRESS SHeat Stack create failed. ignal: deployment 6629a82e-3742-43e8-9e4f-030dce14bafa succeeded 2017-05-26 11:28:54Z [0]: CREATE_COMPLETE state changed 2017-05-26 11:29:03Z [0]: SIGNAL_IN_PROGRESS Signal: deployment 94c4c696-668b-4460-b8d3-7fb8dc9e2c13 succeeded 2017-05-26 11:29:03Z [0]: CREATE_COMPLETE state changed 2017-05-26 11:29:05Z [2]: SIGNAL_IN_PROGRESS Signal: deployment a6de4ffb-ee54-41e1-8002-4a62e329a732 succeeded 2017-05-26 11:29:05Z [2]: CREATE_COMPLETE state changed 2017-05-26 11:29:07Z [0]: SIGNAL_IN_PROGRESS Signal: deployment 3d854d52-3ffb-47e8-a5ea-03b931d0f6d4 succeeded 2017-05-26 11:29:07Z [0]: CREATE_COMPLETE state changed 2017-05-26 11:29:07Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE Stack CREATE completed successfully 2017-05-26 11:29:08Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE state changed 2017-05-26 11:29:11Z [1]: SIGNAL_IN_PROGRESS Signal: deployment 9f8a9784-c7ff-4974-95d5-b19c82358acb succeeded 2017-05-26 11:29:12Z [1]: CREATE_COMPLETE state changed 2017-05-26 11:29:12Z [overcloud.AllNodesDeploySteps.MessagingDeployment_Step5]: CREATE_COMPLETE Stack CREATE completed successfully 2017-05-26 11:29:12Z [overcloud.AllNodesDeploySteps.MessagingDeployment_Step5]: CREATE_COMPLETE state changed 2017-05-26 11:29:18Z [2]: SIGNAL_IN_PROGRESS Signal: deployment 2a7dbe4d-89af-466e-b71a-8792acef94f2 succeeded 2017-05-26 11:29:18Z [2]: CREATE_COMPLETE state changed 2017-05-26 11:29:19Z [overcloud.AllNodesDeploySteps.GaleraDeployment_Step5]: CREATE_COMPLETE Stack CREATE completed successfully 2017-05-26 11:29:19Z [overcloud.AllNodesDeploySteps.GaleraDeployment_Step5]: CREATE_COMPLETE state changed 2017-05-26 11:31:08Z [1]: SIGNAL_IN_PROGRESS Signal: deployment 3f6dbad6-f15e-43a0-a3e9-9e11126b309f succeeded 2017-05-26 11:31:09Z [1]: CREATE_COMPLETE state changed 2017-05-26 11:31:21Z [2]: SIGNAL_IN_PROGRESS Signal: deployment 275d838d-c671-4105-a8c8-5b7d776c958b succeeded 2017-05-26 11:31:22Z [2]: CREATE_COMPLETE state changed 2017-05-26 11:32:44Z [0]: SIGNAL_IN_PROGRESS Signal: deployment a9ef26dd-c29f-4dfc-a4fd-d183efcb9706 failed (6) 2017-05-26 11:32:44Z [0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 2017-05-26 11:32:44Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 2017-05-26 11:32:45Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2017-05-26 11:32:45Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2017-05-26 11:32:46Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2017-05-26 11:32:46Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 Stack overcloud CREATE_FAILED The error I see inside controller-0 is this one: Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0] Error: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0] Let me know if you need something else. [1] http://file.rdu.redhat.com/~rscarazz/BZ1438608/
Hello, I'm hitting this issue on a support case. Customer deployed 1 controller node and 2 compute nodes. Retried the deployment once and still the same issue. [stack@director10 ~]$ openstack overcloud deploy --templates templates/openstack-tripleo-heat-templates/ -e templates/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e templates/network-environment.yaml --control-scale 1 --compute-scale 2 --control-flavor control --compute-flavor compute --ntp-server 192.0.3.1 2017-06-12 09:10:43Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5]: CREATE_COMPLETE state changed 2017-06-12 09:13:57Z [0]: SIGNAL_IN_PROGRESS Signal: deployment 4ac95325-4614-4b95-98fd-576a064ec66f failed (6) 2017-06-12 09:13:57Z [0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 2017-06-12 09:13:57Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 2017-06-12 09:13:57Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2017-06-12 09:13:58Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2017-06-12 09:13:58Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2017-06-12 09:13:58Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 [stack@director10 ~]$ heat deployment-show 4ac95325-4614-4b95-98fd-576a064ec66f oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\\transport_url. Please use oslo::messaging::default::transport_url instead.\u001b[0m\n\u001b[1;33mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;33mWarning: [norpm] Attempting to uninstall openstack-nova-migration but it will not be removed\u001b[0m\n\u001b[1;31m Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31m Error: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31m Error: gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[run gnocchi upgrade with storage]/returns: change from notrun to 0 failed: gnocchi-upgrade --config-file=/etc/gnocchi/gnocchi.conf returned 1 instead of one of [0]\u001b[0m\n", "deploy_status_code": 6 }, gnocchi error log 2017-06-13 06:53:42.441 934382 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://192.168.32.16:5000/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:87 2017-06-13 06:53:42.605 934382 CRITICAL gnocchi [-] ConnectionError: ('Connection aborted.', BadStatusLine("''",)) 2017-06-13 06:53:42.605 934382 ERROR gnocchi Traceback (most recent call last): 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/bin/gnocchi-upgrade", line 10, in <module> 2017-06-13 06:53:42.605 934382 ERROR gnocchi sys.exit(upgrade()) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 70, in upgrade 2017-06-13 06:53:42.605 934382 ERROR gnocchi s = storage.get_driver(conf) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 144, in get_driver 2017-06-13 06:53:42.605 934382 ERROR gnocchi conf.incoming) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/incoming/swift.py", line 35, in __init__ 2017-06-13 06:53:42.605 934382 ERROR gnocchi self.swift.put_container(self.MEASURE_PREFIX) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1755, in put_container 2017-06-13 06:53:42.605 934382 ERROR gnocchi query_string=query_string) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1673, in _retry 2017-06-13 06:53:42.605 934382 ERROR gnocchi service_token=self.service_token, **kwargs) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1014, in put_container 2017-06-13 06:53:42.605 934382 ERROR gnocchi conn.request(method, path, '', headers) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 436, in request 2017-06-13 06:53:42.605 934382 ERROR gnocchi files=files, **self.requests_args) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 419, in _request 2017-06-13 06:53:42.605 934382 ERROR gnocchi return self.request_session.request(*arg, **kwarg) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request 2017-06-13 06:53:42.605 934382 ERROR gnocchi resp = self.send(prep, **send_kwargs) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 596, in send 2017-06-13 06:53:42.605 934382 ERROR gnocchi r = adapter.send(request, **kwargs) 2017-06-13 06:53:42.605 934382 ERROR gnocchi File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 473, in send 2017-06-13 06:53:42.605 934382 ERROR gnocchi raise ConnectionError(err, request=request) 2017-06-13 06:53:42.605 934382 ERROR gnocchi ConnectionError: ('Connection aborted.', BadStatusLine("''",)) 2017-06-13 06:53:42.605 934382 ERROR gnocchi
IHAC, who is facing the same issue (case #01905549). he claims the overcloud deployment completed successfully with 1 controller & 1 compute , however it gets failed with 3 controller . mError: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]\u001b[0m\n var/log/ceilometer/./collector.log:13:2017-08-06 02:30:25.051 150843 ERROR ceilometer.dispatcher.gnocchi [-] <html><body><h1>504 Gateway Time-out</h1> var/log/ceilometer/./collector.log:17:2017-08-06 02:40:26.026 150843 ERROR ceilometer.dispatcher.gnocchi [-] <html><body><h1>504 Gateway Time-out</h1> var/log/ceilometer/./collector.log:21:2017-08-06 02:50:26.165 150843 ERROR ceilometer.dispatcher.gnocchi [-] <html><body><h1>504 Gateway Time-out</h1> var/log/ceilometer/./collector.log:25:2017-08-06 03:00:26.139 150843 ERROR ceilometer.dispatcher.gnocchi [-] <html><body><h1>504 Gateway Time-out</h1> var/log/ceilometer/./ceilometer-upgrade.log:11:<title>500 Internal Server Error</title> var/log/ceilometer/./ceilometer-upgrade.log:13:<h1>Internal Server Error</h1> I have asked to re-run the test, 3 times failed with same error. Executing command "ceilometer-upgrade --skip-metering-database" manually also re-produces the same error.
Hi, FYI I'm having this error when I launch the overcloud deployment with invalid ssl certifcates: /var/log/gnocchi/gnocchi-upgrade.log: ERROR gnocchi SSLError: hostname '172.29.64.103' doesn't match cloud.ccom.xxx.xx' Thanks
Tested on swift as well now. Looks ok
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0310
Hi, I know this bug looks nice and closed...but I have the same behavior and same Internal Server Error 500 message in /var/log/ceilometer-upgrade.log I am using: RHEL 7.4 Ocata / OSP 11 Your fix is since: puppet-tripleo-6.5.4-1.el7ost My RHEL Undercloud/Director has: puppet-tripleo-6.5.5-2.el7ost.noarch I get the feeling that a regression managed to make its way into this version. Would it be possible to check or can I help somehow? This issue seems to be a reoccurring one. Thanks!