Description of problem: ----------------------- RHOS-9 setup deployed with infrared. 3controller + 1compute. Gnocchi manually configured to use swift as storage backend. Failed on step 'controller and block storage upgrade'. Upgrade command: time openstack overcloud deploy \ --templates --libvirt-type kvm \ --ntp-server clock.redhat.com \ --neutron-network-type vxlan \ --neutron-tunnel-types vxlan \ --control-scale 3 \ --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 \ --compute-scale 1 \ --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/gnocchi_swift.yaml \ -e /home/stack/virt/ignore_ceph_waning.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml 2>&1 Excerpt from the log: ... 2016-10-19 08:39:44Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2]: CREATE_IN_PROGRESS state changed 2016-10-19 08:39:44Z [overcloHeat Stack update failed. ud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2]: CREATE_IN_PROGRESS Stack CREATE started 2016-10-19 08:39:44Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.1]: CREATE_IN_PROGRESS state changed 2016-10-19 08:39:45Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.0]: CREATE_IN_PROGRESS state changed 2016-10-19 08:39:46Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.2]: CREATE_IN_PROGRESS state changed 2016-10-19 08:40:26Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.1]: SIGNAL_IN_PROGRESS Signal: deployment e22f2a8a-224c-4cc4-b194-2a2edfa9068c succeeded 2016-10-19 08:40:27Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.1]: CREATE_COMPLETE state changed 2016-10-19 08:40:27Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.2]: SIGNAL_IN_PROGRESS Signal: deployment 784d305e-53bd-4176-ae32-6781013b2ac3 succeeded 2016-10-19 08:40:28Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.2]: CREATE_COMPLETE state changed 2016-10-19 08:42:03Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.0]: SIGNAL_IN_PROGRESS Signal: deployment 3cbc3b90-899d-4a9a-bacb-61028d84730b failed (1) 2016-10-19 08:42:03Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 2016-10-19 08:42:03Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 2016-10-19 08:42:04Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2]: CREATE_FAILED Error: resources.ControllerPacemakerUpgradeDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2016-10-19 08:42:05Z [overcloud-UpdateWorkflow-owtoqtutlc2z]: UPDATE_FAILED Error: resources.ControllerPacemakerUpgradeDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2016-10-19 08:42:05Z [UpdateWorkflow]: UPDATE_FAILED resources.UpdateWorkflow: Error: resources.ControllerPacemakerUpgradeDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2016-10-19 08:42:05Z [overcloud]: UPDATE_FAILED resources.UpdateWorkflow: Error: resources.ControllerPacemakerUpgradeDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 Stack overcloud UPDATE_FAILED Inspecting failed deployment: ----------------------------- heat deployment-show 3cbc3b90-899d-4a9a-bacb-61028d84730b ... gnocchi-upgrade\nCRITI [gnocchi] ClientException: Authorization Failure. Authorization Failed: Service Unavailable (HTTP 503)\nTraceback (most recent call last):\n Fi le \"/usr/bin/gnocchi-upgrade\", line 10, in <module>\n sys.exit(upgrade())\n File \"/usr/lib/python2.7/site-packages/gnocchi/cli.py\", line 62, in upgrade\n s = storage.get_driver(conf)\n File \"/usr/li b/python2.7/site-packages/gnocchi/storage/__init__.py\", line 158, in get_driver\n return get_driver_class(conf)(conf.storage)\n File \"/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py\", line 98, i n __init__\n self.swift.put_container(self.MEASURE_PREFIX)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1728, in put_container\n query_string=query_string)\n File \"/usr/lib/py thon2.7/site-packages/swiftclient/client.py\", line 1635, in _retry\n self.url, self.token = self.get_auth()\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1587, in get_auth\n tim eout=self.timeout)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 662, in get_auth\n auth_version=auth_version)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", li ne 582, in get_auth_keystone\n raise ClientException('Authorization Failure. %s' % err)\nClientException: Authorization Failure. Authorization Failed: Service Unavailable (HTTP 503)\n", "deploy_status_code": 1 }, ... Cluster status from controller: ------------------------------- pcs status Cluster name: tripleo_cluster Stack: corosync Current DC: controller-1 (version 1.1.15-11.el7-e174ec8) - partition with quorum Last updated: Wed Oct 19 09:13:28 2016 Last change: Wed Oct 19 08:41:30 2016 by root via crm_resource on controller-0 3 nodes and 19 resources configured: 4 resources DISABLED and 0 BLOCKED from being started due to failures Online: [ controller-0 controller-1 controller-2 ] Full list of resources: ip-172.17.1.10 (ocf::heartbeat:IPaddr2): Started controller-0 ip-192.0.2.6 (ocf::heartbeat:IPaddr2): Started controller-1 ip-172.17.4.10 (ocf::heartbeat:IPaddr2): Started controller-2 Clone Set: haproxy-clone [haproxy] Started: [ controller-0 controller-1 controller-2 ] Master/Slave Set: galera-master [galera] Masters: [ controller-0 controller-1 controller-2 ] ip-172.17.3.10 (ocf::heartbeat:IPaddr2): Started controller-0 ip-10.0.0.101 (ocf::heartbeat:IPaddr2): Started controller-1 Clone Set: rabbitmq-clone [rabbitmq] Stopped (disabled): [ controller-0 controller-1 controller-2 ] Master/Slave Set: redis-master [redis] Masters: [ controller-1 ] Slaves: [ controller-0 controller-2 ] ip-172.17.1.11 (ocf::heartbeat:IPaddr2): Started controller-2 openstack-cinder-volume (systemd:openstack-cinder-volume): Stopped (disabled) Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled HTTP status on all controllers: ------------------------------- systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─openstack-dashboard.conf Active: inactive (dead) Docs: man:httpd(8) man:apachectl(8) Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-pl.js' Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-no.js' Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-da.js' Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-hu.js' Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-ky.js' Oct 19 06:07:11 controller-2.localdomain systemd[1]: Started Cluster Controlled httpd. Oct 19 07:17:20 controller-2.localdomain systemd[1]: Reloaded Cluster Controlled httpd. Oct 19 08:05:25 controller-2.localdomain systemd[1]: Stopping Cluster Controlled httpd... Oct 19 08:05:27 controller-2.localdomain systemd[1]: Stopped Cluster Controlled httpd. Oct 19 08:11:42 controller-2.localdomain systemd[1]: Stopped The Apache HTTP Server. gnocchi-swift.yaml ------------------ cat /home/stack/virt/gnocchi_swift.yaml parameter_defaults: ## Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'. GnocchiBackend: swift Version-Release number of selected component (if applicable): ------------------------------------------------------------- python-tripleoclient-5.2.0-2.el7ost.noarch openstack-tripleo-heat-templates-compat-2.0.0-34.3.el7ost.noarch openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch openstack-tripleo-common-5.2.1-0.20161007114757.cc19d04.el7ost.noarch puppet-tripleo-5.2.0-2.el7ost.noarch openstack-tripleo-image-elements-5.0.0-1.el7ost.noarch openstack-tripleo-ui-1.0.3-1.el7ost.noarch openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost.noarch openstack-tripleo-puppet-elements-5.0.0-0.20161003213431.200d011.el7ost.noarch Steps to Reproduce: ------------------- 1. Run 'upgrade controller and block storage' step Additional info: ---------------- This is virtual setup. Each controller has 14Gb RAM and 2vcpus.
assigning to Lifecycle for triage
if you've still got the env around can you do curl -O https://raw.githubusercontent.com/goneri/tripleo-stack-dump/master/list_nodes_status chmod +x ./list_nodes_status run ./list_node_status so we can get any more of the err from the heat config stderr/out ... Having said that the error in the description seems clear enough so don't worry if you don't have the env... Also to be clear this is during the controller upgrade step (from the -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml)
FYI Gnocchi needs access to Swift, and therefore to Keystone to authenticate, in order to check and do the upgrade of data format between Gnocchi 2.x and 3.x.
Another upgrade attempt with swift backend failed: ================================================== heat deployment-show fdb0bb4e-13bd-41e7-8550-67ce3060bc15 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "6e24c0df-7fd0-4f27-a062-7b637b5e1a13", "config_id": "4719f338-b438-4ba2-972d-0ccacfb2a95e", "output_values": { "deploy_stdout": "", "deploy_stderr": "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)\n SubjectAltNameWarning\nCRITI [gnocchi] ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))\nTraceback (most recent call last):\n File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>\n sys.exit(upgrade())\n File \"/usr/lib/python2.7/site-packages/gnocchi/cli.py\", line 62, in upgrade\n s = storage.get_driver(conf)\n File \"/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py\", line 158, in get_driver\n return get_driver_class(conf)(conf.storage)\n File \"/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py\", line 98, in __init__\n self.swift.put_container(self.MEASURE_PREFIX)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1728, in put_container\n query_string=query_string)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1647, in _retry\n service_token=self.service_token, **kwargs)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 996, in put_container\n conn.request(method, path, '', headers)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 437, in request\n files=files, **self.requests_args)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 420, in _request\n return self.request_session.request(*arg, **kwarg)\n File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 475, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 585, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python2.7/site-packages/requests/adapters.py\", line 453, in send\n raise ConnectionError(err, request=request)\nConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))\n", "deploy_status_code": 1 }, "creation_time": "2016-11-01T10:09:08Z", "updated_time": "2016-11-01T10:09:53Z", "input_values": { "update_identifier": "", "deploy_identifier": "1477992188" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "fdb0bb4e-13bd-41e7-8550-67ce3060bc15" } List nodes status: ================== +--------------------------------------+--------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------------+--------+------------+-------------+---------------------+ | 2bd8e6f7-fd53-4e98-9274-570af5232147 | ceph-0 | ACTIVE | - | Running | ctlplane=192.0.2.7 | | a81261aa-36c6-487a-963c-8b548e9df444 | compute-0 | ACTIVE | - | Running | ctlplane=192.0.2.8 | | 6e24c0df-7fd0-4f27-a062-7b637b5e1a13 | controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.12 | | 5a6a983b-9da2-40ea-8f86-1ad73525c369 | controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.9 | | 74cc4cb4-62ce-418b-b751-c83907ee3517 | controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.11 | +--------------------------------------+--------------+--------+------------+-------------+---------------------+ ** 192.0.2.7 up and running ** 192.0.2.8 up and running ** 192.0.2.12 up and running /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning CRITI [gnocchi] ConnectionError: ('Connection aborted.', BadStatusLine("''",)) Traceback (most recent call last): File "/usr/bin/gnocchi-upgrade", line 10, in <module> sys.exit(upgrade()) File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 62, in upgrade s = storage.get_driver(conf) File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 158, in get_driver return get_driver_class(conf)(conf.storage) File "/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py", line 98, in __init__ self.swift.put_container(self.MEASURE_PREFIX) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1728, in put_container query_string=query_string) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1647, in _retry service_token=self.service_token, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 996, in put_container conn.request(method, path, '', headers) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 437, in request files=files, **self.requests_args) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 420, in _request return self.request_session.request(*arg, **kwarg) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 585, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 453, in send raise ConnectionError(err, request=request) ConnectionError: ('Connection aborted.', BadStatusLine("''",)) ** 192.0.2.9 up and running ** 192.0.2.11 up and running Pacemaker is OK: ================ pcs status Cluster name: tripleo_cluster Stack: corosync Current DC: controller-0 (version 1.1.15-11.el7-e174ec8) - partition with quorum Last updated: Tue Nov 1 10:29:17 2016 Last change: Tue Nov 1 10:08:48 2016 by root via crm_resource on controller-0 3 nodes and 19 resources configured Online: [ controller-0 controller-1 controller-2 ] Full list of resources: ip-172.17.1.10 (ocf::heartbeat:IPaddr2): Started controller-0 ip-192.0.2.6 (ocf::heartbeat:IPaddr2): Started controller-1 ip-172.17.4.10 (ocf::heartbeat:IPaddr2): Started controller-2 Clone Set: haproxy-clone [haproxy] Started: [ controller-0 controller-1 controller-2 ] Master/Slave Set: galera-master [galera] Masters: [ controller-0 controller-1 controller-2 ] ip-172.17.3.10 (ocf::heartbeat:IPaddr2): Started controller-0 ip-10.0.0.101 (ocf::heartbeat:IPaddr2): Started controller-1 Clone Set: rabbitmq-clone [rabbitmq] Started: [ controller-0 controller-1 controller-2 ] Master/Slave Set: redis-master [redis] Masters: [ controller-2 ] Slaves: [ controller-0 controller-1 ] ip-172.17.1.11 (ocf::heartbeat:IPaddr2): Started controller-2 openstack-cinder-volume (systemd:openstack-cinder-volume): Started controller-0 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled The only thing is that swift services ain't running on any controller: ====================================================================== [root@controller-1 ~ ]# systemctl list-units -t service | grep swift [root@controller-1 ~ ]# echo $? 1
Hmm So currently in the patch i have upstream, gnocchi upgrade happens in step 4 of controller upgrades and swift systemctl start happens in step 5. That could be reason why swift client is aborting the request. We could move the gnocchi upgrade out to last step or move swift start before gnocchi upgrade. i'll look into it now.
Looks like latest change helps, though this time it failed during gnocchi-upgrade: heat deployment-show f92cda16-bc9e-4839-95c0-d1eb410ed8c9 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "ba1a325a-58ea-4e5f-9211-89248d4ea852", "config_id": "7633bd94-5727-4cff-9780-414afe730b81", "output_values": { "deploy_stdout": "", "deploy_stderr": "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonN ame` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)\n SubjectAltNameWarning\nCRITI [gnocchi] ClientException : Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<\nTraceb$ ck (most recent call last):\n File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>\n sys.exit(upgrade())\n File \"/usr/lib/python2.7/site-packages/gnocchi/cli.py\", line 62, in upgrade\n s = storage.$ et_driver(conf)\n File \"/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py\", line 158, in get_driver\n return get_driver_class(conf)(conf.storage)\n File \"/usr/lib/python2.7/site-packages/gnoc$ hi/storage/swift.py\", line 98, in __init__\n self.swift.put_container(self.MEASURE_PREFIX)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1728, in put_container\n query_string=q$ ery_string)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1647, in _retry\n service_token=self.service_token, **kwargs)\n File \"/usr/lib/python2.7/site-packages/swiftclient/clien$ .py\", line 1005, in put_container\n raise ClientException.from_response(resp, 'Container PUT failed', body)\nClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e530$ 2d0e12/measure 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<\n", "deploy_status_code": 1 }, "creation_time": "2016-11-02T11:34:03Z", "updated_time": "2016-11-02T11:34:53Z", "input_values": { "update_identifier": "", "deploy_identifier": "1478083907" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "f92cda16-bc9e-4839-95c0-d1eb410ed8c9" } ./list_nodes_status +--------------------------------------+--------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------------+--------+------------+-------------+---------------------+ | 124d1a61-f077-45ed-beff-d236a7fb15ca | ceph-0 | ACTIVE | - | Running | ctlplane=192.0.2.7 | | c89fbfea-99ba-46e4-b852-d8f9ff331df5 | compute-0 | ACTIVE | - | Running | ctlplane=192.0.2.8 | | ba1a325a-58ea-4e5f-9211-89248d4ea852 | controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.10 | | 7c355b29-fca9-4d7e-bd93-4b9a69197617 | controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.9 | | c859798f-0bba-4341-867a-4ba4fa10765a | controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.11 | +--------------------------------------+--------------+--------+------------+-------------+---------------------+ ** 192.0.2.7 up and running ** 192.0.2.8 up and running ** 192.0.2.10 up and running /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This fea ture is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning CRITI [gnocchi] ClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resourc e could not be found.< Traceback (most recent call last): File "/usr/bin/gnocchi-upgrade", line 10, in <module> sys.exit(upgrade()) File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 62, in upgrade s = storage.get_driver(conf) File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 158, in get_driver return get_driver_class(conf)(conf.storage) File "/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py", line 98, in __init__ self.swift.put_container(self.MEASURE_PREFIX) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1728, in put_container query_string=query_string) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1647, in _retry service_token=self.service_token, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1005, in put_container raise ClientException.from_response(resp, 'Container PUT failed', body) ClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be f ound.< ** 192.0.2.9 up and running ** 192.0.2.11 up and running
the fix landed in newton so moving this to POST https://review.openstack.org/#/c/393424/1 adding needinfo Yurii - do you agree? (I think the stopped swift services you are seeing is another issue right?)
Hi Marios, agree looks we have correct sequence for swift/keystone services.
Verified with openstack-tripleo-heat-templates-5.0.0-1.4.el7ost.noarch, successfully passed convergence step.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html