Bug 1386595 - Upgrade 9-10 failed with gnocchi using swift backend: ClientException: Authorization Failure. Authorization Failed: Service Unavailable (HTTP 503)
Summary: Upgrade 9-10 failed with gnocchi using swift backend: ClientException: Author...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 10.0 (Newton)
Assignee: Pradeep Kilambi
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On:
Blocks: 1337794
TreeView+ depends on / blocked
 
Reported: 2016-10-19 09:32 UTC by Yurii Prokulevych
Modified: 2016-12-29 16:59 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-1.3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:22:02 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 388649 None None None 2016-10-19 13:44:25 UTC
Launchpad 1634897 None None None 2016-10-19 13:43:42 UTC

Description Yurii Prokulevych 2016-10-19 09:32:05 UTC
Description of problem:
-----------------------
RHOS-9 setup deployed with infrared. 
3controller + 1compute. 
Gnocchi manually configured to use swift as storage backend.

Failed on step 'controller and block storage upgrade'.
Upgrade command:

time openstack overcloud deploy \
--templates --libvirt-type kvm \
--ntp-server clock.redhat.com \
--neutron-network-type vxlan \
--neutron-tunnel-types vxlan \
--control-scale 3 \
--control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 \ --compute-scale 1 \
--compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/gnocchi_swift.yaml \
-e /home/stack/virt/ignore_ceph_waning.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml 2>&1

Excerpt from the log:
...
2016-10-19 08:39:44Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2]: CREATE_IN_PROGRESS  state changed
2016-10-19 08:39:44Z [overcloHeat Stack update failed.
ud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2]: CREATE_IN_PROGRESS  Stack CREATE started
2016-10-19 08:39:44Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.1]: CREATE_IN_PROGRESS  state changed
2016-10-19 08:39:45Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.0]: CREATE_IN_PROGRESS  state changed
2016-10-19 08:39:46Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.2]: CREATE_IN_PROGRESS  state changed
2016-10-19 08:40:26Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.1]: SIGNAL_IN_PROGRESS  Signal: deployment e22f2a8a-224c-4cc4-b194-2a2edfa9068c succeeded
2016-10-19 08:40:27Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.1]: CREATE_COMPLETE  state changed
2016-10-19 08:40:27Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.2]: SIGNAL_IN_PROGRESS  Signal: deployment 784d305e-53bd-4176-ae32-6781013b2ac3 succeeded
2016-10-19 08:40:28Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.2]: CREATE_COMPLETE  state changed
2016-10-19 08:42:03Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.0]: SIGNAL_IN_PROGRESS  Signal: deployment 3cbc3b90-899d-4a9a-bacb-61028d84730b failed (1)
2016-10-19 08:42:03Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2.0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
2016-10-19 08:42:03Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
2016-10-19 08:42:04Z [overcloud-UpdateWorkflow-owtoqtutlc2z.ControllerPacemakerUpgradeDeployment_Step2]: CREATE_FAILED  Error: resources.ControllerPacemakerUpgradeDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1
2016-10-19 08:42:05Z [overcloud-UpdateWorkflow-owtoqtutlc2z]: UPDATE_FAILED  Error: resources.ControllerPacemakerUpgradeDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1
2016-10-19 08:42:05Z [UpdateWorkflow]: UPDATE_FAILED  resources.UpdateWorkflow: Error: resources.ControllerPacemakerUpgradeDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1
2016-10-19 08:42:05Z [overcloud]: UPDATE_FAILED  resources.UpdateWorkflow: Error: resources.ControllerPacemakerUpgradeDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1

 Stack overcloud UPDATE_FAILED 


Inspecting failed deployment:
-----------------------------
heat deployment-show 3cbc3b90-899d-4a9a-bacb-61028d84730b
...
gnocchi-upgrade\nCRITI [gnocchi] ClientException: Authorization Failure. Authorization Failed: Service Unavailable (HTTP 503)\nTraceback (most recent call last):\n  Fi
le \"/usr/bin/gnocchi-upgrade\", line 10, in <module>\n    sys.exit(upgrade())\n  File \"/usr/lib/python2.7/site-packages/gnocchi/cli.py\", line 62, in upgrade\n    s = storage.get_driver(conf)\n  File \"/usr/li
b/python2.7/site-packages/gnocchi/storage/__init__.py\", line 158, in get_driver\n    return get_driver_class(conf)(conf.storage)\n  File \"/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py\", line 98, i
n __init__\n    self.swift.put_container(self.MEASURE_PREFIX)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1728, in put_container\n    query_string=query_string)\n  File \"/usr/lib/py
thon2.7/site-packages/swiftclient/client.py\", line 1635, in _retry\n    self.url, self.token = self.get_auth()\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1587, in get_auth\n    tim
eout=self.timeout)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 662, in get_auth\n    auth_version=auth_version)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", li
ne 582, in get_auth_keystone\n    raise ClientException('Authorization Failure. %s' % err)\nClientException: Authorization Failure. Authorization Failed: Service Unavailable (HTTP 503)\n", 
    "deploy_status_code": 1
  }, 
...

Cluster status from controller:
-------------------------------
pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-1 (version 1.1.15-11.el7-e174ec8) - partition with quorum
Last updated: Wed Oct 19 09:13:28 2016          Last change: Wed Oct 19 08:41:30 2016 by root via crm_resource on controller-0

3 nodes and 19 resources configured: 4 resources DISABLED and 0 BLOCKED from being started due to failures

Online: [ controller-0 controller-1 controller-2 ]

Full list of resources:

 ip-172.17.1.10 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-192.0.2.6   (ocf::heartbeat:IPaddr2):       Started controller-1
 ip-172.17.4.10 (ocf::heartbeat:IPaddr2):       Started controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ controller-0 controller-1 controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ controller-0 controller-1 controller-2 ]
 ip-172.17.3.10 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-10.0.0.101  (ocf::heartbeat:IPaddr2):       Started controller-1
 Clone Set: rabbitmq-clone [rabbitmq]
     Stopped (disabled): [ controller-0 controller-1 controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ controller-1 ]
     Slaves: [ controller-0 controller-2 ]
 ip-172.17.1.11 (ocf::heartbeat:IPaddr2):       Started controller-2
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Stopped (disabled)

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

HTTP status on all controllers:
-------------------------------
systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: inactive (dead)
     Docs: man:httpd(8)
           man:apachectl(8)

Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-pl.js'
Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-no.js'
Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-da.js'
Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-hu.js'
Oct 19 06:07:03 controller-2.localdomain python[3463]: Copying '/usr/share/javascript/jquery_ui/ui/i18n/jquery.ui.datepicker-ky.js'
Oct 19 06:07:11 controller-2.localdomain systemd[1]: Started Cluster Controlled httpd.
Oct 19 07:17:20 controller-2.localdomain systemd[1]: Reloaded Cluster Controlled httpd.
Oct 19 08:05:25 controller-2.localdomain systemd[1]: Stopping Cluster Controlled httpd...
Oct 19 08:05:27 controller-2.localdomain systemd[1]: Stopped Cluster Controlled httpd.
Oct 19 08:11:42 controller-2.localdomain systemd[1]: Stopped The Apache HTTP Server.

gnocchi-swift.yaml
------------------
cat /home/stack/virt/gnocchi_swift.yaml 
parameter_defaults:
  ## Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GnocchiBackend: swift


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
python-tripleoclient-5.2.0-2.el7ost.noarch
openstack-tripleo-heat-templates-compat-2.0.0-34.3.el7ost.noarch
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
openstack-tripleo-common-5.2.1-0.20161007114757.cc19d04.el7ost.noarch
puppet-tripleo-5.2.0-2.el7ost.noarch
openstack-tripleo-image-elements-5.0.0-1.el7ost.noarch
openstack-tripleo-ui-1.0.3-1.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost.noarch
openstack-tripleo-puppet-elements-5.0.0-0.20161003213431.200d011.el7ost.noarch


Steps to Reproduce:
-------------------
1. Run 'upgrade controller and block storage' step


Additional info:
----------------
This is virtual setup.
Each controller has 14Gb RAM and 2vcpus.

Comment 4 Marios Andreou 2016-10-19 09:44:22 UTC
assigning to Lifecycle for triage

Comment 5 Marios Andreou 2016-10-19 09:49:36 UTC
if you've still got the env around can you do

curl -O https://raw.githubusercontent.com/goneri/tripleo-stack-dump/master/list_nodes_status
chmod +x ./list_nodes_status

 run ./list_node_status 

so we can get any more of the err from the heat config stderr/out ... Having said that the error in the description seems clear enough so don't worry if you don't have the env...

Also to be clear this is during the controller upgrade step (from the -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml)

Comment 6 Julien Danjou 2016-10-19 10:05:01 UTC
FYI Gnocchi needs access to Swift, and therefore to Keystone to authenticate, in order to check and do the upgrade of data format between Gnocchi 2.x and 3.x.

Comment 11 Yurii Prokulevych 2016-11-01 10:33:24 UTC
Another upgrade attempt with swift backend failed:
==================================================

heat deployment-show fdb0bb4e-13bd-41e7-8550-67ce3060bc15
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED", 
  "server_id": "6e24c0df-7fd0-4f27-a062-7b637b5e1a13", 
  "config_id": "4719f338-b438-4ba2-972d-0ccacfb2a95e", 
  "output_values": {
    "deploy_stdout": "", 
    "deploy_stderr": "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)\n  SubjectAltNameWarning\nCRITI [gnocchi] ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))\nTraceback (most recent call last):\n  File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>\n    sys.exit(upgrade())\n  File \"/usr/lib/python2.7/site-packages/gnocchi/cli.py\", line 62, in upgrade\n    s = storage.get_driver(conf)\n  File \"/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py\", line 158, in get_driver\n    return get_driver_class(conf)(conf.storage)\n  File \"/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py\", line 98, in __init__\n    self.swift.put_container(self.MEASURE_PREFIX)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1728, in put_container\n    query_string=query_string)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1647, in _retry\n    service_token=self.service_token, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 996, in put_container\n    conn.request(method, path, '', headers)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 437, in request\n    files=files, **self.requests_args)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 420, in _request\n    return self.request_session.request(*arg, **kwarg)\n  File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 475, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 585, in send\n    r = adapter.send(request, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/requests/adapters.py\", line 453, in send\n    raise ConnectionError(err, request=request)\nConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))\n", 
    "deploy_status_code": 1
  }, 
  "creation_time": "2016-11-01T10:09:08Z", 
  "updated_time": "2016-11-01T10:09:53Z", 
  "input_values": {
    "update_identifier": "", 
    "deploy_identifier": "1477992188"
  }, 
  "action": "CREATE", 
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", 
  "id": "fdb0bb4e-13bd-41e7-8550-67ce3060bc15"
}

List nodes status:
==================

+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks            |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| 2bd8e6f7-fd53-4e98-9274-570af5232147 | ceph-0       | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| a81261aa-36c6-487a-963c-8b548e9df444 | compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| 6e24c0df-7fd0-4f27-a062-7b637b5e1a13 | controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
| 5a6a983b-9da2-40ea-8f86-1ad73525c369 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 74cc4cb4-62ce-418b-b751-c83907ee3517 | controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
** 192.0.2.7 up and running
** 192.0.2.8 up and running
** 192.0.2.12 up and running
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
CRITI [gnocchi] ConnectionError: ('Connection aborted.', BadStatusLine("''",))
Traceback (most recent call last):
  File "/usr/bin/gnocchi-upgrade", line 10, in <module>
    sys.exit(upgrade())
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 62, in upgrade
    s = storage.get_driver(conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 158, in get_driver
    return get_driver_class(conf)(conf.storage)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py", line 98, in __init__
    self.swift.put_container(self.MEASURE_PREFIX)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1728, in put_container
    query_string=query_string)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1647, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 996, in put_container
    conn.request(method, path, '', headers)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 437, in request
    files=files, **self.requests_args)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 420, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 453, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine("''",))


** 192.0.2.9 up and running
** 192.0.2.11 up and running

Pacemaker is OK:
================
 pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-0 (version 1.1.15-11.el7-e174ec8) - partition with quorum
Last updated: Tue Nov  1 10:29:17 2016          Last change: Tue Nov  1 10:08:48 2016 by root via crm_resource on controller-0

3 nodes and 19 resources configured

Online: [ controller-0 controller-1 controller-2 ]

Full list of resources:

 ip-172.17.1.10 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-192.0.2.6   (ocf::heartbeat:IPaddr2):       Started controller-1
 ip-172.17.4.10 (ocf::heartbeat:IPaddr2):       Started controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ controller-0 controller-1 controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ controller-0 controller-1 controller-2 ]
 ip-172.17.3.10 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-10.0.0.101  (ocf::heartbeat:IPaddr2):       Started controller-1
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ controller-0 controller-1 controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ controller-2 ]
     Slaves: [ controller-0 controller-1 ]
 ip-172.17.1.11 (ocf::heartbeat:IPaddr2):       Started controller-2
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

The only thing is that swift services ain't running on any controller:
======================================================================
[root@controller-1 ~ ]# systemctl list-units -t service | grep swift
[root@controller-1 ~ ]# echo $?
1

Comment 13 Pradeep Kilambi 2016-11-01 12:25:03 UTC
Hmm So currently in the patch i have upstream, gnocchi upgrade happens in step 4 of controller upgrades and swift systemctl start happens in step 5. That could be reason why swift client is aborting the request. We could move the gnocchi upgrade out to last step or move swift start before gnocchi upgrade. i'll look into it now.

Comment 14 Yurii Prokulevych 2016-11-02 12:01:23 UTC
Looks like latest change helps, though this time it failed during gnocchi-upgrade:

heat deployment-show f92cda16-bc9e-4839-95c0-d1eb410ed8c9
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED", 
  "server_id": "ba1a325a-58ea-4e5f-9211-89248d4ea852", 
  "config_id": "7633bd94-5727-4cff-9780-414afe730b81", 
  "output_values": {
    "deploy_stdout": "", 
    "deploy_stderr": "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonN
ame` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)\n  SubjectAltNameWarning\nCRITI [gnocchi] ClientException
: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<\nTraceb$
ck (most recent call last):\n  File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>\n    sys.exit(upgrade())\n  File \"/usr/lib/python2.7/site-packages/gnocchi/cli.py\", line 62, in upgrade\n    s = storage.$
et_driver(conf)\n  File \"/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py\", line 158, in get_driver\n    return get_driver_class(conf)(conf.storage)\n  File \"/usr/lib/python2.7/site-packages/gnoc$
hi/storage/swift.py\", line 98, in __init__\n    self.swift.put_container(self.MEASURE_PREFIX)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1728, in put_container\n    query_string=q$
ery_string)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1647, in _retry\n    service_token=self.service_token, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/clien$
.py\", line 1005, in put_container\n    raise ClientException.from_response(resp, 'Container PUT failed', body)\nClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e530$
2d0e12/measure 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<\n", 
    "deploy_status_code": 1
  }, 
  "creation_time": "2016-11-02T11:34:03Z", 
  "updated_time": "2016-11-02T11:34:53Z", 
  "input_values": {
    "update_identifier": "", 
    "deploy_identifier": "1478083907"
  }, 
  "action": "CREATE", 
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", 
  "id": "f92cda16-bc9e-4839-95c0-d1eb410ed8c9"
}

./list_nodes_status 
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks            |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| 124d1a61-f077-45ed-beff-d236a7fb15ca | ceph-0       | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| c89fbfea-99ba-46e4-b852-d8f9ff331df5 | compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| ba1a325a-58ea-4e5f-9211-89248d4ea852 | controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| 7c355b29-fca9-4d7e-bd93-4b9a69197617 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| c859798f-0bba-4341-867a-4ba4fa10765a | controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
** 192.0.2.7 up and running
** 192.0.2.8 up and running
** 192.0.2.10 up and running
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This fea
ture is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
CRITI [gnocchi] ClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resourc
e could not be found.<
Traceback (most recent call last):
  File "/usr/bin/gnocchi-upgrade", line 10, in <module>
    sys.exit(upgrade())
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 62, in upgrade
    s = storage.get_driver(conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 158, in get_driver
    return get_driver_class(conf)(conf.storage)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py", line 98, in __init__
    self.swift.put_container(self.MEASURE_PREFIX)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1728, in put_container
    query_string=query_string)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1647, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1005, in put_container
    raise ClientException.from_response(resp, 'Container PUT failed', body)
ClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be f
ound.<


** 192.0.2.9 up and running
** 192.0.2.11 up and running

Comment 16 Marios Andreou 2016-11-04 15:19:44 UTC
the fix landed in newton so moving this to POST https://review.openstack.org/#/c/393424/1

adding needinfo Yurii - do you agree? (I think the stopped swift services you are seeing is another issue right?)

Comment 17 Yurii Prokulevych 2016-11-04 15:35:41 UTC
Hi Marios, agree looks we have correct sequence for swift/keystone services.

Comment 19 Yurii Prokulevych 2016-11-09 16:48:43 UTC
Verified with openstack-tripleo-heat-templates-5.0.0-1.4.el7ost.noarch,
successfully passed convergence step.

Comment 21 errata-xmlrpc 2016-12-14 16:22:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.