Bug 1391082 - swift proxy cannot connect to account server during controller upgrade - Container PUT failed 404 Not Found
Summary: swift proxy cannot connect to account server during controller upgrade - Cont...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: 10.0 (Newton)
Assignee: Jiri Stransky
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-02 14:17 UTC by Yurii Prokulevych
Modified: 2016-12-14 16:28 UTC (History)
17 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-1.3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:28:19 UTC
scohen: needinfo+
scohen: needinfo+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 392680 None None None 2016-11-09 09:43:24 UTC
Launchpad 1638821 None None None 2016-11-09 09:40:50 UTC

Description Yurii Prokulevych 2016-11-02 14:17:11 UTC
Description of problem:
-----------------------
During upgrade from RHOS-9 to RHOS-10 process failed during running gnocchi-upgrade:

    heat deployment-show f92cda16-bc9e-4839-95c0-d1eb410ed8c9
    WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
    {
      "status": "FAILED",
      "server_id": "ba1a325a-58ea-4e5f-9211-89248d4ea852",
      "config_id": "7633bd94-5727-4cff-9780-414afe730b81",
      "output_values": {
        "deploy_stdout": "",
        "deploy_stderr": "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)\n  SubjectAltNameWarning\nCRITI [gnocchi] ClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<\nTraceback (most recent call last):\n  File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>\n    sys.exit(upgrade())\n  File \"/usr/lib/python2.7/site-packages/gnocchi/cli.py\", line 62, in upgrade\n    s = storage.get_driver(conf)\n  File \"/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py\", line 158, in get_driver\n    return get_driver_class(conf)(conf.storage)\n  File \"/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py\", line 98, in __init__\n    self.swift.put_container(self.MEASURE_PREFIX)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1728, in put_container\n    query_string=query_string)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1647, in _retry\n    service_token=self.service_token, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1005, in put_container\n    raise ClientException.from_response(resp, 'Container PUT failed', body)\nClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<\n", 
        "deploy_status_code": 1
      },
      "creation_time": "2016-11-02T11:34:03Z",
      "updated_time": "2016-11-02T11:34:53Z",
      "input_values": {
        "update_identifier": "",
        "deploy_identifier": "1478083907"
      },
      "action": "CREATE",
      "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1",
      "id": "f92cda16-bc9e-4839-95c0-d1eb410ed8c9"
    }




Version-Release number of selected component (if applicable):
-------------------------------------------------------------
python-gnocchiclient-2.6.0-1.el7ost.noarch
openstack-gnocchi-common-3.0.1-1.1.el7ost.noarch
openstack-gnocchi-carbonara-3.0.1-1.1.el7ost.noarch
openstack-gnocchi-api-3.0.1-1.1.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-3.0.1-1.1.el7ost.noarch
openstack-gnocchi-metricd-3.0.1-1.1.el7ost.noarch
python-gnocchi-3.0.1-1.1.el7ost.noarch
openstack-gnocchi-statsd-3.0.1-1.1.el7ost.noarch
puppet-gnocchi-9.4.0-2.el7ost.noarch


Steps to Reproduce:
-------------------
1. Upgrade from RHOS-9 to RHOS-10
2. Failure happened during 'Controller and Block storage' step

Comment 1 Yurii Prokulevych 2016-11-02 14:18:24 UTC
When gnocchi uses ceph backend I don't see this issue.

Comment 2 Mehdi ABAAKOUK 2016-11-02 15:48:40 UTC
This looks like we are not talking to swift or swift is missing configured or have not started correctly.

At first glance, the swift and haproxy configuration looks good.
But swift proxy is reporting some issue with account servers:

At the beginning the swift just works well, the PUT container works as excepted :

Nov 02 07:26:24 controller-0.localdomain proxy-server[26593]: 172.17.3.12 172.17.3.12 02/Nov/2016/07/26/24 PUT /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure HTTP/1.0 201 - python-swiftclient-3.0.0 4f2b874cc4214834... - - - tx4e24fe753a33432c8d03b-0058199520 - 0.0685 - 


And after upgrade it returns 503:

Nov 02 15:36:25 controller-0.localdomain proxy-server[27404]: ERROR with Account server 172.17.4.12:6002/d1 re: Trying to GET /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12: Connection refused (txn: txfc82227fb2ff47d48bb91-00581a07f8) (client_ip: 172.17.3.12)
Nov 02 15:36:25 controller-0.localdomain proxy-server[27404]: ERROR with Account server 172.17.4.11:6002/d1 re: Trying to GET /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12: Connection refused (txn: txfc82227fb2ff47d48bb91-00581a07f8) (client_ip: 172.17.3.12)
Nov 02 15:36:25 controller-0.localdomain proxy-server[27404]: ERROR with Account server 172.17.4.14:6002/d1 re: Trying to GET /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12: Connection refused (txn: txfc82227fb2ff47d48bb91-00581a07f8) (client_ip: 172.17.3.12)
Nov 02 15:36:25 controller-0.localdomain proxy-server[27404]: Account GET returning 503 for [] (txn: txfc82227fb2ff47d48bb91-00581a07f8) (client_ip: 172.17.3.12)
Nov 02 11:34:50 controller-1.localdomain proxy-server[17773]: 172.17.3.12 172.17.3.12 02/Nov/2016/11/34/50 PUT /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure HTTP/1.0 404 - python-swiftclient-3.1.0 c07d1096dcc34bf1... - 70 - tx4210afe905d84d949d98c-005819cf55 - 0.0490 -

Comment 8 Christian Schwede (cschwede) 2016-11-09 09:43:25 UTC
This has been fixed in openstack-tripleo-heat-templates-5.0.0-1.3.el7ost. Changing to MODIFIED.

Comment 14 errata-xmlrpc 2016-12-14 16:28:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.