Bug 1391082
| Summary: | swift proxy cannot connect to account server during controller upgrade - Container PUT failed 404 Not Found | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Yurii Prokulevych <yprokule> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Jiri Stransky <jstransk> |
| Status: | CLOSED ERRATA | QA Contact: | Arik Chernetsky <achernet> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 10.0 (Newton) | CC: | apevec, cschwede, derekh, egafford, jdanjou, jschluet, lhh, mabaakou, mabrams, mburns, pkilambi, rhel-osp-director-maint, sclewis, scohen, srevivo, zaitcev |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 10.0 (Newton) | Flags: | scohen:
needinfo+
scohen: needinfo+ |
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-5.0.0-1.3.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-12-14 16:28:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
When gnocchi uses ceph backend I don't see this issue. This looks like we are not talking to swift or swift is missing configured or have not started correctly. At first glance, the swift and haproxy configuration looks good. But swift proxy is reporting some issue with account servers: At the beginning the swift just works well, the PUT container works as excepted : Nov 02 07:26:24 controller-0.localdomain proxy-server[26593]: 172.17.3.12 172.17.3.12 02/Nov/2016/07/26/24 PUT /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure HTTP/1.0 201 - python-swiftclient-3.0.0 4f2b874cc4214834... - - - tx4e24fe753a33432c8d03b-0058199520 - 0.0685 - And after upgrade it returns 503: Nov 02 15:36:25 controller-0.localdomain proxy-server[27404]: ERROR with Account server 172.17.4.12:6002/d1 re: Trying to GET /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12: Connection refused (txn: txfc82227fb2ff47d48bb91-00581a07f8) (client_ip: 172.17.3.12) Nov 02 15:36:25 controller-0.localdomain proxy-server[27404]: ERROR with Account server 172.17.4.11:6002/d1 re: Trying to GET /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12: Connection refused (txn: txfc82227fb2ff47d48bb91-00581a07f8) (client_ip: 172.17.3.12) Nov 02 15:36:25 controller-0.localdomain proxy-server[27404]: ERROR with Account server 172.17.4.14:6002/d1 re: Trying to GET /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12: Connection refused (txn: txfc82227fb2ff47d48bb91-00581a07f8) (client_ip: 172.17.3.12) Nov 02 15:36:25 controller-0.localdomain proxy-server[27404]: Account GET returning 503 for [] (txn: txfc82227fb2ff47d48bb91-00581a07f8) (client_ip: 172.17.3.12) Nov 02 11:34:50 controller-1.localdomain proxy-server[17773]: 172.17.3.12 172.17.3.12 02/Nov/2016/11/34/50 PUT /v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure HTTP/1.0 404 - python-swiftclient-3.1.0 c07d1096dcc34bf1... - 70 - tx4210afe905d84d949d98c-005819cf55 - 0.0490 - This has been fixed in openstack-tripleo-heat-templates-5.0.0-1.3.el7ost. Changing to MODIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html |
Description of problem: ----------------------- During upgrade from RHOS-9 to RHOS-10 process failed during running gnocchi-upgrade: heat deployment-show f92cda16-bc9e-4839-95c0-d1eb410ed8c9 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "ba1a325a-58ea-4e5f-9211-89248d4ea852", "config_id": "7633bd94-5727-4cff-9780-414afe730b81", "output_values": { "deploy_stdout": "", "deploy_stderr": "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 10.0.0.101 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)\n SubjectAltNameWarning\nCRITI [gnocchi] ClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<\nTraceback (most recent call last):\n File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>\n sys.exit(upgrade())\n File \"/usr/lib/python2.7/site-packages/gnocchi/cli.py\", line 62, in upgrade\n s = storage.get_driver(conf)\n File \"/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py\", line 158, in get_driver\n return get_driver_class(conf)(conf.storage)\n File \"/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py\", line 98, in __init__\n self.swift.put_container(self.MEASURE_PREFIX)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1728, in put_container\n query_string=query_string)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1647, in _retry\n service_token=self.service_token, **kwargs)\n File \"/usr/lib/python2.7/site-packages/swiftclient/client.py\", line 1005, in put_container\n raise ClientException.from_response(resp, 'Container PUT failed', body)\nClientException: Container PUT failed: https://10.0.0.101:13808/v1/AUTH_cbbff59ed2b54a358abe4e53052d0e12/measure 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<\n", "deploy_status_code": 1 }, "creation_time": "2016-11-02T11:34:03Z", "updated_time": "2016-11-02T11:34:53Z", "input_values": { "update_identifier": "", "deploy_identifier": "1478083907" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "f92cda16-bc9e-4839-95c0-d1eb410ed8c9" } Version-Release number of selected component (if applicable): ------------------------------------------------------------- python-gnocchiclient-2.6.0-1.el7ost.noarch openstack-gnocchi-common-3.0.1-1.1.el7ost.noarch openstack-gnocchi-carbonara-3.0.1-1.1.el7ost.noarch openstack-gnocchi-api-3.0.1-1.1.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-3.0.1-1.1.el7ost.noarch openstack-gnocchi-metricd-3.0.1-1.1.el7ost.noarch python-gnocchi-3.0.1-1.1.el7ost.noarch openstack-gnocchi-statsd-3.0.1-1.1.el7ost.noarch puppet-gnocchi-9.4.0-2.el7ost.noarch Steps to Reproduce: ------------------- 1. Upgrade from RHOS-9 to RHOS-10 2. Failure happened during 'Controller and Block storage' step