Bug 1724685
| Summary: | OSP 14->15: Keystone broken after controller-0 upgrade, returns 500s | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jiri Stransky <jstransk> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Carlos Camacho <ccamacho> |
| Status: | CLOSED ERRATA | QA Contact: | Sasha Smolyak <ssmolyak> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 15.0 (Stein) | CC: | ccamacho, jfrancoa, lbezdick, mburns, sclewis |
| Target Milestone: | --- | Keywords: | Triaged, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-common-10.8.1-0.20190813170455.913b8de.el8ost.noarch openstack-tripleo-heat-templates-10.6.1-0.20190815230440.9adae50.el8ost.noarch | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-05 11:59:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1727807 | ||
|
Description
Jiri Stransky
2019-06-27 14:27:47 UTC
Ok another interesting find: (overcloud) [stack@undercloud-0 ~]$ openstack token issue +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | expires | 2019-07-01T14:05:05+0000 | | id | gAAAAABdGgUBc8GubF7yPmoUE6qOCwTGLKCTiGlYCg4E5AB45PQuAvxpmLCgJKpK1gvIIJrphvnNCAydsZ6LqrY49UflsrV-M7hbu4SOc2n6R7E-3Snkhxkpcywp1QczzDA7d9AsxaKs_XnkTo82NCfHAMjw8Ekd7FIyXACzLcPFD0aX18gPTYk | | project_id | 07f954174fc348d38dabe1785ba24ad3 | | user_id | 8be265d069864b3f95f5919d78526077 | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack token issue An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-7620771b-12a5-4416-b9f5-898df04a35a2) (overcloud) [stack@undercloud-0 ~]$ openstack token issue An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-6f3a0dc7-f5a9-4c15-bcba-7e507b071524) (overcloud) [stack@undercloud-0 ~]$ openstack token issue +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | expires | 2019-07-01T14:05:12+0000 | | id | gAAAAABdGgUIPI3rM5idJLhjUwmfXnGfN4s7-cwBFdw78YeIpHpLJcVlOhnwhtfrCbdbBJWpiENMu4AHXR5gvVFLtAi3DSDk35sz7JrLOXfpehSXnE7w0HIqNUgrLsn6F5wxi-4r0bQYu7uXRQ63nMnyE7PPQHg2Ta1qKbO6U-UWXpOYh7HiTTg | | project_id | 07f954174fc348d38dabe1785ba24ad3 | | user_id | 8be265d069864b3f95f5919d78526077 | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack token issue An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-c2cca510-3dd0-4a1a-ad49-1f00a748d24d) (overcloud) [stack@undercloud-0 ~]$ openstack token issue An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-e927ce24-80b6-45df-9943-6b809e6176f1) (overcloud) [stack@undercloud-0 ~]$ openstack token issue +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | expires | 2019-07-01T14:05:18+0000 | | id | gAAAAABdGgUOcKr6KcMgRBbXzKbvsbZ08tZ7WdgDFlqOQSLHiccmMGWf__UwlWXjBZH5jg-Wv2RcmAzVcyPeOlsKwJHz2YeJlC_kJnX-HQlXtgK3P88qMQQmhMkGbJdWKahgQkMFC4cAxJJEkkEW0uOBFQBP7AwITAc3mtX8I689nsuMvid-JvM | | project_id | 07f954174fc348d38dabe1785ba24ad3 | | user_id | 8be265d069864b3f95f5919d78526077 | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack token issue An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-6a7ad49f-ead0-4e57-8a48-2afff3ba2421) (overcloud) [stack@undercloud-0 ~]$ openstack token issue An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-3867ec74-bc34-4851-841d-4eb0ad925514) (overcloud) [stack@undercloud-0 ~]$ openstack token issue +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | expires | 2019-07-01T14:05:24+0000 | | id | gAAAAABdGgUUFicKaga6KB04nVNO5TwrOOrScVfuLBot-sbmJICHpKWBH2Jt83y1BrhKjnXtcclNI6eOfsDzWOqro1hKcWBP9jzVWmkMfi2pbAZeZiGg_hv7glZMB2XKCWJ2HfH07oG8qsGubbyBA1fGm6d2zv0Zd6P3fs8Dk6jLLc7iugu-L-I | | project_id | 07f954174fc348d38dabe1785ba24ad3 | | user_id | 8be265d069864b3f95f5919d78526077 | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Looks like some sort of round-robin issue with 2 out of 3 backends being down. This is at a stage where just 1 controller is up in the upgrade, so it needs to be ensured that any round-robin balancing does not try to distribute anything to the other 2 controllers. This is due to missing stopping tasks for old controllers. Haproxy mistakenly load-balances to old controllers too when it should be limited to the upgraded ones. This was fixed by Carlos, adding patch links. Re-setting Target Milestone z1 to --- to begin the 15z1 Maintenance Release. Working in THT package:
(undercloud) [stack@undercloud-0 ~]$ rpm -qa | grep tripleo-heat-templates openstack-tripleo-heat-templates-10.6.2-0.20191202200455.41d9f8a.el8ost.noarch
[root@undercloud-0 stack]# rpm -qa | grep tripleo-common
python3-tripleo-common-10.8.2-0.20191125220527.c2a83c1.el8ost.noarch
openstack-tripleo-common-10.8.2-0.20191125220527.c2a83c1.el8ost.noarch
openstack-tripleo-common-containers-10.8.2-0.20191125220527.c2a83c1.el8ost.noarch
2019-12-12 19:54:33 | PLAY [External upgrade step 1] *************************************************
2019-12-12 19:54:33 | Thursday 12 December 2019 19:54:09 -0500 (0:00:00.131) 0:00:03.936 *****
2019-12-12 19:54:33 | included: /usr/share/ansible/roles/tripleo-container-stop/tasks/container_stop.yaml for undercloud
2019-12-12 19:54:33 | Thursday 12 December 2019 19:54:10 -0500 (0:00:00.373) 0:00:04.310 *****
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | TASK [tripleo-container-stop : Make sure the container is stopped even if container_cli do not match] ***
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.16] => (item=controller-0) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_api /bin/true; then\n systemctl stop tripleo_aodh_api.service\nfi\nif type docker &> /dev/null && docker exec aodh_api /bin/true; then\n docker stop aodh_api\nfi\n", "delta": "0:00:00.003134", "end": "2019-12-13 00:54:10.717540", "rc": 0, "start": "2019-12-13 00:54:10.714406", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": [], "tripleo_delegate_to_item": "controller-0"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.23] => (item=controller-1) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_api /bin/true; then\n systemctl stop tripleo_aodh_api.service\nfi\nif type docker &> /dev/null && docker exec aodh_api /bin/true; then\n docker stop aodh_api\nfi\n", "delta": "0:00:00.429473", "end": "2019-12-13 00:54:11.693939", "rc": 0, "start": "2019-12-13 00:54:11.264466", "stderr": "", "stderr_lines": [], "stdout": "aodh_api", "stdout_lines": ["aodh_api"], "tripleo_delegate_to_item": "controller-1"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.20] => (item=controller-2) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_api /bin/true; then\n systemctl stop tripleo_aodh_api.service\nfi\nif type docker &> /dev/null && docker exec aodh_api /bin/true; then\n docker stop aodh_api\nfi\n", "delta": "0:00:00.444288", "end": "2019-12-13 00:54:12.688377", "rc": 0, "start": "2019-12-13 00:54:12.244089", "stderr": "", "stderr_lines": [], "stdout": "aodh_api", "stdout_lines": ["aodh_api"], "tripleo_delegate_to_item": "controller-2"}
2019-12-12 19:54:33 | Thursday 12 December 2019 19:54:12 -0500 (0:00:02.635) 0:00:06.946 *****
2019-12-12 19:54:33 | included: /usr/share/ansible/roles/tripleo-container-stop/tasks/container_stop.yaml for undercloud
2019-12-12 19:54:33 | Thursday 12 December 2019 19:54:13 -0500 (0:00:00.364) 0:00:07.311 *****
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | TASK [tripleo-container-stop : Make sure the container is stopped even if container_cli do not match] ***
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.16] => (item=controller-0) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_evaluator /bin/true; then\n systemctl stop tripleo_aodh_evaluator.service\nfi\nif type docker &> /dev/null && docker exec aodh_evaluator /bin/true; then\n docker stop aodh_evaluator\nfi\n", "delta": "0:00:00.003249", "end": "2019-12-13 00:54:13.360661", "rc": 0, "start": "2019-12-13 00:54:13.357412", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": [], "tripleo_delegate_to_item": "controller-0"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.23] => (item=controller-1) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_evaluator /bin/true; then\n systemctl stop tripleo_aodh_evaluator.service\nfi\nif type docker &> /dev/null && docker exec aodh_evaluator /bin/true; then\n docker stop aodh_evaluator\nfi\n", "delta": "0:00:00.476587", "end": "2019-12-13 00:54:14.037990", "rc": 0, "start": "2019-12-13 00:54:13.561403", "stderr": "", "stderr_lines": [], "stdout": "aodh_evaluator", "stdout_lines": ["aodh_evaluator"], "tripleo_delegate_to_item": "controller-1"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.20] => (item=controller-2) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_evaluator /bin/true; then\n systemctl stop tripleo_aodh_evaluator.service\nfi\nif type docker &> /dev/null && docker exec aodh_evaluator /bin/true; then\n docker stop aodh_evaluator\nfi\n", "delta": "0:00:00.383456", "end": "2019-12-13 00:54:14.587164", "rc": 0, "start": "2019-12-13 00:54:14.203708", "stderr": "", "stderr_lines": [], "stdout": "aodh_evaluator", "stdout_lines": ["aodh_evaluator"], "tripleo_delegate_to_item": "controller-2"}
2019-12-12 19:54:33 | Thursday 12 December 2019 19:54:14 -0500 (0:00:01.528) 0:00:08.839 *****
2019-12-12 19:54:33 | included: /usr/share/ansible/roles/tripleo-container-stop/tasks/container_stop.yaml for undercloud
2019-12-12 19:54:33 | Thursday 12 December 2019 19:54:15 -0500 (0:00:00.353) 0:00:09.192 *****
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | TASK [tripleo-container-stop : Make sure the container is stopped even if container_cli do not match] ***
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.16] => (item=controller-0) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_listener /bin/true; then\n systemctl stop tripleo_aodh_listener.service\nfi\nif type docker &> /dev/null && docker exec aodh_listener /bin/true; then\n docker stop aodh_listener\nfi\n", "delta": "0:00:00.002851", "end": "2019-12-13 00:54:15.255944", "rc": 0, "start": "2019-12-13 00:54:15.253093", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": [], "tripleo_delegate_to_item": "controller-0"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.23] => (item=controller-1) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_listener /bin/true; then\n systemctl stop tripleo_aodh_listener.service\nfi\nif type docker &> /dev/null && docker exec aodh_listener /bin/true; then\n docker stop aodh_listener\nfi\n", "delta": "0:00:10.244591", "end": "2019-12-13 00:54:25.684357", "rc": 0, "start": "2019-12-13 00:54:15.439766", "stderr": "", "stderr_lines": [], "stdout": "aodh_listener", "stdout_lines": ["aodh_listener"], "tripleo_delegate_to_item": "controller-1"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.20] => (item=controller-2) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_listener /bin/true; then\n systemctl stop tripleo_aodh_listener.service\nfi\nif type docker &> /dev/null && docker exec aodh_listener /bin/true; then\n docker stop aodh_listener\nfi\n", "delta": "0:00:01.857296", "end": "2019-12-13 00:54:27.728160", "rc": 0, "start": "2019-12-13 00:54:25.870864", "stderr": "", "stderr_lines": [], "stdout": "aodh_listener", "stdout_lines": ["aodh_listener"], "tripleo_delegate_to_item": "controller-2"}
2019-12-12 19:54:33 | Thursday 12 December 2019 19:54:27 -0500 (0:00:12.800) 0:00:21.993 *****
2019-12-12 19:54:33 | included: /usr/share/ansible/roles/tripleo-container-stop/tasks/container_stop.yaml for undercloud
2019-12-12 19:54:33 | Thursday 12 December 2019 19:54:28 -0500 (0:00:00.360) 0:00:22.354 *****
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | TASK [tripleo-container-stop : Make sure the container is stopped even if container_cli do not match] ***
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.16] => (item=controller-0) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_notifier /bin/true; then\n systemctl stop tripleo_aodh_notifier.service\nfi\nif type docker &> /dev/null && docker exec aodh_notifier /bin/true; then\n docker stop aodh_notifier\nfi\n", "delta": "0:00:00.003380", "end": "2019-12-13 00:54:28.437905", "rc": 0, "start": "2019-12-13 00:54:28.434525", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": [], "tripleo_delegate_to_item": "controller-0"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.23] => (item=controller-1) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_notifier /bin/true; then\n systemctl stop tripleo_aodh_notifier.service\nfi\nif type docker &> /dev/null && docker exec aodh_notifier /bin/true; then\n docker stop aodh_notifier\nfi\n", "delta": "0:00:01.966848", "end": "2019-12-13 00:54:30.609018", "rc": 0, "start": "2019-12-13 00:54:28.642170", "stderr": "", "stderr_lines": [], "stdout": "aodh_notifier", "stdout_lines": ["aodh_notifier"], "tripleo_delegate_to_item": "controller-1"}
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0643 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |