Bug 1732573
| Summary: | TLS everywhere in place upgrade fails with a http 503 error | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jeremy Agee <jagee> |
| Component: | documentation | Assignee: | Roger Heslop <rheslop> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jeremy Agee <jagee> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 13.0 (Queens) | CC: | alee, broose, dmendiza, ggrasza, marjones, pkesavar, rcritten, rheslop |
| Target Milestone: | --- | Keywords: | Documentation, Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-09-15 11:54:52 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Jeremy Agee
2019-07-23 18:23:11 UTC
Description of problem: When doing a Brownfield TLS everywhere upgrade the overcloud deploy fails. Version-Release number of selected component (if applicable): Steps to Reproduce: 1. upgrade the undercloud to enable novajoin. undercloud.conf [DEFAULT] # BEGIN TLS EVERYWHERE SETTINGS --> enable_novajoin = True ipa_otp = 9Yx7dnbFEIvGPZ0X7MS4x02WDNaBL4G3JR8w3vsjB0p4 undercloud_hostname = undercloud-0.redhat.local undercloud_nameservers = 10.0.0.37 overcloud_domain_name = redhat.local # END TLS EVERYWHERE SETTINGS --> openstack undercloud install 2. upgrade the overcloud to enable tls everywhere openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server 10.0.0.37 \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /home/stack/virt/cloud-names.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-images.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml 2019-07-19 18:20:25Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.0]: UPDATE_IN_PROGRESS state changed 2019-07-19 18:20:26Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: UPDATE_IN_PROGRESS state changed 2019-07-19 18:20:27Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: UPDATE_IN_PROGRESS state changed 2019-07-19 18:21:08Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.0]: UPDATE_COMPLETE state changed 2019-07-19 18:21:08Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.0]: SIGNAL_IN_PROGRESS Signal: deployment 8fdb99a9-6726-423c-a4da-7401a5ddbfaa succeeded 2019-07-19 18:21:10Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.1]: SIGNAL_IN_PROGRESS Signal: deployment 8bfa71b2-2ca7-49e9-b16b-621c5b155688 succeeded 2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.1]: UPDATE_COMPLETE state changed 2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b]: UPDATE_COMPLETE Stack UPDATE completed successfully 2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu.ComputeDeployment_Step3]: UPDATE_COMPLETE state changed 2019-07-19 18:21:39Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: SIGNAL_IN_PROGRESS Signal: deployment 93875ebd-b8b7-480e-9f4c-f54547be1d45 succeeded 2019-07-19 18:21:39Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: UPDATE_COMPLETE state changed 2019-07-19 18:21:42Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: SIGNAL_IN_PROGRESS Signal: deployment a8def5a3-3b36-4e30-891b-28cada1fbd40 succeeded 2019-07-19 18:21:43Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: UPDATE_COMPLETE state changed ERROR: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>503 Service Unavailable</title> </head><body> <h1>Service Unavailable</h1> <p>The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.</p> </body></html> All nodes are enrolled in IPA and the endpoints are created. The heat logs show some tasks looping untill the timeout value is reached. [root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log 2019-07-19 14:28:58.761 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:28:58.887 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:28:58.987 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:28:59.762 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:28:59.763 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:28:59.765 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:28:59.770 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:28:59.771 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:28:59.988 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:29:00.069 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 [root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log 2019-07-19 14:43:17.193 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:17.198 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:43:17.199 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:43:17.872 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:17.949 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:43:18.199 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:18.200 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:18.204 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:18.208 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:43:18.211 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 [root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log 2019-07-19 15:09:24.066 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:24.082 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 15:09:24.089 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 15:09:25.044 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:25.044 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:25.050 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 15:09:25.083 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:25.095 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:25.108 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 15:09:25.165 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 How reproducible: Actual results: The upgrade deployment fails with a 503 error Expected results: All nodes enroll in IPA and the overcloud upgrade is successful. Additional info: Requested content is in documentation in version 13. Brownfield implementation of novajoin is not supported in OSP 161 and higher, no further action needed in documentation. If you feel this BZ is closed in error, please open another BZ. |