| Summary: | TLS everywhere in place upgrade fails with a http 503 error | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jeremy Agee <jagee> |
| Component: | documentation | Assignee: | RHOS Documentation Team <rhos-docs> |
| Status: | NEW --- | QA Contact: | Jeremy Agee <jagee> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 13.0 (Queens) | CC: | alee, broose, dmendiza, ggrasza, marjones, pkesavar, rcritten |
| Target Milestone: | --- | Keywords: | Documentation, Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
|
Description
Jeremy Agee
2019-07-23 18:23:11 UTC
Description of problem: When doing a Brownfield TLS everywhere upgrade the overcloud deploy fails. Version-Release number of selected component (if applicable): Steps to Reproduce: 1. upgrade the undercloud to enable novajoin. undercloud.conf [DEFAULT] # BEGIN TLS EVERYWHERE SETTINGS --> enable_novajoin = True ipa_otp = 9Yx7dnbFEIvGPZ0X7MS4x02WDNaBL4G3JR8w3vsjB0p4 undercloud_hostname = undercloud-0.redhat.local undercloud_nameservers = 10.0.0.37 overcloud_domain_name = redhat.local # END TLS EVERYWHERE SETTINGS --> openstack undercloud install 2. upgrade the overcloud to enable tls everywhere openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server 10.0.0.37 \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /home/stack/virt/cloud-names.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-images.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml 2019-07-19 18:20:25Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.0]: UPDATE_IN_PROGRESS state changed 2019-07-19 18:20:26Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: UPDATE_IN_PROGRESS state changed 2019-07-19 18:20:27Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: UPDATE_IN_PROGRESS state changed 2019-07-19 18:21:08Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.0]: UPDATE_COMPLETE state changed 2019-07-19 18:21:08Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.0]: SIGNAL_IN_PROGRESS Signal: deployment 8fdb99a9-6726-423c-a4da-7401a5ddbfaa succeeded 2019-07-19 18:21:10Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.1]: SIGNAL_IN_PROGRESS Signal: deployment 8bfa71b2-2ca7-49e9-b16b-621c5b155688 succeeded 2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.1]: UPDATE_COMPLETE state changed 2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b]: UPDATE_COMPLETE Stack UPDATE completed successfully 2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu.ComputeDeployment_Step3]: UPDATE_COMPLETE state changed 2019-07-19 18:21:39Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: SIGNAL_IN_PROGRESS Signal: deployment 93875ebd-b8b7-480e-9f4c-f54547be1d45 succeeded 2019-07-19 18:21:39Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: UPDATE_COMPLETE state changed 2019-07-19 18:21:42Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: SIGNAL_IN_PROGRESS Signal: deployment a8def5a3-3b36-4e30-891b-28cada1fbd40 succeeded 2019-07-19 18:21:43Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: UPDATE_COMPLETE state changed ERROR: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>503 Service Unavailable</title> </head><body> <h1>Service Unavailable</h1> <p>The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.</p> </body></html> All nodes are enrolled in IPA and the endpoints are created. The heat logs show some tasks looping untill the timeout value is reached. [root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log 2019-07-19 14:28:58.761 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:28:58.887 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:28:58.987 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:28:59.762 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:28:59.763 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:28:59.765 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:28:59.770 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:28:59.771 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:28:59.988 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:29:00.069 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 [root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log 2019-07-19 14:43:17.193 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:17.198 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:43:17.199 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:43:17.872 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:17.949 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:43:18.199 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:18.200 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:18.204 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 14:43:18.208 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 14:43:18.211 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 [root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log 2019-07-19 15:09:24.066 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:24.082 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 15:09:24.089 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 15:09:25.044 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:25.044 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:25.050 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 15:09:25.083 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:25.095 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2019-07-19 15:09:25.108 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 2019-07-19 15:09:25.165 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150 How reproducible: Actual results: The upgrade deployment fails with a 503 error Expected results: All nodes enroll in IPA and the overcloud upgrade is successful. Additional info: |