Bug 1732573 - TLS everywhere in place upgrade fails with a http 503 error
Summary: TLS everywhere in place upgrade fails with a http 503 error
Keywords:
Status: NEW
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: RHOS Documentation Team
QA Contact: Jeremy Agee
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-23 18:23 UTC by Jeremy Agee
Modified: 2020-02-20 04:53 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Jeremy Agee 2019-07-23 18:23:11 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Jeremy Agee 2019-07-23 18:23:46 UTC
Description of problem:
When doing a Brownfield TLS everywhere upgrade the overcloud deploy fails.

Version-Release number of selected component (if applicable):


Steps to Reproduce:

1. upgrade the undercloud to enable novajoin.

undercloud.conf

[DEFAULT]
# BEGIN TLS EVERYWHERE SETTINGS -->
enable_novajoin = True
ipa_otp = 9Yx7dnbFEIvGPZ0X7MS4x02WDNaBL4G3JR8w3vsjB0p4
undercloud_hostname = undercloud-0.redhat.local
undercloud_nameservers = 10.0.0.37
overcloud_domain_name = redhat.local
# END TLS EVERYWHERE SETTINGS -->

openstack undercloud install

2. upgrade the overcloud to enable tls everywhere

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server 10.0.0.37 \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /home/stack/virt/cloud-names.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml

2019-07-19 18:20:25Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.0]: UPDATE_IN_PROGRESS  state changed
2019-07-19 18:20:26Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: UPDATE_IN_PROGRESS  state changed
2019-07-19 18:20:27Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: UPDATE_IN_PROGRESS  state changed
2019-07-19 18:21:08Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.0]: UPDATE_COMPLETE  state changed
2019-07-19 18:21:08Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.0]: SIGNAL_IN_PROGRESS  Signal: deployment 8fdb99a9-6726-423c-a4da-7401a5ddbfaa succeeded
2019-07-19 18:21:10Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.1]: SIGNAL_IN_PROGRESS  Signal: deployment 8bfa71b2-2ca7-49e9-b16b-621c5b155688 succeeded
2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b.1]: UPDATE_COMPLETE  state changed
2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ComputeDeployment_Step3-6h4enzzxk66b]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2019-07-19 18:21:11Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu.ComputeDeployment_Step3]: UPDATE_COMPLETE  state changed
2019-07-19 18:21:39Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: SIGNAL_IN_PROGRESS  Signal: deployment 93875ebd-b8b7-480e-9f4c-f54547be1d45 succeeded
2019-07-19 18:21:39Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.1]: UPDATE_COMPLETE  state changed
2019-07-19 18:21:42Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: SIGNAL_IN_PROGRESS  Signal: deployment a8def5a3-3b36-4e30-891b-28cada1fbd40 succeeded
2019-07-19 18:21:43Z [overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm.2]: UPDATE_COMPLETE  state changed
ERROR: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

All nodes are enrolled in IPA and the endpoints are created. The heat logs show some tasks looping untill the timeout value is reached.

[root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log
2019-07-19 14:28:58.761 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 14:28:58.887 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:28:58.987 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 14:28:59.762 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:28:59.763 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:28:59.765 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:28:59.770 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 14:28:59.771 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 14:28:59.988 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:29:00.069 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150


[root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log
2019-07-19 14:43:17.193 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:43:17.198 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 14:43:17.199 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 14:43:17.872 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:43:17.949 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 14:43:18.199 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:43:18.200 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:43:18.204 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 14:43:18.208 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 14:43:18.211 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150

[root@undercloud-0 log]# tail -n10 /var/log/heat/heat-engine.log
2019-07-19 15:09:24.066 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 15:09:24.082 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 15:09:24.089 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 15:09:25.044 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 15:09:25.044 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task _run_to_completion from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 15:09:25.050 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeploymentGroup "ControllerDeployment_Step3" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 15:09:25.083 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 15:09:25.095 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-07-19 15:09:25.108 19991 DEBUG heat.engine.scheduler [req-698d48a2-133d-45db-b817-cd453f1f7b8c - - - - -] Task update from TemplateResource "AllNodesDeploySteps" [c0f56527-c4ae-4eab-8148-b60ce8b3c2b8] Stack "overcloud" [f0259fbf-2f88-4f23-b496-febc364aff3e] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150
2019-07-19 15:09:25.165 19991 DEBUG heat.engine.scheduler [req-6bdad891-017e-428f-924e-f75ff9a2475a - admin - default default] Task update from StructuredDeployment "0" [ad264b40-67d4-489b-ae20-a8dbff9355f3] Stack "overcloud-AllNodesDeploySteps-crpvtqxtvypu-ControllerDeployment_Step3-6h3vnntn22jm" [b8f266c9-76aa-4e20-a9de-7272c95dac6c] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:150


How reproducible:

Actual results:
The upgrade deployment fails with a 503 error

Expected results:
All nodes enroll in IPA and the overcloud upgrade is successful.

Additional info:


Note You need to log in before you can comment on or make changes to this bug.