Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1757074

Summary: Overcloud deployment immediately fails with Python exception 503 service unavailable
Product: Red Hat OpenStack Reporter: Nathan Weinberg <nweinber>
Component: openstack-heatAssignee: Zane Bitter <zbitter>
Status: CLOSED DUPLICATE QA Contact: Victor Voronkov <vvoronko>
Severity: urgent Docs Contact:
Priority: high    
Version: 16.0 (Train)CC: aschultz, hbrock, jslagle, mburns, sbaker, shardy
Target Milestone: ---Keywords: Triaged
Target Release: ---Flags: nweinber: needinfo+
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-30 17:40:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Files and folders from undercloud
none
sosreport from undercloud none

Description Nathan Weinberg 2019-09-30 14:04:43 UTC 
Created attachment 1621157 [details]
Files and folders from undercloud

Description of problem:
Upon executing the deployment script for OpenStack `openstack overcloud deploy` the command fails with a 503 error from Python almost immediately after running. When testing other OpenStack nouns, the command completes successfully; for example, `openstack baremetal node list`. Heat commands are verified to work as well.

Version-Release number of selected component (if applicable):
Core Puddle Version: RHOS_TRUNK-16.0-RHEL-8-20190916.n.0
OpenStack RPMs: [1]

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP16 
2. After introspection and tagging run `openstack overcloud deploy...`
3.

Actual results:
[2]

Expected results:
Successful Overcloud Deployment

Additional info:
[1] http://pastebin.test.redhat.com/801723
[2] http://pastebin.test.redhat.com/801725
Comment 1 Alex Schultz 2019-09-30 15:04:56 UTC 
Please provide a sosreport. It seems to indicate that keystone is unavailable.
Comment 2 Nathan Weinberg 2019-09-30 15:19:02 UTC 
Created attachment 1621167 [details]
sosreport from undercloud
Comment 3 Alex Schultz 2019-09-30 16:40:10 UTC 
sosreport needs to be run as root. There are no logs in that file.
Comment 4 Alex Schultz 2019-09-30 17:36:46 UTC 
The actual error:

[Mon Sep 30 15:25:48.266482 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] Traceback (most recent call last):
[Mon Sep 30 15:25:48.266544 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
[Mon Sep 30 15:25:48.266550 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]     application = init_application()
[Mon Sep 30 15:25:48.266557 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 35, in init_application
[Mon Sep 30 15:25:48.266561 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]     logging.register_options(cfg.CONF)
[Mon Sep 30 15:25:48.266566 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]   File "/usr/lib/python3.6/site-packages/oslo_log/log.py", line 260, in register_options
[Mon Sep 30 15:25:48.266569 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]     conf.register_cli_opts(_options.common_cli_opts)
[Mon Sep 30 15:25:48.266574 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2051, in __inner
[Mon Sep 30 15:25:48.266577 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]     result = f(self, *args, **kwargs)
[Mon Sep 30 15:25:48.266582 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2337, in register_cli_opts
[Mon Sep 30 15:25:48.266584 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]     self.register_cli_opt(opt, group, clear_cache=False)
[Mon Sep 30 15:25:48.266589 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2055, in __inner
[Mon Sep 30 15:25:48.266592 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]     return f(self, *args, **kwargs)
[Mon Sep 30 15:25:48.266596 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2329, in register_cli_opt
[Mon Sep 30 15:25:48.266599 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092]     raise ArgsAlreadyParsedError("cannot register CLI option")
[Mon Sep 30 15:25:48.266614 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] oslo_config.cfg.ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option
[Mon Sep 30 15:25:50.267290 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:33120] mod_wsgi (pid=21): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'.
[Mon Sep 30 15:25:50.267334 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:33120] mod_wsgi (pid=21): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
Comment 5 Alex Schultz 2019-09-30 17:40:09 UTC 
The containers are too old and are missing the fix.

[stack@undercloud-0 oslo_config]$ sudo podman exec -it -u root heat_api rpm -qa | grep oslo-config
python3-oslo-config-6.11.1-0.20190904074339.c4d8fa7.el8ost.noarch
Comment 6 Alex Schultz 2019-09-30 17:40:23 UTC 

*** This bug has been marked as a duplicate of bug 1751718 ***