Created attachment 1621157 [details] Files and folders from undercloud Description of problem: Upon executing the deployment script for OpenStack `openstack overcloud deploy` the command fails with a 503 error from Python almost immediately after running. When testing other OpenStack nouns, the command completes successfully; for example, `openstack baremetal node list`. Heat commands are verified to work as well. Version-Release number of selected component (if applicable): Core Puddle Version: RHOS_TRUNK-16.0-RHEL-8-20190916.n.0 OpenStack RPMs: [1] How reproducible: 100% Steps to Reproduce: 1. Deploy OSP16 2. After introspection and tagging run `openstack overcloud deploy...` 3. Actual results: [2] Expected results: Successful Overcloud Deployment Additional info: [1] http://pastebin.test.redhat.com/801723 [2] http://pastebin.test.redhat.com/801725
Please provide a sosreport. It seems to indicate that keystone is unavailable.
Created attachment 1621167 [details] sosreport from undercloud
sosreport needs to be run as root. There are no logs in that file.
The actual error: [Mon Sep 30 15:25:48.266482 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] Traceback (most recent call last): [Mon Sep 30 15:25:48.266544 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] File "/var/www/cgi-bin/heat/heat_api", line 52, in <module> [Mon Sep 30 15:25:48.266550 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] application = init_application() [Mon Sep 30 15:25:48.266557 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 35, in init_application [Mon Sep 30 15:25:48.266561 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] logging.register_options(cfg.CONF) [Mon Sep 30 15:25:48.266566 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] File "/usr/lib/python3.6/site-packages/oslo_log/log.py", line 260, in register_options [Mon Sep 30 15:25:48.266569 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] conf.register_cli_opts(_options.common_cli_opts) [Mon Sep 30 15:25:48.266574 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2051, in __inner [Mon Sep 30 15:25:48.266577 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] result = f(self, *args, **kwargs) [Mon Sep 30 15:25:48.266582 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2337, in register_cli_opts [Mon Sep 30 15:25:48.266584 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] self.register_cli_opt(opt, group, clear_cache=False) [Mon Sep 30 15:25:48.266589 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2055, in __inner [Mon Sep 30 15:25:48.266592 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] return f(self, *args, **kwargs) [Mon Sep 30 15:25:48.266596 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2329, in register_cli_opt [Mon Sep 30 15:25:48.266599 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] raise ArgsAlreadyParsedError("cannot register CLI option") [Mon Sep 30 15:25:48.266614 2019] [wsgi:error] [pid 22] [remote 192.168.24.1:33092] oslo_config.cfg.ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option [Mon Sep 30 15:25:50.267290 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:33120] mod_wsgi (pid=21): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'. [Mon Sep 30 15:25:50.267334 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:33120] mod_wsgi (pid=21): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
The containers are too old and are missing the fix. [stack@undercloud-0 oslo_config]$ sudo podman exec -it -u root heat_api rpm -qa | grep oslo-config python3-oslo-config-6.11.1-0.20190904074339.c4d8fa7.el8ost.noarch
*** This bug has been marked as a duplicate of bug 1751718 ***