Bug 1751718 - dist.conf lookup dropped from python-oslo-config causing Containerized heat-api fails to start/operate properly, locate api-paste.ini
Summary: dist.conf lookup dropped from python-oslo-config causing Containerized heat-a...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-oslo-config
Version: 16.0 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Upstream M3
: 16.0 (Train on RHEL 8.1)
Assignee: Hervé Beraud
QA Contact: nlevinki
URL:
Whiteboard:
: 1757074 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-12 13:01 UTC by Attila Fazekas
Modified: 2020-02-06 14:42 UTC (History)
12 users (show)

Fixed In Version: python-oslo-config-6.11.1-0.20190913082449.c4d8fa7.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-02-06 14:42:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2020:0283 0 None None None 2020-02-06 14:42:30 UTC

Description Attila Fazekas 2019-09-12 13:01:44 UTC 
Description of problem:

Overcloud deployment fails, when the client is tring to communicate with the undercloud heat.


(undercloud) [root@undercloud-0 heat]# podman ps |grep cc5ea3672199
cc5ea3672199  192.168.24.1:8787/rhosp16/openstack-heat-api:20190910.1                   dumb-init --singl...  About an hour ago  Up About an hour ago         heat_api
(undercloud) [root@undercloud-0 heat]# podman exec cc5ea3672199 head -n 80 /var/log/httpd/heat_api_wsgi_error.log
[Thu Sep 12 11:26:37.587552 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354] mod_wsgi (pid=23): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:37.587668 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354] mod_wsgi (pid=23): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:37.588224 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354] Traceback (most recent call last):
[Thu Sep 12 11:26:37.588268 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
[Thu Sep 12 11:26:37.588273 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354]     application = init_application()
[Thu Sep 12 11:26:37.588280 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 47, in init_application
[Thu Sep 12 11:26:37.588284 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354]     return config.load_paste_app()
[Thu Sep 12 11:26:37.588290 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354]   File "/usr/lib/python3.6/site-packages/heat/common/config.py", line 514, in load_paste_app
[Thu Sep 12 11:26:37.588293 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354]     cfg.CONF.paste_deploy['api_paste_config'])
[Thu Sep 12 11:26:37.588311 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48354] RuntimeError: Unable to locate config file [api-paste.ini]
[Thu Sep 12 11:26:42.273244 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412] mod_wsgi (pid=21): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:42.273360 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412] mod_wsgi (pid=21): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:42.274014 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412] Traceback (most recent call last):
[Thu Sep 12 11:26:42.274081 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
[Thu Sep 12 11:26:42.274089 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412]     application = init_application()
[Thu Sep 12 11:26:42.274099 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 47, in init_application
[Thu Sep 12 11:26:42.274105 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412]     return config.load_paste_app()
[Thu Sep 12 11:26:42.274114 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412]   File "/usr/lib/python3.6/site-packages/heat/common/config.py", line 514, in load_paste_app
[Thu Sep 12 11:26:42.274120 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412]     cfg.CONF.paste_deploy['api_paste_config'])
[Thu Sep 12 11:26:42.274145 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48412] RuntimeError: Unable to locate config file [api-paste.ini]
[Thu Sep 12 11:26:44.277492 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534] mod_wsgi (pid=23): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:44.277604 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534] mod_wsgi (pid=23): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:44.279139 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534] Traceback (most recent call last):
[Thu Sep 12 11:26:44.279245 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
[Thu Sep 12 11:26:44.279254 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]     application = init_application()
[Thu Sep 12 11:26:44.279267 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 35, in init_application
[Thu Sep 12 11:26:44.279274 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]     logging.register_options(cfg.CONF)
[Thu Sep 12 11:26:44.279285 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]   File "/usr/lib/python3.6/site-packages/oslo_log/log.py", line 260, in register_options
[Thu Sep 12 11:26:44.279319 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]     conf.register_cli_opts(_options.common_cli_opts)
[Thu Sep 12 11:26:44.279333 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2051, in __inner
[Thu Sep 12 11:26:44.279339 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]     result = f(self, *args, **kwargs)
[Thu Sep 12 11:26:44.279350 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2337, in register_cli_opts
[Thu Sep 12 11:26:44.279356 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]     self.register_cli_opt(opt, group, clear_cache=False)
[Thu Sep 12 11:26:44.279367 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2055, in __inner
[Thu Sep 12 11:26:44.279373 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]     return f(self, *args, **kwargs)
[Thu Sep 12 11:26:44.279383 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2329, in register_cli_opt
[Thu Sep 12 11:26:44.279390 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534]     raise ArgsAlreadyParsedError("cannot register CLI option")
[Thu Sep 12 11:26:44.279421 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48534] oslo_config.cfg.ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option
[Thu Sep 12 11:26:48.372963 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562] mod_wsgi (pid=20): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:48.373041 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562] mod_wsgi (pid=20): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:48.373456 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562] Traceback (most recent call last):
[Thu Sep 12 11:26:48.373497 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
[Thu Sep 12 11:26:48.373502 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562]     application = init_application()
[Thu Sep 12 11:26:48.373510 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 47, in init_application
[Thu Sep 12 11:26:48.373513 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562]     return config.load_paste_app()
[Thu Sep 12 11:26:48.373519 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562]   File "/usr/lib/python3.6/site-packages/heat/common/config.py", line 514, in load_paste_app
[Thu Sep 12 11:26:48.373523 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562]     cfg.CONF.paste_deploy['api_paste_config'])
[Thu Sep 12 11:26:48.373541 2019] [wsgi:error] [pid 20] [remote 192.168.24.1:48562] RuntimeError: Unable to locate config file [api-paste.ini]
[Thu Sep 12 11:26:50.378079 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626] mod_wsgi (pid=23): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:50.378173 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626] mod_wsgi (pid=23): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:50.378499 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626] Traceback (most recent call last):
[Thu Sep 12 11:26:50.378641 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
[Thu Sep 12 11:26:50.378661 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]     application = init_application()
[Thu Sep 12 11:26:50.378678 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 35, in init_application
[Thu Sep 12 11:26:50.378685 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]     logging.register_options(cfg.CONF)
[Thu Sep 12 11:26:50.378710 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]   File "/usr/lib/python3.6/site-packages/oslo_log/log.py", line 260, in register_options
[Thu Sep 12 11:26:50.378718 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]     conf.register_cli_opts(_options.common_cli_opts)
[Thu Sep 12 11:26:50.378730 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2051, in __inner
[Thu Sep 12 11:26:50.378737 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]     result = f(self, *args, **kwargs)
[Thu Sep 12 11:26:50.378747 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2337, in register_cli_opts
[Thu Sep 12 11:26:50.378754 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]     self.register_cli_opt(opt, group, clear_cache=False)
[Thu Sep 12 11:26:50.378765 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2055, in __inner
[Thu Sep 12 11:26:50.378771 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]     return f(self, *args, **kwargs)
[Thu Sep 12 11:26:50.378782 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2329, in register_cli_opt
[Thu Sep 12 11:26:50.378788 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626]     raise ArgsAlreadyParsedError("cannot register CLI option")
[Thu Sep 12 11:26:50.378823 2019] [wsgi:error] [pid 23] [remote 192.168.24.1:48626] oslo_config.cfg.ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option
[Thu Sep 12 11:26:52.383134 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654] mod_wsgi (pid=21): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:52.383225 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654] mod_wsgi (pid=21): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
[Thu Sep 12 11:26:52.385734 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654] Traceback (most recent call last):
[Thu Sep 12 11:26:52.385842 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
[Thu Sep 12 11:26:52.385853 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]     application = init_application()
[Thu Sep 12 11:26:52.385867 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 35, in init_application
[Thu Sep 12 11:26:52.385874 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]     logging.register_options(cfg.CONF)
[Thu Sep 12 11:26:52.385886 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]   File "/usr/lib/python3.6/site-packages/oslo_log/log.py", line 260, in register_options
[Thu Sep 12 11:26:52.385893 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]     conf.register_cli_opts(_options.common_cli_opts)
[Thu Sep 12 11:26:52.385904 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2051, in __inner
[Thu Sep 12 11:26:52.385911 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]     result = f(self, *args, **kwargs)
[Thu Sep 12 11:26:52.385923 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2337, in register_cli_opts
[Thu Sep 12 11:26:52.385929 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]     self.register_cli_opt(opt, group, clear_cache=False)
[Thu Sep 12 11:26:52.385940 2019] [wsgi:error] [pid 21] [remote 192.168.24.1:48654]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2055, in __inner


Version-Release number of selected component (if applicable):

RHOS_TRUNK-16.0-RHEL-8-20190911.n.0

Some pkg version in the container:
puppet-heat-15.3.0-0.20190907001309.e96248b.el8ost.noarch
python3-heatclient-1.18.0-0.20190829192744.eca1637.el8ost.noarch
openstack-heat-common-12.1.0-0.20190906115132.6c21719.el8ost.noarch
openstack-heat-api-12.1.0-0.20190906115132.6c21719.el8ost.noarch
python3-oslo-config-6.11.1-0.20190904074339.c4d8fa7.el8ost.noarch


Additional info:

"RuntimeError: Unable to locate config file [api-paste.ini]  " is the first error, 
the 'arguments already parsed: cannot register CLI option' is repeated many times. 


(undercloud) [root@undercloud-0 heat]# podman exec cc5ea3672199 ls -lR /etc/heat
/etc/heat:
total 72
drwxr-xr-x. 2 root root    26 Sep 11 03:31 environment.d
-rw-r-----. 1 root heat 68863 Sep 12 11:12 heat.conf
-rw-r-----. 1 root heat    40 Sep 12 11:12 policy.json
drwxr-xr-x. 2 root root    70 Sep 11 03:31 templates

/etc/heat/environment.d:
total 4
-rw-r--r--. 1 root heat 831 Jun 27 23:31 default.yaml

/etc/heat/templates:
total 8
-rw-r--r--. 1 root heat 2470 Jun 27 23:31 AWS_CloudWatch_Alarm.yaml
-rw-r--r--. 1 root heat 3804 Jun 27 23:31 AWS_RDS_DBInstance.yaml


(undercloud) [root@undercloud-0 heat]# podman exec cc5ea3672199 ls -l /usr/share/heat/api-paste-dist.ini
-rw-r--r--. 1 root heat 3498 Jun 27 23:31 /usr/share/heat/api-paste-dist.ini

The api-paste-dist.ini usage typically comes from /usr/share/heat/heat-dist.conf which is usually passed to the heat api service (when it is a service), but now it  is mod-wsgi daemon.
Comment 1 Alex Schultz 2019-09-12 21:01:46 UTC 
Found the issue. We have a downstream patch in the python-oslo-config package that adds our /usr/share/<project>/<project>-dist.conf to the default lookup path. This is missing for the osp16 package at teh moment.
Comment 2 Alex Schultz 2019-09-12 21:04:31 UTC 
Specifically, https://github.com/rdo-packages/oslo-config-distgit/blob/rpm-master/0001-add-usr-share-project-dist.conf-to-the-default-confi.patch
Comment 14 Alex Schultz 2019-09-30 17:40:23 UTC 
*** Bug 1757074 has been marked as a duplicate of this bug. ***
Comment 16 shreshtha joshi 2019-10-09 04:28:54 UTC 
Latest compose has python-oslo-config-6.11.1-0.20191004140143.c4d8fa7.el8ost. The fix has been in for a couple of last composes.
Comment 20 errata-xmlrpc 2020-02-06 14:42:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283
Note You need to log in before you can comment on or make changes to this bug.