Bug 1796098

Summary:	heat_api_wsgi fails on undercloud - unable to locate config file api-paste.ini
Product:	Red Hat OpenStack	Reporter:	Pavel Sedlák <psedlak>
Component:	python-oslo-config	Assignee:	Hervé Beraud <hberaud>
Status:	CLOSED ERRATA	QA Contact:	nlevinki <nlevinki>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	15.0 (Stein)	CC:	apevec, aschultz, bdobreli, eglynn, lhh, markmc, mburns, sbaker, shardy
Target Milestone:	---	Keywords:	AutomationBlocker, Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-oslo-config-6.8.2-0.20200129215025.0a9c6c1.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-05 12:03:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Pavel Sedlák 2020-01-29 15:46:10 UTC

OSP15 OC deploy fails in ~2 seconds with:

> Exception occured while running the command
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
>     super(Command, self).run(parsed_args)
>   File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
>     return super(Command, self).run(parsed_args)
>   File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
>     return_code = self.take_action(parsed_args) or 0
>   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 914, in take_action
>     stack = utils.get_stack(self.orchestration_client, parsed_args.stack)
>   File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 723, in get_stack
>     stack = orchestration_client.stacks.get(stack_name)
>   File "/usr/lib/python3.6/site-packages/heatclient/v1/stacks.py", line 280, in get
>     resp = self.client.get('/stacks/%s' % stack_id, **kwargs)
>   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 375, in get
>     return self.request(url, 'GET', **kwargs)
>   File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 320, in request
>     raise exc.from_response(resp)
> heatclient.exc.HTTPServiceUnavailable: ERROR: b'<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n'
> ERROR: b'<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n'
> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 38018), raddr=('192.168.24.2', 13000)>

Seems heat-api is not responding/working (ip/port obtained from haproxy config for heat-api's backend value):
> $ curl -v 'http://192.168.24.1:8004/v1/'
> ...
> < HTTP/1.1 500 Internal Server Error
same as when using the client:
> [stack@undercloud-0 ~]$ source stackrc
>
> (undercloud) [stack@undercloud-0 ~]$ openstack stack list
> ERROR: b'<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n'
>
> (undercloud) [stack@undercloud-0 ~]$ heat stack-list
> WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
> /usr/lib/python3.6/site-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
>   SubjectAltNameWarning
> /usr/lib/python3.6/site-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
>   SubjectAltNameWarning
> b"ERROR: b'<html><body><h1>503 Service Unavailable</h1>\\nNo server is available to handle this request.\\n</body></html>\\n'"

Inside heat-api containers, found errors in httpd logs:
> [root@undercloud-0 ~]# podman exec -it heat_api bash
> ()[root@undercloud-0 /]# tail /var/log/httpd/heat_api_wsgi_error.log -n18
> [Wed Jan 29 13:22:00.019905 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256] mod_wsgi (pid=19): Failed to exec Python script file '/var/www/cgi-bin/heat/heat_api'.
> [Wed Jan 29 13:22:00.019953 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256] mod_wsgi (pid=19): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
> [Wed Jan 29 13:22:00.020105 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256] Traceback (most recent call last):
> [Wed Jan 29 13:22:00.020146 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
> [Wed Jan 29 13:22:00.020150 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     application = init_application()
> [Wed Jan 29 13:22:00.020156 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 35, in init_application
> [Wed Jan 29 13:22:00.020164 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     logging.register_options(cfg.CONF)
> [Wed Jan 29 13:22:00.020169 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/usr/lib/python3.6/site-packages/oslo_log/log.py", line 260, in register_options
> [Wed Jan 29 13:22:00.020173 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     conf.register_cli_opts(_options.common_cli_opts)
> [Wed Jan 29 13:22:00.020178 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2041, in __inner
> [Wed Jan 29 13:22:00.020181 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     result = f(self, *args, **kwargs)
> [Wed Jan 29 13:22:00.020185 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2327, in register_cli_opts
> [Wed Jan 29 13:22:00.020188 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     self.register_cli_opt(opt, group, clear_cache=False)
> [Wed Jan 29 13:22:00.020193 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2045, in __inner
> [Wed Jan 29 13:22:00.020196 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     return f(self, *args, **kwargs)
> [Wed Jan 29 13:22:00.020201 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2319, in register_cli_opt
> [Wed Jan 29 13:22:00.020204 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     raise ArgsAlreadyParsedError("cannot register CLI option")
> [Wed Jan 29 13:22:00.020219 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256] oslo_config.cfg.ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option

but seems that first issue was actually, as also same appears after systemctl restart tripleo_heat_api.service:
> [Wed Jan 29 10:52:04.746623 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256] mod_wsgi (pid=19): Exception occurred processing WSGI script '/var/www/cgi-bin/heat/heat_api'.
> [Wed Jan 29 10:52:04.747070 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256] Traceback (most recent call last):
> [Wed Jan 29 10:52:04.747097 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/var/www/cgi-bin/heat/heat_api", line 52, in <module>
> [Wed Jan 29 10:52:04.747101 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     application = init_application()
> [Wed Jan 29 10:52:04.747106 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/usr/lib/python3.6/site-packages/heat/httpd/heat_api.py", line 47, in init_application
> [Wed Jan 29 10:52:04.747108 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     return config.load_paste_app()
> [Wed Jan 29 10:52:04.747112 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]   File "/usr/lib/python3.6/site-packages/heat/common/config.py", line 502, in load_paste_app
> [Wed Jan 29 10:52:04.747115 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256]     cfg.CONF.paste_deploy['api_paste_config'])
> [Wed Jan 29 10:52:04.747127 2020] [wsgi:error] [pid 19] [remote 192.168.24.1:60256] RuntimeError: Unable to locate config file [api-paste.ini]


rpm versions on undercloud:
> heat-cfntools.noarch                          1.4.2-6.el8ost                                  @rhelosp-15.0-trunk                     
> openstack-heat-agents.noarch                  1.8.1-0.20190523210450.1e15344.el8ost           @rhelosp-15.0-trunk                     
> openstack-heat-api.noarch                     1:12.0.1-0.20200114010454.3899367.el8ost        @rhelosp-15.0-trunk                     
> openstack-heat-common.noarch                  1:12.0.1-0.20200114010454.3899367.el8ost        @rhelosp-15.0-trunk                     
> openstack-heat-engine.noarch                  1:12.0.1-0.20200114010454.3899367.el8ost        @rhelosp-15.0-trunk                     
> openstack-heat-monolith.noarch                1:12.0.1-0.20200114010454.3899367.el8ost        @rhelosp-15.0-trunk                     
> openstack-tripleo-heat-templates.noarch       10.6.3-0.20200125110450.0119358.el8ost          @rhelosp-15.0-trunk                     
> puppet-heat.noarch                            14.4.1-0.20190420110320.4425351.el8ost          @rhelosp-15.0-trunk                     
> python3-heat-agent.noarch                     1.8.1-0.20190523210450.1e15344.el8ost           @rhelosp-15.0-trunk                     
> python3-heat-agent-ansible.noarch             1.8.1-0.20190523210450.1e15344.el8ost           @rhelosp-15.0-trunk                     
> python3-heat-agent-apply-config.noarch        1.8.1-0.20190523210450.1e15344.el8ost           @rhelosp-15.0-trunk                     
> python3-heat-agent-docker-cmd.noarch          1.8.1-0.20190523210450.1e15344.el8ost           @rhelosp-15.0-trunk                     
> python3-heat-agent-hiera.noarch               1.8.1-0.20190523210450.1e15344.el8ost           @rhelosp-15.0-trunk                     
> python3-heat-agent-json-file.noarch           1.8.1-0.20190523210450.1e15344.el8ost           @rhelosp-15.0-trunk                     
> python3-heat-agent-puppet.noarch              1.8.1-0.20190523210450.1e15344.el8ost           @rhelosp-15.0-trunk                     
> python3-heatclient.noarch                     1.17.0-0.20190312144725.8af5deb.el8ost          @rhelosp-15.0-trunk                     
> python3-tripleoclient-heat-installer.noarch   11.5.3-0.20200114200459.d09212f.el8ost          @rhelosp-15.0-trunk                     
> python3-oslo-config.noarch                    2:6.8.2-0.20200113183332.0a9c6c1.el8ost         @rhelosp-15.0-trunk                     
heat-api container:
> 192.168.24.1:8787/rh-osbs/rhosp15-openstack-heat-api:20200124.1

Comment 2 Alex Schultz 2020-01-29 21:25:37 UTC

Looks like https://bugzilla.redhat.com/show_bug.cgi?id=1751718 is back/affecting OSP15 now. We have a downstream patch that needs to be applied to the python-oslo-config packaging so that it properly finds the dist versions that we ship.

Comment 3 Alex Schultz 2020-01-29 21:35:58 UTC

We're missing https://github.com/rdo-packages/oslo-config-distgit/blob/stein-rdo/0001-add-usr-share-project-dist.conf-to-the-default-confi.patch from the OSP15 branch in the distgit.

Comment 5 Bogdan Dobrelya 2020-01-30 09:11:20 UTC

As Hervé and Rabi mentioned in bz 1751718,

- there is https://docs.openstack.org/oslo.config/latest/configuration/options.html#DEFAULT.config_dir
- we can't change caller/upstream code for distro specific config-dir location

To stop carrying on that downstream-only change, shall we instead move configs to the not distro specific path, like /etc/project/project.conf.d/ ?

Comment 6 Hervé Beraud 2020-01-30 09:56:59 UTC

Hello,

I agree with the Bogdan's proposal which seems to be a good and proper solution (moving config to /etc/project/project.conf.d/)

Hervé

Comment 8 Alan Pevec 2020-01-30 14:22:07 UTC

> To stop carrying on that downstream-only change, shall we instead move configs to the not distro specific path, like /etc/project/project.conf.d/ ?

yes please!

Comment 12 errata-xmlrpc 2020-03-05 12:03:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0710