openstack-heat-api-cfn.service is also down:

[root@director ~]# systemctl status openstack-heat-api.service
● openstack-heat-api.service - OpenStack Heat API Service
   Loaded: loaded (/usr/lib/systemd/system/openstack-heat-api.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-06-07 14:33:43 CDT; 1h 0min ago
  Process: 5190 ExecStart=/usr/bin/heat-api (code=exited, status=1/FAILURE)
 Main PID: 5190 (code=exited, status=1/FAILURE)

Jun 07 14:33:42 director.fv1dci.org systemd[1]: Unit openstack-heat-api.service entered failed state.
Jun 07 14:33:42 director.fv1dci.org systemd[1]: openstack-heat-api.service failed.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: openstack-heat-api.service holdoff time over, scheduling restart.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: start request repeated too quickly for openstack-heat-api.service
Jun 07 14:33:43 director.fv1dci.org systemd[1]: Failed to start OpenStack Heat API Service.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: Unit openstack-heat-api.service entered failed state.
Jun 07 14:33:43 director.fv1dci.org systemd[1]: openstack-heat-api.service failed.
[root@director ~]# journalctl -u openstack-heat-api.service|grep -i error
Jun 07 14:23:00 director.fv1dci.org heat-api[29433]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:01 director.fv1dci.org heat-api[29496]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:02 director.fv1dci.org heat-api[29547]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:04 director.fv1dci.org heat-api[29652]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:05 director.fv1dci.org heat-api[29724]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:06 director.fv1dci.org heat-api[29744]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:07 director.fv1dci.org heat-api[29774]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:23:09 director.fv1dci.org heat-api[29797]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:37 director.fv1dci.org heat-api[1529]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:38 director.fv1dci.org heat-api[2626]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:40 director.fv1dci.org heat-api[3001]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:41 director.fv1dci.org heat-api[3561]: ERROR: Unable to locate config file [api-paste.ini]
Jun 07 14:33:42 director.fv1dci.org heat-api[5190]: ERROR: Unable to locate config file [api-paste.ini]

Created attachment 1285913 [details]
sosreport from the undercloud node

We appear to be missing the following for the services:

https://github.com/rdo-packages/heat-distgit/commit/9c7d76cdcfa27335986235072ba622a173034452

starting the service by hand with these command line works, the one shipped lacks the dist.conf items so it cannot find the dist provided api-paste.ini

That explains why bug 1452677 showed up in OSP10.

Taking a further look, it's not that commit. It is that problem where the *-dist.conf files aren't being loaded but I'm not sure where that was accomplished previously.  I just spun up a new 9 undercloud and it's working and loading the heat-dist.conf but i'm not sure where that is being injected.

So i tracked this down to python-oslo-config. In the shipped versions there's a specific code to handle the -dist.config files in /usr/lib/python2.7/site-packages/oslo_config/cfg.py

    config_files = []
    if project:
        config_files.append(_search_dirs(['/usr/share/%s/' % project],
                                         project, '-dist%s' % extension))
        config_files.append(_search_dirs(['/usr/share/%s/' % project],
                                         prog, '-dist%s' % extension))
        config_files.append(_search_dirs(cfg_dirs, project, extension))
    config_files.append(_search_dirs(cfg_dirs, prog, extension))


This is missing from the file on the system and doing an rpm verification shows that the files are not the package provided files. Additionally pip info oslo.config shows vs 4.3.0 but the package version should be 3.9.0. Someone/something upgraded oslo.config using a non-packaged version.

[root@director cmd]# pip show oslo.config                                                                                                                                                     |
Name: oslo.config
Version: 4.3.0
Summary: Oslo Configuration API
Home-page: http://docs.openstack.org/developer/oslo.config/
Author: OpenStack
Author-email: openstack-dev.org
License: UNKNOWN
Location: /usr/lib/python2.7/site-packages
Requires: rfc3986, oslo.i18n, six, netaddr, debtcollector, stevedore

*** Bug 1459694 has been marked as a duplicate of this bug. ***

I tracked it down to the tempest-setup.sh script being used which does a sudo pip install tempestconfig. This caused a newer version of oslo.config to be installed which broke the services. 

From the tempest_output.log on the host...
2017-06-07 12:08:14 | + '[' -f sudo pip install tempestconf ']'
2017-06-07 12:08:14 | /home/stack/tempest-setup.sh: line 85: [: too many arguments
2017-06-07 12:08:14 | + sudo pip install tempestconf
2017-06-07 12:08:15 | Collecting tempestconf
2017-06-07 12:08:15 |   Downloading tempestconf-0.0.1.dev8.tar.gz
...snip...
2017-06-07 12:08:18 | Collecting oslo.config>=3.22.0 (from tempest>=14.0.0->tempestconf)
2017-06-07 12:08:18 |   Downloading oslo.config-4.3.0-py2.py3-none-any.whl (106kB)
...snip...

Moving this over to python-tempestconfig as in IRC this was mentioned as missing for this version of OSP9 which is why they were pip installing tempestconf.  The recommendation would be to either provide the appropriate tempest items in package form or use a virtualenv if needing to pip install.

python-tempestconf has been added in OSP11 release and is available from this realease onwards. For OSP9 tempest conf is provided by tempest rpm.