I'm not sure if this is still a problem - RHEL 7.3 was released recently.  Can you attach /var/log/audit/audit.log

Created attachment 1255868 [details]
heat wsgi audit log

Yea it still happens.

No AVCs in the log, but there might be an AVC elsewhere.  Try running the command below and see if that helps.

semanage port -N -m -t http_port_t -p tcp 8000

Adding the semanage call didn't seem to make a difference: http://logs.openstack.org/35/394835/17/check/gate-puppet-heat-puppet-beaker-rspec-centos-7/410cee6/console.html

2017-02-27 15:59:44.618220 |   1) basic heat default parameters should work with no errors
2017-02-27 15:59:44.618259 |      Failure/Error: apply_manifest(pp, :catch_failures => true)
2017-02-27 15:59:44.618288 |      Beaker::Host::CommandFailure:
2017-02-27 15:59:44.618322 |        Host 'centos-70-x64' exited with 1 running:
2017-02-27 15:59:44.618367 |         puppet apply --verbose --detailed-exitcodes /tmp/apply_manifest.pp.SDXV2l
2017-02-27 15:59:44.618398 |        Last 10 lines of output were:
2017-02-27 15:59:44.618447 |        	Feb 27 15:56:49 centos-7-infracloud-vanilla-7497790 systemd[1]: httpd.service failed.
2017-02-27 15:59:44.618470 |        	[0m
2017-02-27 15:59:44.618543 |        	[mNotice: /Stage[main]/Apache::Service/Service[httpd]: Triggered 'refresh' from 2 events[0m
2017-02-27 15:59:44.618613 |        	[mNotice: /Stage[main]/Keystone::Deps/Anchor[keystone::service::end]: Dependency Service[httpd] has failures: true[0m
2017-02-27 15:59:44.618676 |        	[1;33mWarning: /Stage[main]/Keystone::Deps/Anchor[keystone::service::end]: Skipping because of failed dependencies[0m
2017-02-27 15:59:44.618747 |        	[0;32mInfo: /Stage[main]/Keystone::Deps/Anchor[keystone::service::end]: Unscheduling all events on Anchor[keystone::service::end][0m
2017-02-27 15:59:44.618800 |        	[0;32mInfo: Class[Keystone::Deps]: Unscheduling all events on Class[Keystone::Deps][0m
2017-02-27 15:59:44.618853 |        	[0;32mInfo: Class[Apache::Service]: Unscheduling all events on Class[Apache::Service][0m
2017-02-27 15:59:44.618904 |        	[0;32mInfo: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml[0m
2017-02-27 15:59:44.619115 |        	[1;31mError: Failed to apply catalog: Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: Unable to establish connection to http://127.0.0.1:35357/v3/projects?: HTTPConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with url: /v3/projects (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3c12950>: Failed to establish a new connection: [Errno 111] Connection refused',)) (tried 34, for a total of 170 seconds)[0m
2017-02-27 15:59:44.619136 |        
2017-02-27 15:59:44.619179 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/host.rb:351:in `exec'
2017-02-27 15:59:44.619232 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/dsl/helpers/host_helpers.rb:83:in `block in on'
2017-02-27 15:59:44.619284 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/shared/host_manager.rb:127:in `run_block_on'
2017-02-27 15:59:44.619333 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/dsl/patterns.rb:37:in `block_on'
2017-02-27 15:59:44.619382 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/dsl/helpers/host_helpers.rb:63:in `on'
2017-02-27 15:59:44.619443 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/dsl/helpers/puppet_helpers.rb:480:in `block in apply_manifest_on'
2017-02-27 15:59:44.619495 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/shared/host_manager.rb:127:in `run_block_on'
2017-02-27 15:59:44.619563 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/dsl/patterns.rb:37:in `block_on'
2017-02-27 15:59:44.619622 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/dsl/helpers/puppet_helpers.rb:409:in `apply_manifest_on'
2017-02-27 15:59:44.619678 |      # ./.bundled_gems/gems/beaker-2.52.0/lib/beaker/dsl/helpers/puppet_helpers.rb:487:in `apply_manifest'
2017-02-27 15:59:44.619726 |      # ./spec/acceptance/basic_heat_spec.rb:70:in `block (3 levels) in <top (required)>'
2017-02-27 15:59:44.619743 | 
2017-02-27 15:59:44.619786 |   2) basic heat default parameters Port "8000" should be listening with tcp
2017-02-27 15:59:44.619826 |      Failure/Error: it { is_expected.to be_listening.with('tcp') }
2017-02-27 15:59:44.619861 |        expected Port "8000" to be listening with tcp
2017-02-27 15:59:44.619880 |        
2017-02-27 15:59:44.619928 |      # ./spec/acceptance/basic_heat_spec.rb:75:in `block (4 levels) in <top (required)>'
2017-02-27 15:59:44.619951 | 
2017-02-27 15:59:44.619996 |   3) basic heat default parameters Port "8003" should be listening with tcp
2017-02-27 15:59:44.620037 |      Failure/Error: it { is_expected.to be_listening.with('tcp') }
2017-02-27 15:59:44.620072 |        expected Port "8003" to be listening with tcp
2017-02-27 15:59:44.620091 |        
2017-02-27 15:59:44.620139 |      # ./spec/acceptance/basic_heat_spec.rb:79:in `block (4 levels) in <top (required)>'
2017-02-27 15:59:44.620156 | 
2017-02-27 15:59:44.620200 |   4) basic heat default parameters Port "8004" should be listening with tcp
2017-02-27 15:59:44.620241 |      Failure/Error: it { is_expected.to be_listening.with('tcp') }
2017-02-27 15:59:44.620277 |        expected Port "8004" to be listening with tcp


I'm a bit confused. Is it possible the failure to obtain the service catalog caused the rest of these failures here?

So when I looked at this previously the problem was that port 8000 overlapped with another thing in the selinux rules so httpd couldn't listen. I'm attempting to reproduce it locally again to get the specific details.  The failure in the beaker tests is because apache can't start at all when configured with port 8000. So the other ports (which do work) then fail because apache didn't start.

It overlapps with the soundd_port_t.

[root@centos-libvirt-7-x64 ~]# semanage port -l  | grep 8000
soundd_port_t                  tcp      8000, 9433, 16001
[root@centos-libvirt-7-x64 ~]# semanage port -a -t http_port_t -p tcp 8000
ValueError: Port tcp/8000 already define

What is soundd, it does not seem to exist on latest Fedora??

Looks like it was there since initial commit in https://github.com/TresysTechnology/refpolicy

The only option seems to be to move heat-api-cfn to some other port, soundd_port_t is defined in the base policy and cannot be touched e.g.
# semanage port -d -t soundd_port_t -p tcp 8000
ValueError: Port tcp/8000 is defined in policy, cannot be deleted

It looks like reloading the policy with semanage fixed our CI:

semanage port -m -t http_port_t -p tcp 8000

Can we get this added to the openstack-selinux policy so we don't have to hack in this semange via puppet?

would sure be nice to have it.

https://github.com/redhat-openstack/openstack-selinux/commit/ae36afd94c8d603481af4326b27bf16d9b906084

lon, don't we want a -N for no reload so that all the policy reloads at once when the image gets built?

It's been fixed there: https://github.com/redhat-openstack/openstack-selinux/commit/79832ec45fc56cd704d9d25280b31757be66fcbb

[stack@undercloud-0 ~]$ rpm -qa | grep openstack-selinux ; sudo semanage port -l | grep 8000
openstack-selinux-0.8.6-2.el7ost.noarch
http_port_t                    tcp      8000, 8000, 8000, 8000, 8000, 80, 81, 443, 488, 8008, 8009, 8443, 9000
soundd_port_t                  tcp      8000, 9433, 1600

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245