Description of problem:
It seems that Openshift 3.x installation on OSP 9 is broken due to some firewalld issues. After creating 2 master nodes and 3 other nodes, the  heat stack create failed.

Here's what we see in /var/log/messages 

messages:Mar  8 10:56:21 localhost kdumpctl: cat: write error: Broken pipe
messages:Mar  8 11:03:32 oss-ocp-openshift-node-r3t4l84s NetworkManager[454]: <warn>  (6) failed to call dispatcher scripts: (dbus-glib-error-quark:16) Type of message, '(sa{sa{sv}}a{sv}a{sv}a{sv}a
{sv}a{sv}a{sv}sa{sv}a{sv}b)', does not match expected type '(sa{sa{sv}}a{sv}a{sv}a{sv}a{sv}a{sv}a{sv}ssa{sv}a{sv}b)'
messages:Mar  8 11:14:50 oss-ocp-openshift-node-r3t4l84s yum[20880]: Installed: 1:perl-Error-0.17020-2.el7.noarch
messages:Mar  8 11:15:16 oss-ocp-openshift-node-r3t4l84s dbus[458]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.1" (uid=0 pid=454 comm="/usr/sbin/NetworkManager 
--no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.fedoraproject.FirewallD1" (uid=0 pid=21171 comm="/usr/bin/
python -Es /usr/sbin/firewalld --nofork -")
messages:Mar  8 11:15:16 oss-ocp-openshift-node-r3t4l84s dbus[458]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.1" (uid=0 pid=454 comm="/usr/sbin/NetworkManager 
--no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.fedoraproject.FirewallD1" (uid=0 pid=21171 comm="/usr/bin/
python -Es /usr/sbin/firewalld --nofork -")
messages:Mar  8 11:15:16 oss-ocp-openshift-node-r3t4l84s NetworkManager[454]: <warn>  (eth1) firewall zone add/change failed [3]: (9) Rejected send message, 1 matched rules; type="method_call", sen
der=":1.1" (uid=0 pid=454 comm="/usr/sbin/NetworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.f
edoraproject.FirewallD1" (uid=0 pid=21171 comm="/usr/bin/python -Es /usr/sbin/firewalld --nofork -")
messages:Mar  8 11:15:16 oss-ocp-openshift-node-r3t4l84s NetworkManager[454]: <warn>  (eth0) firewall zone add/change failed [4]: (9) Rejected send message, 1 matched rules; type="method_call", sen
der=":1.1" (uid=0 pid=454 comm="/usr/sbin/NetworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.f
edoraproject.FirewallD1" (uid=0 pid=21171 comm="/usr/bin/python -Es /usr/sbin/firewalld --nofork -")
messages:Mar  8 11:15:16 oss-ocp-openshift-node-r3t4l84s dbus-daemon: dbus[458]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.1" (uid=0 pid=454 comm="/usr/sbin/Ne
tworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.fedoraproject.FirewallD1" (uid=0 pid=21171 co
mm="/usr/bin/python -Es /usr/sbin/firewalld --nofork -")
messages:Mar  8 11:15:16 oss-ocp-openshift-node-r3t4l84s dbus-daemon: dbus[458]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.1" (uid=0 pid=454 comm="/usr/sbin/Ne
tworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.fedoraproject.FirewallD1" (uid=0 pid=21171 co
mm="/usr/bin/python -Es /usr/sbin/firewalld --nofork -")
messages:Mar  8 11:15:24 oss-ocp-openshift-node-r3t4l84s NetworkManager[454]: <warn>  (8) failed to call dispatcher scripts: (dbus-glib-error-quark:16) Type of message, '(sa{sa{sv}}a{sv}a{sv}a{sv}a
{sv}a{sv}a{sv}sa{sv}a{sv}b)', does not match expected type '(sa{sa{sv}}a{sv}a{sv}a{sv}a{sv}a{sv}a{sv}ssa{sv}a{sv}b)'
messages:Mar  8 11:16:04 oss-ocp-openshift-node-r3t4l84s dockerd-current: time="2017-03-08T11:16:04.308523453-05:00" level=error msg="libcontainerd: failed to receive event from containerd: rpc err
or: code = 13 desc = transport is closing"


On checking the firewalld status it looks like a firewalld issue.

[root@oss-ocp-openshift-node-r3t4l84s log]# systemctl status firewalld -l
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2017-03-08 11:15:16 EST; 20min ago
     Docs: man:firewalld(1)
 Main PID: 21171 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─21171 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' fa
iled:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst
 127.0.0.0/8' failed:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.30.12.0/24 ! -o docker0 -j MASQUER
ADE' failed:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,
ESTABLISHED -j ACCEPT' failed:
Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: