Description of problem: It seems that Openshift 3.x installation on OSP 9 is broken due to some firewalld issues. After creating 2 master nodes and 3 other nodes, the heat stack create failed. Here's what we see in /var/log/messages messages:Mar 8 10:56:21 localhost kdumpctl: cat: write error: Broken pipe messages:Mar 8 11:03:32 oss-ocp-openshift-node-r3t4l84s NetworkManager[454]: <warn> (6) failed to call dispatcher scripts: (dbus-glib-error-quark:16) Type of message, '(sa{sa{sv}}a{sv}a{sv}a{sv}a {sv}a{sv}a{sv}sa{sv}a{sv}b)', does not match expected type '(sa{sa{sv}}a{sv}a{sv}a{sv}a{sv}a{sv}a{sv}ssa{sv}a{sv}b)' messages:Mar 8 11:14:50 oss-ocp-openshift-node-r3t4l84s yum[20880]: Installed: 1:perl-Error-0.17020-2.el7.noarch messages:Mar 8 11:15:16 oss-ocp-openshift-node-r3t4l84s dbus[458]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.1" (uid=0 pid=454 comm="/usr/sbin/NetworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.fedoraproject.FirewallD1" (uid=0 pid=21171 comm="/usr/bin/ python -Es /usr/sbin/firewalld --nofork -") messages:Mar 8 11:15:16 oss-ocp-openshift-node-r3t4l84s dbus[458]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.1" (uid=0 pid=454 comm="/usr/sbin/NetworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.fedoraproject.FirewallD1" (uid=0 pid=21171 comm="/usr/bin/ python -Es /usr/sbin/firewalld --nofork -") messages:Mar 8 11:15:16 oss-ocp-openshift-node-r3t4l84s NetworkManager[454]: <warn> (eth1) firewall zone add/change failed [3]: (9) Rejected send message, 1 matched rules; type="method_call", sen der=":1.1" (uid=0 pid=454 comm="/usr/sbin/NetworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.f edoraproject.FirewallD1" (uid=0 pid=21171 comm="/usr/bin/python -Es /usr/sbin/firewalld --nofork -") messages:Mar 8 11:15:16 oss-ocp-openshift-node-r3t4l84s NetworkManager[454]: <warn> (eth0) firewall zone add/change failed [4]: (9) Rejected send message, 1 matched rules; type="method_call", sen der=":1.1" (uid=0 pid=454 comm="/usr/sbin/NetworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.f edoraproject.FirewallD1" (uid=0 pid=21171 comm="/usr/bin/python -Es /usr/sbin/firewalld --nofork -") messages:Mar 8 11:15:16 oss-ocp-openshift-node-r3t4l84s dbus-daemon: dbus[458]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.1" (uid=0 pid=454 comm="/usr/sbin/Ne tworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.fedoraproject.FirewallD1" (uid=0 pid=21171 co mm="/usr/bin/python -Es /usr/sbin/firewalld --nofork -") messages:Mar 8 11:15:16 oss-ocp-openshift-node-r3t4l84s dbus-daemon: dbus[458]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.1" (uid=0 pid=454 comm="/usr/sbin/Ne tworkManager --no-daemon ") interface="org.fedoraproject.FirewallD1.zone" member="changeZone" error name="(unset)" requested_reply="0" destination="org.fedoraproject.FirewallD1" (uid=0 pid=21171 co mm="/usr/bin/python -Es /usr/sbin/firewalld --nofork -") messages:Mar 8 11:15:24 oss-ocp-openshift-node-r3t4l84s NetworkManager[454]: <warn> (8) failed to call dispatcher scripts: (dbus-glib-error-quark:16) Type of message, '(sa{sa{sv}}a{sv}a{sv}a{sv}a {sv}a{sv}a{sv}sa{sv}a{sv}b)', does not match expected type '(sa{sa{sv}}a{sv}a{sv}a{sv}a{sv}a{sv}a{sv}ssa{sv}a{sv}b)' messages:Mar 8 11:16:04 oss-ocp-openshift-node-r3t4l84s dockerd-current: time="2017-03-08T11:16:04.308523453-05:00" level=error msg="libcontainerd: failed to receive event from containerd: rpc err or: code = 13 desc = transport is closing" On checking the firewalld status it looks like a firewalld issue. [root@oss-ocp-openshift-node-r3t4l84s log]# systemctl status firewalld -l ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2017-03-08 11:15:16 EST; 20min ago Docs: man:firewalld(1) Main PID: 21171 (firewalld) CGroup: /system.slice/firewalld.service └─21171 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' fa iled: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.30.12.0/24 ! -o docker0 -j MASQUER ADE' failed: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED, ESTABLISHED -j ACCEPT' failed: Mar 08 11:30:59 oss-ocp-openshift-node-r3t4l84s.manishaexample.com firewalld[21171]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed:
*** Bug 1430474 has been marked as a duplicate of this bug. ***
Thank you for the report. What are the specific versions of OCP, RHEL and the Heat templates? Are you using a git checkout or the packaged version? Would it be possible to attach the Heat environment file you used to deploy the openshift-on-openstack templates as well? Did you use flannel or openshift-sdn? I have not seen this before so anything you can give us about your environment and configuration will help us tracking it down.
Thanks for your response. I'll have a look shortly. I've marked your last comment private since some of the information in there might be sensitive.
This doesn't appear to be directly related to the Heat templates. I'll need more details about the heat/ansible errors. Could you please try running: openstack stack failures list Oss-ocp It should show the resources Heat failed to create as well as the reason for the failure. I believe there's going to be additional information/error during the Ansible run. You should be able to see the output of that by running this as root on the Bastion node: journalctl -u os-collect-config
The Heat templates have been deprecated. Any future automated deployments of OpenShift on OpenStack should use the code in openshift-ansible.