Description of problem:
Using ansible to redeploy the certificates, the cluster ends up with iptables enabled and firewalld masked.
The playbook execution doesn't finish because of a communication error between node and master.
Master becomes unaccessible on port 8443

Version-Release number of selected component (if applicable):
openshift v3.4.1.2
kubernetes v1.4.0+776c994
openshift-ansible-3.4.56-1.git.0.7ba9968.el7.noarch

How reproducible: I tried twice


Steps to Reproduce:
ansible-playbook -i /etc/ansible/hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-certificates.yml --extra-vars "openshift_certificates_redeploy_ca=true"

Actual results:
iptables active
firewalld masked

TASK [restart node] ************************************************************
fatal: [node.example.com]: FAILED! => {
    "changed": false, 
    "failed": true
}

MSG:

Unable to restart service atomic-openshift-node: Job for atomic-openshift-node.service failed because the control process exited with error code. See "systemctl status atomic-openshift-node.service" and "journalctl -xe" for details.


Expected results:
iptables masked
firewalld active
# iptables -nvL | grep 8443
243 14580 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8443 ctstate NEW


Additional info:
Ansible output related to iptables changes

2017-02-09 12:48:24,404 p=11054 u=root |  TASK [os_firewall : Install firewalld packages] ********************************
2017-02-09 12:48:24,437 p=11054 u=root |  skipping: [master.example.com]
2017-02-09 12:48:24,454 p=11054 u=root |  TASK [os_firewall : Ensure iptables services are not enabled] ******************
2017-02-09 12:48:24,492 p=11054 u=root |  skipping: [master.example.com] => (item=iptables) 
2017-02-09 12:48:24,514 p=11054 u=root |  skipping: [master.example.com] => (item=ip6tables) 
2017-02-09 12:48:24,532 p=11054 u=root |  TASK [os_firewall : Start and enable firewalld service] ************************
2017-02-09 12:48:24,567 p=11054 u=root |  skipping: [master.example.com]
2017-02-09 12:48:24,585 p=11054 u=root |  TASK [os_firewall : need to pause here, otherwise the firewalld service starting can sometimes cause ssh to fail] ***
2017-02-09 12:48:24,616 p=11054 u=root |  skipping: [master.example.com]
2017-02-09 12:48:24,639 p=11054 u=root |  TASK [os_firewall : Add firewalld allow rules] *********************************
2017-02-09 12:48:24,697 p=11054 u=root |  TASK [os_firewall : Remove firewalld allow rules] ******************************
2017-02-09 12:48:24,748 p=11054 u=root |  TASK [os_firewall : Ensure firewalld service is not enabled] *******************
2017-02-09 12:48:25,701 p=11054 u=root |  changed: [master.example.com]
2017-02-09 12:48:25,714 p=11054 u=root |  TASK [os_firewall : Install iptables packages] *********************************
2017-02-09 12:48:26,757 p=11054 u=root |  ok: [master.example.com] => (item=[u'iptables', u'iptables-services'])
2017-02-09 12:48:26,774 p=11054 u=root |  TASK [os_firewall : Start and enable iptables service] *************************
2017-02-09 12:48:27,127 p=11054 u=root |  changed: [master.example.com]
2017-02-09 12:48:27,144 p=11054 u=root |  TASK [os_firewall : need to pause here, otherwise the iptables service starting can sometimes cause ssh to fail] ***
2017-02-09 12:48:27,201 p=11054 u=root |  Pausing for 10 seconds
2017-02-09 12:48:27,201 p=11054 u=root |  (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
2017-02-09 12:48:37,235 p=11054 u=root |  ok: [master.example.com]
2017-02-09 12:48:37,262 p=11054 u=root |  TASK [os_firewall : Add iptables allow rules] **********************************
2017-02-09 12:48:37,304 p=11054 u=root |  TASK [os_firewall : Remove iptables rules] *************************************
2017-02-09 12:48:37,343 p=11054 u=root |  TASK [docker : Get current installed Docker version] ***************************
2017-02-09 12:48:38,315 p=11054 u=root |  ok: [master.example.com]