Bug 1415800

Summary: Installer fails to add/check iptables rule due to lock on xtables.
Product: OpenShift Container Platform Reporter: Ryan Howe <rhowe>
Component: InstallerAssignee: Andrew Butcher <abutcher>
Status: CLOSED ERRATA QA Contact: Wenkai Shi <weshi>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: aos-bugs, jokerman, mmccomas, tdawson
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously the installer may have failed to add iptables rules if other iptables rules were being updated at the same time. Now the installer waits to obtain a lock when updating iptables rules ensuring that rules are properly created.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-12 18:49:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1445194    

Description Ryan Howe 2017-01-23 18:34:32 UTC
Description of problem:

When running the playbooks, the installer will fail during the os_firewall : Add iptables allow rules role/task due to a lock on xtables. 


Version-Release number of selected component (if applicable):
OpenShift 3.x

How reproducible:
Random 

Additional info:

Kubernetes hit this issue with OpenShift seen here in upstream issue:  https://github.com/kubernetes/kubernetes/issues/7370

The solution could be to add the -w flag to  wait for the xtables lock in the os_firewall_manage_iptables module.  

https://github.com/openshift/openshift-ansible/blob/release-1.4/roles/os_firewall/library/os_firewall_manage_iptables.py


######################
ERROR:
failed: [1_2_6_1] (item={u'port': u'10250/tcp', u'service': u'Kubernetes kubelet'}) => {
    "failed": true, 
    "invocation": {
        "module_args": {
            "action": "add", 
            "chain": "OS_FIREWALL_ALLOW", 
            "create_jump_rule": true, 
            "ip_version": "ipv4", 
            "jump_rule_chain": "INPUT", 
            "name": "Kubernetes kubelet", 
            "port": "10250", 
            "protocol": "tcp"
        }, 
        "module_name": "os_firewall_manage_iptables"
    }, 
    "item": {
        "port": "10250/tcp", 
        "service": "Kubernetes kubelet"
    }, 
    "msg": "Failed to create chain: OS_FIREWALL_ALLOW"
}

Checking if the rules generates a exit code not equal to 0 and then the module will try to create the chain or rule and then fail because it is already created. 

# iptables -C OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 10250  -j ACCEPT
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
# echo $?
4

# iptables -L OS_FIREWALL_ALLOW
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
# echo $?
4

# iptables -C INPUT  -j OS_FIREWALL_ALLOW 
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
# echo $?
4

Comment 1 openshift-github-bot 2017-01-24 14:52:46 UTC
Commit pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/182a07ad58972cddb3747a18d6a8e3818492488c
Merge pull request #3152 from abutcher/iptables-w

Bug 1415800 - Installer fails to add/check iptables rule due to lock on xtables.

Comment 2 Wenkai Shi 2017-02-03 10:18:37 UTC
Verified with version openshift-ansible-3.5.3-1.
Currently installer use firewalld modules rather than iptables to manage firewall by default, add "os_firewall_use_firewalld=false" to make sure installer use iptables. 
During installation, run a while loop of iptables command on hosts, installer will waiting at "TASK [os_firewall : Add iptables allow rules]". Once stop the while loop, installer continue.

[root@master ~]# while true; do iptables -nL; done
...

[root@ansible ~]# ansible-playbook -i hosts -v /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
...
TASK [os_firewall : Add iptables allow rules] **********************************
...
(waiting here during the "while loop" alive)
...

Comment 5 errata-xmlrpc 2017-04-12 18:49:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903