Bug 1415800 - Installer fails to add/check iptables rule due to lock on xtables.
Summary: Installer fails to add/check iptables rule due to lock on xtables.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Andrew Butcher
QA Contact: Wenkai Shi
URL:
Whiteboard:
Depends On:
Blocks: 1445194
TreeView+ depends on / blocked
 
Reported: 2017-01-23 18:34 UTC by Ryan Howe
Modified: 2017-07-24 14:11 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously the installer may have failed to add iptables rules if other iptables rules were being updated at the same time. Now the installer waits to obtain a lock when updating iptables rules ensuring that rules are properly created.
Clone Of:
Environment:
Last Closed: 2017-04-12 18:49:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0903 0 normal SHIPPED_LIVE OpenShift Container Platform atomic-openshift-utils bug fix and enhancement 2017-04-12 22:45:42 UTC

Description Ryan Howe 2017-01-23 18:34:32 UTC
Description of problem:

When running the playbooks, the installer will fail during the os_firewall : Add iptables allow rules role/task due to a lock on xtables. 


Version-Release number of selected component (if applicable):
OpenShift 3.x

How reproducible:
Random 

Additional info:

Kubernetes hit this issue with OpenShift seen here in upstream issue:  https://github.com/kubernetes/kubernetes/issues/7370

The solution could be to add the -w flag to  wait for the xtables lock in the os_firewall_manage_iptables module.  

https://github.com/openshift/openshift-ansible/blob/release-1.4/roles/os_firewall/library/os_firewall_manage_iptables.py


######################
ERROR:
failed: [1_2_6_1] (item={u'port': u'10250/tcp', u'service': u'Kubernetes kubelet'}) => {
    "failed": true, 
    "invocation": {
        "module_args": {
            "action": "add", 
            "chain": "OS_FIREWALL_ALLOW", 
            "create_jump_rule": true, 
            "ip_version": "ipv4", 
            "jump_rule_chain": "INPUT", 
            "name": "Kubernetes kubelet", 
            "port": "10250", 
            "protocol": "tcp"
        }, 
        "module_name": "os_firewall_manage_iptables"
    }, 
    "item": {
        "port": "10250/tcp", 
        "service": "Kubernetes kubelet"
    }, 
    "msg": "Failed to create chain: OS_FIREWALL_ALLOW"
}

Checking if the rules generates a exit code not equal to 0 and then the module will try to create the chain or rule and then fail because it is already created. 

# iptables -C OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 10250  -j ACCEPT
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
# echo $?
4

# iptables -L OS_FIREWALL_ALLOW
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
# echo $?
4

# iptables -C INPUT  -j OS_FIREWALL_ALLOW 
Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
# echo $?
4

Comment 1 openshift-github-bot 2017-01-24 14:52:46 UTC
Commit pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/182a07ad58972cddb3747a18d6a8e3818492488c
Merge pull request #3152 from abutcher/iptables-w

Bug 1415800 - Installer fails to add/check iptables rule due to lock on xtables.

Comment 2 Wenkai Shi 2017-02-03 10:18:37 UTC
Verified with version openshift-ansible-3.5.3-1.
Currently installer use firewalld modules rather than iptables to manage firewall by default, add "os_firewall_use_firewalld=false" to make sure installer use iptables. 
During installation, run a while loop of iptables command on hosts, installer will waiting at "TASK [os_firewall : Add iptables allow rules]". Once stop the while loop, installer continue.

[root@master ~]# while true; do iptables -nL; done
...

[root@ansible ~]# ansible-playbook -i hosts -v /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
...
TASK [os_firewall : Add iptables allow rules] **********************************
...
(waiting here during the "while loop" alive)
...

Comment 5 errata-xmlrpc 2017-04-12 18:49:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903


Note You need to log in before you can comment on or make changes to this bug.