Bug 1783764

Summary: Unexpected iptables rules are saved to "/etc/sysconfig/iptables" on first master host
Product: OpenShift Container Platform Reporter: Daein Park <dapark>
Component: InstallerAssignee: Russell Teague <rteague>
Installer sub component: openshift-ansible QA Contact: Gaoyun Pei <gpei>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: tkimura
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: cockpit role is run after openshift_sdn role Consequence: During the cockpit role, iptables rules are saved which also save unnecessary sdn rules. Fix: Moved the cockpit role to run before openshift_sdn role Result: Unnecessary sdn rules are not saved during installation
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-19 19:53:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daein Park 2019-12-15 12:30:27 UTC
Description of problem:

Unexpected iptables rules are saved to "/etc/sysconfig/iptables" on only first master as follows.
Added iptables rules are related with SDN.

* first master
~~~
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [21:2268]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-NODEPORT-NON-LOCAL - [0:0]
:KUBE-SERVICES - [0:0]
:OPENSHIFT-ADMIN-OUTPUT-RULES - [0:0]
:OPENSHIFT-FIREWALL-ALLOW - [0:0]
:OPENSHIFT-FIREWALL-FORWARD - [0:0]
:OS_FIREWALL_ALLOW - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -m comment --comment "Ensure that non-local NodePort traffic can flow" -j KUBE-NODEPORT-NON-LOCAL
-A INPUT -m comment --comment "firewall overrides" -j OPENSHIFT-FIREWALL-ALLOW
-A INPUT -j KUBE-FIREWALL
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j OS_FIREWALL_ALLOW
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -i tun0 ! -o tun0 -m comment --comment "administrator overrides" -j OPENSHIFT-ADMIN-OUTPUT-RULES
-A FORWARD -m comment --comment "firewall overrides" -j OPENSHIFT-FIREWALL-FORWARD
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x1/0x1 -j ACCEPT
-A KUBE-FORWARD -s 10.128.0.0/14 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.128.0.0/14 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OPENSHIFT-FIREWALL-ALLOW -p udp -m udp --dport 4789 -m comment --comment "VXLAN incoming" -j ACCEPT
-A OPENSHIFT-FIREWALL-ALLOW -i tun0 -m comment --comment "from SDN to localhost" -j ACCEPT
-A OPENSHIFT-FIREWALL-ALLOW -i docker0 -m comment --comment "from docker to localhost" -j ACCEPT
-A OPENSHIFT-FIREWALL-FORWARD -s 10.128.0.0/14 -m comment --comment "attempted resend after connection close" -m conntrack --ctstate INVALID -j DROP
-A OPENSHIFT-FIREWALL-FORWARD -d 10.128.0.0/14 -m comment --comment "forward traffic from SDN" -j ACCEPT
-A OPENSHIFT-FIREWALL-FORWARD -s 10.128.0.0/14 -m comment --comment "forward traffic to SDN" -j ACCEPT
:
~~~

* Other master hosts
~~~
# Generated by iptables-save v1.4.21 on Sun May 12 21:15:13 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:OS_FIREWALL_ALLOW - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j OS_FIREWALL_ALLOW
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A DOCKER-ISOLATION -j RETURN
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 10010 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 10250 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 10256 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A OS_FIREWALL_ALLOW -p udp -m state --state NEW -m udp --dport 4789 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 9000:10000 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 1936 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 8444 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 8053 -j ACCEPT
-A OS_FIREWALL_ALLOW -p udp -m state --state NEW -m udp --dport 8053 -j ACCEPT
COMMIT
# Completed on Sun May 12 21:15:13 2019
# Generated by iptables-save v1.4.21 on Sun May 12 21:15:13 2019
*nat
:PREROUTING ACCEPT [1420:153726]
:INPUT ACCEPT [3:180]
:OUTPUT ACCEPT [9248:649431]
:POSTROUTING ACCEPT [9248:649431]
:DOCKER - [0:0]
:KUBE-MARK-MASQ - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Sun May 12 21:15:13 2019

~~~

Version-Release number of the following components:

rpm -q openshift-ansible
openshift-ansible-3.11.154-2.git.0.1640c49.el7.noarch

rpm -q ansible
ansible-2.6.14-1.el7ae.noarch

ansible --version
ansible 2.6.14
  config file = /usr/share/ansible/openshift-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

You can always reproduce to install openshift 3.11 initially.

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Some sdn iptables rules are saved to "/etc/sysconfig/iptables" on only first master host.

Expected results:

SDN rules are not saved to "/etc/sysconfig/iptables" like other master hosts.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

This issue cause is to run cockpit roles after running SDN roles as follows.
Since cockpit roles has configurations, such as adding port and save the rules to "/etc/sysconfig/iptables". 

* playbooks/openshift-master/private/additional_config.yml
~~~
  roles:
  # TODO: this is currently required in order to schedule pods onto the masters, but
  #   should be moved into components once nodes are using dynamic config
  - role: openshift_sdn
    when: openshift_use_openshift_sdn | default(True) | bool
  - role: openshift_project_request_template
    when: openshift_project_request_template_manage
  - role: openshift_examples
    when:
    - openshift_install_examples | default(true) | bool
  - role: openshift_cluster_autoscaler
    when: openshift_cluster_autoscaler_deploy | default(false) | bool
  - role: openshift_manageiq
    when: openshift_use_manageiq | default(true) | bool
  - role: cockpit
    when:
    - not openshift_is_atomic | bool
    - openshift_deployment_type == 'openshift-enterprise'
    - osm_use_cockpit is undefined or osm_use_cockpit | bool
    - (openshift_deployment_subtype | default('')) != 'registry'
:
~~~

Comment 4 Daein Park 2019-12-20 06:05:33 UTC
Hi team,

I've created PR here: https://github.com/openshift/openshift-ansible/pull/12052

Comment 7 Gaoyun Pei 2020-01-19 07:04:31 UTC
Could reproduce this issue with openshift-ansible-3.11.161-1.git.0.376158f.el7.noarch.rpm

When osm_use_cockpit=true, setup a 3-master ha cluster, check /etc/sysconfig/iptables on the first master

[root@ip-172-18-1-124 ~]# grep SDN /etc/sysconfig/iptables
-A OPENSHIFT-FIREWALL-ALLOW -i tun0 -m comment --comment "from SDN to localhost" -j ACCEPT
-A OPENSHIFT-FIREWALL-FORWARD -d 10.2.0.0/16 -m comment --comment "forward traffic from SDN" -j ACCEPT
-A OPENSHIFT-FIREWALL-FORWARD -s 10.2.0.0/16 -m comment --comment "forward traffic to SDN" -j ACCEPT

The other two masters doesn't have SDN related rules in /etc/sysconfig/iptables.


With openshift-ansible-3.11.165-1.git.0.2b41335.el7.noarch.rpm used, all three masters doesn't have such SDN rules saved in /etc/sysconfig/iptables.

Comment 9 errata-xmlrpc 2020-02-19 19:53:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0402