Bug 1273659

Summary: FORWARD -j REJECT --reject-with icmp-host-prohibited rule results in being "trapped" in SDN
Product: OpenShift Container Platform Reporter: Erik M Jacobs <ejacobs>
Component: NetworkingAssignee: Dan Winship <danw>
Status: CLOSED CURRENTRELEASE QA Contact: Meng Bo <bmeng>
Severity: high Docs Contact:
Priority: high    
Version: 3.1.0CC: akostadi, aos-bugs, danw, jialiu, jokerman, kzhang, mmccomas, rkhan, sdodson, xiama
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: atomic-openshift-3.0.2.903-0.git.73.16a78c7.el7aos Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-23 14:24:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Erik M Jacobs 2015-10-20 22:17:41 UTC 
This rule exists after running an installation. However, it prevents traffic from leaving the SDN.

SDN ping:
nsenter -n -t 4817 -- ping 10.1.0.1
PING 10.1.0.1 (10.1.0.1) 56(84) bytes of data.
64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=0.536 ms

non-SDN ping with rule:
[root@ose3-node1 ~]# nsenter -n -t 4817 -- ping -c1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 10.1.1.1 icmp_seq=1 Destination Host Prohibited


Remove rule and ping works:
[root@ose3-node1 ~]# iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@ose3-node1 ~]# 
[root@ose3-node1 ~]# nsenter -n -t 4817 -- ping -c1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=45 time=88.3 ms
Comment 1 Dan Winship 2015-10-20 22:24:45 UTC 
Actually, now that I think about it, this probably ought to be fixed in openshift-sdn (and, unlike the port 4789 problem, it actually can be fixed there).
Comment 3 Johnny Liu 2015-10-27 10:37:02 UTC 
Seem like this bug and BZ#1273294 have the same cause root.
Comment 4 Dan Winship 2015-10-27 18:30:31 UTC 
should be fixed in origin master
Comment 5 Scott Dodson 2015-10-27 19:15:06 UTC 
*** Bug 1273294 has been marked as a duplicate of this bug. ***
Comment 7 Ma xiaoqiang 2015-10-28 02:23:40 UTC 
Check on puddle on [3.1/2015-10.27.1]

The iptable rules have been added. User can deploy pod successfully. Can access www.baidu.com in the pod
Comment 8 Erik M Jacobs 2015-11-04 17:58:05 UTC 
Doesn't appear to be working:

atomic-openshift-3.0.2.905-0.git.0.85d6f88.el7aos.x86_64
atomic-openshift-clients-3.0.2.905-0.git.0.85d6f88.el7aos.x86_64
atomic-openshift-master-3.0.2.905-0.git.0.85d6f88.el7aos.x86_64
atomic-openshift-node-3.0.2.905-0.git.0.85d6f88.el7aos.x86_64
atomic-openshift-sdn-ovs-3.0.2.905-0.git.0.85d6f88.el7aos.x86_64
atomic-openshift-utils-3.0.6-1.git.36.3d3f287.el7aos.noarch
openshift-ansible-3.0.6-1.git.36.3d3f287.el7aos.noarch
openshift-ansible-filter-plugins-3.0.6-1.git.36.3d3f287.el7aos.noarch
openshift-ansible-lookup-plugins-3.0.6-1.git.36.3d3f287.el7aos.noarch
openshift-ansible-playbooks-3.0.6-1.git.36.3d3f287.el7aos.noarch
openshift-ansible-roles-3.0.6-1.git.36.3d3f287.el7aos.noarch
tuned-profiles-atomic-openshift-node-3.0.2.905-0.git.0.85d6f88.el7aos.x86_64


Post-installation with the quick installer, I still have:

[root@ose3-master ~]# iptables-save | grep -i reject
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
Comment 9 Dan Winship 2015-11-04 18:14:06 UTC 
Right, the fix was to add new rules to unfirewall specific traffic:

  -A FORWARD -d ${CLUSTER_NETWORK} -j ACCEPT
  -A FORWARD -s ${CLUSTER_NETWORK} -j ACCEPT

(Previously those rules only got installed if firewalld was running. Now they're always installed.)
Comment 10 Erik M Jacobs 2015-11-05 01:38:30 UTC 
NVM this seems to be working. Sorry for the noise -- I didn't realize the resolution wasn't to remove that rule!
Comment 11 Brenton Leanhardt 2015-11-23 14:24:52 UTC 
This fix is available in OpenShift Enterprise 3.1.