Bug 1506149 - [hwivBoNF] Be able to access the node through the egress IP after restart iptables service
Summary: [hwivBoNF] Be able to access the node through the egress IP after restart ipt...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.7.0
Assignee: Dan Winship
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-25 09:09 UTC by Meng Bo
Modified: 2017-11-28 22:19 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-11-28 22:19:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Meng Bo 2017-10-25 09:09:09 UTC
Description of problem:
After the egressIP be added to the node via hostsubnet, user can access node via the egressIP directly, which should be prohibited. 

Version-Release number of selected component (if applicable):
v3.7.0-0.178.0

How reproducible:
always

Steps to Reproduce:
1. Setup multi-node env with multitenant or networkpolicy plugin
2. Update the node's hostsubnet to add the egressIP
# oc patch hostsubnet ose-node1.bmeng.local -p '{"egressIPs":["10.66.140.100"]}'
3. Try to access the node via the egressIP

Actual results:
Can access the node through the egress IP.

Expected results:
Should not be able to access the egress IP.


Additional info:
There is no packet counted by the openflow when accessing the egress IP from the local network.

Here is the if info:
[root@ose-node1 ~]# ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:b9:4e:04 brd ff:ff:ff:ff:ff:ff
    inet 10.66.140.199/23 brd 10.66.141.255 scope global dynamic eth0
       valid_lft 63454sec preferred_lft 63454sec
    inet 10.66.140.100/23 brd 10.66.141.255 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feb9:4e04/64 scope link
       valid_lft forever preferred_lft forever

Comment 1 Meng Bo 2017-10-25 09:09:50 UTC
Here is the full dump of the openflow

# ovs-ofctl dump-flows br0 -O openflow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
 cookie=0x0, duration=1034.633s, table=0, n_packets=0, n_bytes=0, priority=400,ip,in_port=2,nw_src=10.128.0.1 actions=goto_table:30
 cookie=0x0, duration=1034.649s, table=0, n_packets=0, n_bytes=0, priority=300,ct_state=-trk,ip actions=ct(table=0)
 cookie=0x0, duration=1034.629s, table=0, n_packets=0, n_bytes=0, priority=300,ip,in_port=2,nw_src=10.128.0.0/23,nw_dst=10.128.0.0/14 actions=goto_table:25
 cookie=0x0, duration=1034.624s, table=0, n_packets=0, n_bytes=0, priority=250,ip,in_port=2,nw_dst=224.0.0.0/4 actions=drop
 cookie=0x0, duration=1034.646s, table=0, n_packets=0, n_bytes=0, priority=200,arp,in_port=1,arp_spa=10.128.0.0/14,arp_tpa=10.128.0.0/23 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
 cookie=0x0, duration=1034.641s, table=0, n_packets=0, n_bytes=0, priority=200,ip,in_port=1,nw_src=10.128.0.0/14 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
 cookie=0x0, duration=1034.621s, table=0, n_packets=0, n_bytes=0, priority=200,arp,in_port=2,arp_spa=10.128.0.1,arp_tpa=10.128.0.0/14 actions=goto_table:30
 cookie=0x0, duration=1034.617s, table=0, n_packets=0, n_bytes=0, priority=200,ip,in_port=2 actions=goto_table:30
 cookie=0x0, duration=1034.637s, table=0, n_packets=0, n_bytes=0, priority=150,in_port=1 actions=drop
 cookie=0x0, duration=1034.614s, table=0, n_packets=8, n_bytes=648, priority=150,in_port=2 actions=drop
 cookie=0x0, duration=1034.611s, table=0, n_packets=0, n_bytes=0, priority=100,arp actions=goto_table:20
 cookie=0x0, duration=1034.607s, table=0, n_packets=0, n_bytes=0, priority=100,ip actions=goto_table:20
 cookie=0x0, duration=1034.604s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.430s, table=10, n_packets=0, n_bytes=0, priority=100,tun_src=10.66.140.15 actions=goto_table:30
 cookie=0x0, duration=1034.601s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.598s, table=20, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.433s, table=21, n_packets=0, n_bytes=0, priority=200,ip,nw_dst=10.128.0.0/14 actions=ct(commit,table=30)
 cookie=0x0, duration=1034.595s, table=21, n_packets=0, n_bytes=0, priority=0 actions=goto_table:30
 cookie=0x0, duration=1034.591s, table=25, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.585s, table=30, n_packets=0, n_bytes=0, priority=300,arp,arp_tpa=10.128.0.1 actions=output:2
 cookie=0x0, duration=1034.574s, table=30, n_packets=0, n_bytes=0, priority=300,ip,nw_dst=10.128.0.1 actions=output:2
 cookie=0x0, duration=1034.560s, table=30, n_packets=0, n_bytes=0, priority=300,ct_state=+rpl,ip,nw_dst=10.128.0.0/23 actions=ct(table=70,nat)
 cookie=0x0, duration=1034.581s, table=30, n_packets=0, n_bytes=0, priority=200,arp,arp_tpa=10.128.0.0/23 actions=goto_table:40
 cookie=0x0, duration=1034.555s, table=30, n_packets=0, n_bytes=0, priority=200,ip,nw_dst=10.128.0.0/23 actions=goto_table:70
 cookie=0x0, duration=1034.578s, table=30, n_packets=0, n_bytes=0, priority=100,arp,arp_tpa=10.128.0.0/14 actions=goto_table:50
 cookie=0x0, duration=1034.543s, table=30, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.128.0.0/14 actions=goto_table:90
 cookie=0x0, duration=1034.570s, table=30, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=172.30.0.0/16 actions=goto_table:60
 cookie=0x0, duration=1034.530s, table=30, n_packets=0, n_bytes=0, priority=50,ip,in_port=1,nw_dst=224.0.0.0/4 actions=goto_table:120
 cookie=0x0, duration=1034.523s, table=30, n_packets=0, n_bytes=0, priority=25,ip,nw_dst=224.0.0.0/4 actions=goto_table:110
 cookie=0x0, duration=1034.518s, table=30, n_packets=0, n_bytes=0, priority=0,ip actions=goto_table:100
 cookie=0x0, duration=1034.514s, table=30, n_packets=0, n_bytes=0, priority=0,arp actions=drop
 cookie=0x0, duration=1034.510s, table=40, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.425s, table=50, n_packets=0, n_bytes=0, priority=100,arp,arp_tpa=10.129.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.140.15->tun_dst,output:1
 cookie=0x0, duration=1034.505s, table=50, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.501s, table=60, n_packets=0, n_bytes=0, priority=200 actions=output:2
 cookie=0x0, duration=1034.498s, table=60, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.495s, table=70, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.491s, table=80, n_packets=0, n_bytes=0, priority=300,ip,nw_src=10.128.0.1 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=1034.426s, table=80, n_packets=0, n_bytes=0, priority=200,ct_state=+rpl,ip actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=1034.486s, table=80, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.418s, table=90, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=10.129.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.140.15->tun_dst,output:1
 cookie=0x0, duration=1034.475s, table=90, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.469s, table=100, n_packets=0, n_bytes=0, priority=0 actions=goto_table:101
 cookie=0x0, duration=1034.466s, table=101, n_packets=0, n_bytes=0, priority=51,tcp,nw_dst=10.66.140.199,tp_dst=53 actions=output:2
 cookie=0x0, duration=1034.462s, table=101, n_packets=0, n_bytes=0, priority=51,udp,nw_dst=10.66.140.199,tp_dst=53 actions=output:2
 cookie=0x0, duration=1034.459s, table=101, n_packets=0, n_bytes=0, priority=0 actions=output:2
 cookie=0x0, duration=1034.456s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.415s, table=111, n_packets=0, n_bytes=0, priority=100 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.140.15->tun_dst,output:1,goto_table:120
 cookie=0x0, duration=1034.449s, table=120, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=1034.446s, table=253, n_packets=0, n_bytes=0, actions=note:02.05.00.00.00.00

Comment 2 Meng Bo 2017-10-25 10:28:09 UTC
After check the iptables, I found that the REJECT rule for the dest to egress IP is missing after restart iptables and cannot be recovered anymore.


Steps as below:
1. Add egress IP to node

2. Check the iptables
[root@ose-node1 ~]# iptables -nL OPENSHIFT-FIREWALL-ALLOW 
Chain OPENSHIFT-FIREWALL-ALLOW (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:4789 /* VXLAN incoming */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* from SDN to localhost */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* from docker to localhost */
REJECT     all  --  0.0.0.0/0            10.66.140.100        ctstate NEW reject-with icmp-port-unreachable

3. Restart the iptables service on node and restart atomic-openshift-node to recreate the iptables rules
[root@ose-node1 ~]# systemctl restart iptables
[root@ose-node1 ~]# systemctl restart atomic-openshift-node 

4. Check the iptables again
[root@ose-node1 ~]# iptables -nL OPENSHIFT-FIREWALL-ALLOW 
Chain OPENSHIFT-FIREWALL-ALLOW (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:4789 /* VXLAN incoming */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* from SDN to localhost */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* from docker to localhost */


The reject rule will be removed and will never be re-added.

Comment 3 Dan Winship 2017-10-26 15:26:14 UTC
https://github.com/openshift/origin/pull/17054

Comment 5 Meng Bo 2017-10-30 06:38:40 UTC
Tested on build v3.7.0-0.185.0, issue has been fixed.
The iptables rule to block the egressIP to be accessed will be recreated after the atomic-openshift-node restart.

Verify the bug.

Comment 8 errata-xmlrpc 2017-11-28 22:19:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.