Bug 1273377

Summary:	Iptables can't create chains/rules if using pure iptables
Product:	OKD	Reporter:	Qixuan Wang <qixuan.wang>
Component:	Networking	Assignee:	Dan Winship <danw>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Meng Bo <bmeng>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.x	CC:	aos-bugs, eparis, kzhang, mmccomas, rkhan, yadu
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-05-12 17:12:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Qixuan Wang 2015-10-20 10:32:18 UTC

Description of problem:
Switch  from userspace kube-proxy to pure iptables and create pod/service, iptables can't create the corresponding chains/rules. This issue only exists on origin.

Version-Release number of selected component (if applicable):
openshift v1.0.6-748-gd9fde09
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4
etcd 2.1.2

How reproducible:
Always

Steps to Reproduce:
1. Switch to pure iptables
# kube-proxy --proxy-mode=iptables --master=http://127.0.0.1:8080 --v=3

2. Create pod and service
# oc create -f mysql.yaml
# oc create -f mysql-service.yaml

3. Check iptables
# iptables-save | grep mysql
# iptables -nL -t nat | grep mysql

Actual results:
1. [root@ip-172-18-4-41 amd64]# ./kube-proxy --proxy-mode=iptables --master=http://127.0.0.1:8080 --v=3
I1020 09:20:34.173062    1424 server.go:172] Running in resource-only container "/kube-proxy"
I1020 09:20:34.174368    1424 server.go:310] Flag proxy-mode allows iptables proxy
I1020 09:20:34.175294    1424 server.go:219] Using iptables Proxier.
I1020 09:20:34.176709    1424 server.go:227] Tearing down userspace rules. Errors here are acceptable.
E1020 09:20:34.193372    1424 event.go:207] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)
E1020 09:20:42.266128    1424 event.go:207] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)
E1020 09:20:52.266724    1424 event.go:207] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)
E1020 09:21:02.267230    1424 event.go:207] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)
I1020 09:21:04.192871    1424 proxier.go:429] Not syncing iptables until Services and Endpoints have been received from master
E1020 09:21:12.267773    1424 event.go:207] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: connection refused' (may retry after sleeping)



3. [root@ip-172-18-4-41 fedora]# iptables-save 
# Generated by iptables-save v1.4.21 on Tue Oct 20 09:22:40 2015
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [404:24409]
:POSTROUTING ACCEPT [404:24409]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
COMMIT
# Completed on Tue Oct 20 09:22:40 2015
# Generated by iptables-save v1.4.21 on Tue Oct 20 09:22:40 2015
*filter
:INPUT ACCEPT [117583:135316027]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [116793:23286619]
:DOCKER - [0:0]
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
COMMIT
# Completed on Tue Oct 20 09:22:40 2015


Expected results:
1. [root@ip-172-18-11-121 work]# kube-proxy --proxy-mode=iptables --master=http://127.0.0.1:8080 --v=3
I1020 09:53:30.770245   16925 server.go:175] Running in resource-only container "/kube-proxy"
I1020 09:53:30.775727   16925 server.go:313] Flag proxy-mode allows iptables proxy
I1020 09:53:30.776698   16925 server.go:222] Using iptables Proxier.
I1020 09:53:30.806962   16925 server.go:230] Tearing down userspace rules. Errors here are acceptable.
I1020 09:53:30.825659   16925 config.go:194] Calling handler.OnServiceUpdate()
I1020 09:53:30.825707   16925 proxier.go:294] Adding new service "default/kubernetes:https" at 10.0.0.1:443/TCP
I1020 09:53:30.825789   16925 proxier.go:429] Not syncing iptables until Services and Endpoints have been received from master
I1020 09:53:30.826967   16925 config.go:95] Calling handler.OnEndpointsUpdate()
I1020 09:53:30.827001   16925 proxier.go:352] Setting endpoints for "default/kubernetes:https" to [172.18.11.121:6443]
I1020 09:53:30.827055   16925 proxier.go:432] Syncing iptables rules
I1020 09:53:30.851469   16925 proxier.go:735] Syncing rules: *nat
:KUBE-SERVICES - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SEP-I4UBTXQ6KJ4UAWQL - [0:0]
-A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-I4UBTXQ6KJ4UAWQL
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -s 172.18.11.121/32 -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -m tcp -p tcp -j DNAT --to-destination 172.18.11.121:6443
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
COMMIT
I1020 09:54:00.823047   16925 proxier.go:432] Syncing iptables rules
I1020 09:54:00.830183   16925 proxier.go:735] Syncing rules: *nat
:KUBE-SERVICES - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SEP-I4UBTXQ6KJ4UAWQL - [0:0]
-A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-I4UBTXQ6KJ4UAWQL
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -s 172.18.11.121/32 -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -m tcp -p tcp -j DNAT --to-destination 172.18.11.121:6443
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
COMMIT
I1020 09:54:26.135337   16925 config.go:194] Calling handler.OnServiceUpdate()
I1020 09:54:26.135373   16925 proxier.go:294] Adding new service "default/mysql:" at 10.0.0.204:3306/TCP
I1020 09:54:26.135458   16925 proxier.go:432] Syncing iptables rules
I1020 09:54:26.142743   16925 proxier.go:735] Syncing rules: *nat
:KUBE-SERVICES - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SEP-I4UBTXQ6KJ4UAWQL - [0:0]
:KUBE-SVC-M7XME3WTB36R42AM - [0:0]
-A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-I4UBTXQ6KJ4UAWQL
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -s 172.18.11.121/32 -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -m tcp -p tcp -j DNAT --to-destination 172.18.11.121:6443
-A KUBE-SERVICES -m comment --comment "default/mysql: cluster IP" -m tcp -p tcp -d 10.0.0.204/32 --dport 3306 -j KUBE-SVC-M7XME3WTB36R42AM
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
COMMIT
I1020 09:54:27.230763   16925 config.go:95] Calling handler.OnEndpointsUpdate()
I1020 09:54:27.230802   16925 proxier.go:432] Syncing iptables rules
I1020 09:54:27.238058   16925 proxier.go:735] Syncing rules: *nat
:KUBE-SERVICES - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SEP-I4UBTXQ6KJ4UAWQL - [0:0]
:KUBE-SVC-M7XME3WTB36R42AM - [0:0]
-A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-I4UBTXQ6KJ4UAWQL
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -s 172.18.11.121/32 -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -m tcp -p tcp -j DNAT --to-destination 172.18.11.121:6443
-A KUBE-SERVICES -m comment --comment "default/mysql: cluster IP" -m tcp -p tcp -d 10.0.0.204/32 --dport 3306 -j KUBE-SVC-M7XME3WTB36R42AM
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
COMMIT
I1020 09:54:28.442878   16925 config.go:95] Calling handler.OnEndpointsUpdate()
I1020 09:54:28.442917   16925 proxier.go:352] Setting endpoints for "default/mysql:" to [172.17.0.10:3306]
I1020 09:54:28.442941   16925 proxier.go:432] Syncing iptables rules
I1020 09:54:28.450699   16925 proxier.go:735] Syncing rules: *nat
:KUBE-SERVICES - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SEP-I4UBTXQ6KJ4UAWQL - [0:0]
:KUBE-SVC-M7XME3WTB36R42AM - [0:0]
:KUBE-SEP-BM6UIWVLWUZJXFZG - [0:0]
-A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-I4UBTXQ6KJ4UAWQL
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -s 172.18.11.121/32 -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-I4UBTXQ6KJ4UAWQL -m comment --comment default/kubernetes:https -m tcp -p tcp -j DNAT --to-destination 172.18.11.121:6443
-A KUBE-SERVICES -m comment --comment "default/mysql: cluster IP" -m tcp -p tcp -d 10.0.0.204/32 --dport 3306 -j KUBE-SVC-M7XME3WTB36R42AM
-A KUBE-SVC-M7XME3WTB36R42AM -m comment --comment default/mysql: -j KUBE-SEP-BM6UIWVLWUZJXFZG
-A KUBE-SEP-BM6UIWVLWUZJXFZG -m comment --comment default/mysql: -s 172.17.0.10/32 -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-BM6UIWVLWUZJXFZG -m comment --comment default/mysql: -m tcp -p tcp -j DNAT --to-destination 172.17.0.10:3306
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
COMMIT


3. [root@ip-172-18-11-121 work]# iptables -nL -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4d415351

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination         

Chain KUBE-SEP-BM6UIWVLWUZJXFZG (1 references)
target     prot opt source               destination         
MARK       all  --  172.17.0.10          0.0.0.0/0            /* default/mysql: */ MARK set 0x4d415351
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/mysql: */ tcp to:172.17.0.10:3306

Chain KUBE-SEP-I4UBTXQ6KJ4UAWQL (1 references)
target     prot opt source               destination         
MARK       all  --  172.18.11.121        0.0.0.0/0            /* default/kubernetes:https */ MARK set 0x4d415351
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ tcp to:172.18.11.121:6443

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            10.0.0.1             /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-M7XME3WTB36R42AM  tcp  --  0.0.0.0/0            10.0.0.204           /* default/mysql: cluster IP */ tcp dpt:3306
KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-M7XME3WTB36R42AM (1 references)
target     prot opt source               destination         
KUBE-SEP-BM6UIWVLWUZJXFZG  all  --  0.0.0.0/0            0.0.0.0/0            /* default/mysql: */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
target     prot opt source               destination         
KUBE-SEP-I4UBTXQ6KJ4UAWQL  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */



Additional info:
# vi mysql.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mysql
  labels:
    name: mysql
spec:
  containers:
    - image: mysql
      name: mysql
      env:
        - name: MYSQL_ROOT_PASSWORD
          value: yourpassword
      ports:
        - containerPort: 3306
          name: mysql



# vi mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    name: mysql
  name: mysql
spec:
  ports:
    - port: 3306
  selector:
    name: mysql

Comment 1 Dan Winship 2015-10-21 17:21:41 UTC

Right. OpenShift reimplements the core of kube-proxy itself, and that code has not been updated to support iptables mode.

Do we even document the openshift-wrapper kube-proxy command anywhere? Normally you would not use this since the node runs it internally.

Filed https://trello.com/c/KdRjKrld

Comment 2 Dan Winship 2015-10-21 17:24:12 UTC

This is in the blocker bug report but I don't think it should be for 3.1. The iptables-based code has still not been thoroughly tested upstream.

Comment 4 Dan Winship 2015-12-17 13:04:07 UTC

origin master now uses the pure-iptables proxy

Comment 5 Yan Du 2015-12-18 09:42:43 UTC

Test on latest origin code, issue could not be reproduced. 

Move bug to verified.