Bug 1537780

Description of problem:

The pod is restarted and has a new pod IP. There is a long living stream of udp packets to the nodeport created for this this pod. The old conntrack table entry pointing to the old IP of the pod is never cleaned up. 

This pull was created for new services created for Node Port but I do not think it takes in account when the endpoint is deleted. 
https://github.com/kubernetes/kubernetes/pull/32561   

I believe we might need to add somthing here to clean up any entries for endpoint changes for pods with node ports and not just endpoint changes for services. 

https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/proxy/iptables/proxier.go#L964-L980


Version-Release number of selected component (if applicable):
 3.X  

How reproducible:
100%


Info:

App 11.11.2.139 
NodePort: 31803

UDP Traffic continuous test A: 22.22.100.123:5060 to 22.22.113.140:31803.  

Non continuous test B 22.22.113.58:2000 to 22.22.113.140:31803



# conntrack -L | grep 31803
Test B:
udp 17 46 src=22.22.113.58 dst=22.22.113.140 sport=2000 dport=31803 src=11.11.2.139 dst=11.11.2.1 sport=5060 dport=2000 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1

Test A: 
udp 17 177 src=22.22.100.123 dst=22.22.113.140 sport=5060 dport=31803 src=11.11.2.139 dst=11.11.2.1 sport=5060 dport=5060 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=2
conntrack v1.4.4 (conntrack-tools): 653 flow entries have been shown.


- The app is restarted and is now running on 11.11.2.142. 
- Test A has been sending UDP packets every 5 seconds and is getting no replies.  
- Conntrack shows a rule which show traffic from the client (22.22.100.123 ) should still being sent to 11.11.2.139.  

- Test B sourcing from 22.22.113.58:2000 and destined for 22.22.113.14:31803 works because the traffic was not continuous, the conntrack entry shows this traffic is now sent to 11.11.2.142. 

# conntrack -L | grep 31803
TEST B
udp 17 178 src=22.22.113.58 dst=22.22.113.140 sport=2000 dport=31803 src=11.11.2.142 dst=11.11.2.1 sport=5060 dport=2000 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1

TEST A
udp 17 177 src=22.22.100.123 dst=22.22.113.140 sport=5060 dport=31803 src=11.11.2.139 dst=11.11.2.1 sport=5060 dport=5060 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=2
conntrack v1.4.4 (conntrack-tools): 674 flow entries have been shown.



The conntrack entry for 22.22.100.123 is manually deleted and a new entry shows up using the correct destination IP address of 11.11.2.142. TEST A is now showing traffic.

# conntrack -D -s 22.22.100.123
udp 17 176 src=22.22.100.123 dst=22.22.113.140 sport=5060 dport=31803 src=11.11.2.139 dst=11.11.2.1 sport=5060 dport=5060 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been deleted.

# conntrack -L | grep 31803
dp 17 28 src=22.22.100.123 dst=22.22.113.140 sport=5060 dport=31803 src=11.11.2.142 dst=11.11.2.1 sport=5060 dport=5060 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 648 flow entries have been shown.