Description of problem:

When a service that only has one pod linked, has that pod removed. UDP connections using the service IP after the removal create a conntrack table entry that does not get cleaned up when the pod is added back to the cluster. 


Version-Release number of selected component (if applicable):
3.5

How reproducible:
100% 

Steps to Reproduce:


Service IP = 172.25.5.107 

UDPSEND is on IP 10.0.2.198
UDPSINK is on IP 10.0.2.197



UDPSINK is seeing hello world messages:

$ oc logs udpsink-1-q507v -f
hello world
hello world
hello world

# conntrack -L -d 172.25.5.107
udp      17 29 src=10.0.2.198 dst=172.25.5.107 sport=35238 dport=8000 [UNREPLIED] src=10.0.2.197 dst=10.0.2.1 sport=8000 dport=35238 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1


Delete the pod 

# oc delete po/udpsink-1-q507v
pod "udpsink-1-q507v" deleted

Make UDP connections 

The following is seen in the conntrack table: 

# conntrack -L -d 172.25.5.107
udp      17 29 src=10.0.2.198 dst=172.25.5.107 sport=35238 dport=8000 [UNREPLIED] src=172.25.5.107 dst=10.0.2.1 sport=8000 dport=35238 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=7

Bring the pod up again and we continue to fail to get the UDP connects that are sent

To correct the table entry needs to be deleted. 


# conntrack -D -d 172.25.5.107
udp      17 29 src=10.0.2.198 dst=172.25.5.107 sport=35238 dport=8000 [UNREPLIED] src=172.25.5.107 dst=10.0.2.1 sport=8000 dport=35238 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=9


After it is deleted everything now works. 

# conntrack -L -d 172.25.5.107
udp      17 29 src=10.0.2.198 dst=172.25.5.107 sport=35238 dport=8000 [UNREPLIED] src=10.0.3.187 dst=10.0.2.1 sport=8000 dport=35238 mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1
conntrack v1.4.4 (conntrack-tools): 1 flow entries have been shown.


https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/proxy/iptables/proxier.go