https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/22858/pull-ci-openshift-origin-master-e2e-aws/8784#openshift-tests-sig-network-services-should-be-rejected-when-no-endpoints-exist-suiteopenshiftconformanceparallel-suitek8s fail [k8s.io/kubernetes/test/e2e/network/service.go:1840]: Unexpected error: <*errors.errorString | 0xc0002a1520>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred
Set up tracing on the node [root@ip-10-0-155-248 log]# iptables-save -c | grep 172.30.112.100 [0:0] -A KUBE-SERVICES -d 172.30.112.100/32 -p tcp -m comment --comment "default/headless:hello1 has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable [11:660] -A PREROUTING -d 172.30.112.100/32 -p tcp -j TRACE [0:0] -A OUTPUT -d 172.30.112.100/32 -p tcp -j TRACE And a trace of a single syn: trace id 1b5f0f07 ip raw PREROUTING packet: iif "tun0" ether saddr 0a:58:0a:81:02:08 ether daddr d2:2f:8c:3b:f1:af ip saddr 10.129.2.8 ip daddr 172.30.112.100 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 38623 ip length 60 tcp sport 45994 tcp dport http tcp flags == syn tcp window 26733 trace id 1b5f0f07 ip raw PREROUTING rule meta l4proto tcp ip daddr 172.30.112.100 counter packets 10 bytes 600 nftrace set 1 (verdict continue) trace id 1b5f0f07 ip raw PREROUTING verdict continue trace id 1b5f0f07 ip raw PREROUTING policy accept trace id 1b5f0f07 ip nat PREROUTING packet: iif "tun0" ether saddr 0a:58:0a:81:02:08 ether daddr d2:2f:8c:3b:f1:af ip saddr 10.129.2.8 ip daddr 172.30.112.100 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 38623 ip length 60 tcp sport 45994 tcp dport http tcp flags == syn tcp window 26733 trace id 1b5f0f07 ip nat PREROUTING rule counter packets 53309 bytes 34643910 jump KUBE-SERVICES (verdict jump KUBE-SERVICES) trace id 1b5f0f07 ip nat KUBE-SERVICES verdict continue trace id 1b5f0f07 ip nat PREROUTING rule counter packets 39385 bytes 32788458 jump KUBE-PORTALS-CONTAINER (verdict jump KUBE-PORTALS-CONTAINER) trace id 1b5f0f07 ip nat KUBE-PORTALS-CONTAINER verdict continue trace id 1b5f0f07 ip nat PREROUTING verdict continue trace id 1b5f0f07 ip nat PREROUTING policy accept trace id 1b5f0f07 ip filter FORWARD packet: iif "tun0" oif "tun0" ether saddr 0a:58:0a:81:02:08 ether daddr d2:2f:8c:3b:f1:af ip saddr 10.129.2.8 ip daddr 172.30.112.100 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 38623 ip length 60 tcp sport 45994 tcp dport http tcp flags == syn tcp window 26733 trace id 1b5f0f07 ip filter FORWARD rule counter packets 207427 bytes 700613558 jump KUBE-FORWARD (verdict jump KUBE-FORWARD) trace id 1b5f0f07 ip filter KUBE-FORWARD verdict continue trace id 1b5f0f07 ip filter FORWARD rule counter packets 207395 bytes 700609910 jump OPENSHIFT-FIREWALL-FORWARD (verdict jump OPENSHIFT-FIREWALL-FORWARD) trace id 1b5f0f07 ip filter OPENSHIFT-FIREWALL-FORWARD rule ip saddr 10.128.0.0/14 counter packets 96019 bytes 19277735 accept (verdict accept) trace id 1b5f0f07 ip nat POSTROUTING packet: oif "tun0" @ll,0,112 4263073140774338806410333311731712 ip saddr 10.129.2.8 ip daddr 172.30.112.100 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 38623 ip length 60 tcp sport 45994 tcp dport http tcp flags == syn tcp window 26733 trace id 1b5f0f07 ip nat POSTROUTING rule counter packets 58858 bytes 13201355 jump OPENSHIFT-MASQUERADE (verdict jump OPENSHIFT-MASQUERADE) trace id 1b5f0f07 ip nat OPENSHIFT-MASQUERADE rule ip saddr 10.128.0.0/14 counter packets 16928 bytes 2059373 jump OPENSHIFT-MASQUERADE-2 (verdict jump OPENSHIFT-MASQUERADE-2) trace id 1b5f0f07 ip nat OPENSHIFT-MASQUERADE-2 rule counter packets 923 bytes 79061 masquerade (verdict accept)
I'm not the only one who thought this was a bug. Thockin fixed this in https://github.com/kubernetes/kubernetes/pull/72534, which is in v1.14.0.
Good backport candidate, but agree not blocking GA
So, the discussion of this has also moved to 1711538, where it was noticed that the upstream test that fixed this in 1.14 is still failing in OpenShift. Ricardo has made some progress in fixing 1711438, so I'm assigning this to him. Ricky, when you like, you can close 1711538 as a dup of this bug.
*** This bug has been marked as a duplicate of bug 1711538 ***