Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1711605

Summary: Services should be rejected when no endpoint e2e test fails
Product: OpenShift Container Platform Reporter: Clayton Coleman <ccoleman>
Component: NetworkingAssignee: Ricardo Carrillo Cruz <ricarril>
Status: CLOSED DUPLICATE QA Contact: zhaozhanqi <zzhao>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-29 14:48:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Clayton Coleman 2019-05-18 23:22:55 UTC
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/22858/pull-ci-openshift-origin-master-e2e-aws/8784#openshift-tests-sig-network-services-should-be-rejected-when-no-endpoints-exist-suiteopenshiftconformanceparallel-suitek8s

fail [k8s.io/kubernetes/test/e2e/network/service.go:1840]: Unexpected error:
    <*errors.errorString | 0xc0002a1520>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred

Comment 4 Casey Callendrello 2019-05-20 16:17:23 UTC
Set up tracing on the node

[root@ip-10-0-155-248 log]# iptables-save -c | grep 172.30.112.100
[0:0] -A KUBE-SERVICES -d 172.30.112.100/32 -p tcp -m comment --comment "default/headless:hello1 has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable
[11:660] -A PREROUTING -d 172.30.112.100/32 -p tcp -j TRACE
[0:0] -A OUTPUT -d 172.30.112.100/32 -p tcp -j TRACE


And a trace of a single syn:

trace id 1b5f0f07 ip raw PREROUTING packet: iif "tun0" ether saddr 0a:58:0a:81:02:08 ether daddr d2:2f:8c:3b:f1:af ip saddr 10.129.2.8 ip daddr 172.30.112.100 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 38623 ip length 60 tcp sport 45994 tcp dport http tcp flags == syn tcp window 26733 
trace id 1b5f0f07 ip raw PREROUTING rule meta l4proto tcp ip daddr 172.30.112.100 counter packets 10 bytes 600 nftrace set 1 (verdict continue)
trace id 1b5f0f07 ip raw PREROUTING verdict continue 
trace id 1b5f0f07 ip raw PREROUTING policy accept 
trace id 1b5f0f07 ip nat PREROUTING packet: iif "tun0" ether saddr 0a:58:0a:81:02:08 ether daddr d2:2f:8c:3b:f1:af ip saddr 10.129.2.8 ip daddr 172.30.112.100 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 38623 ip length 60 tcp sport 45994 tcp dport http tcp flags == syn tcp window 26733 
trace id 1b5f0f07 ip nat PREROUTING rule  counter packets 53309 bytes 34643910 jump KUBE-SERVICES (verdict jump KUBE-SERVICES)
trace id 1b5f0f07 ip nat KUBE-SERVICES verdict continue 
trace id 1b5f0f07 ip nat PREROUTING rule  counter packets 39385 bytes 32788458 jump KUBE-PORTALS-CONTAINER (verdict jump KUBE-PORTALS-CONTAINER)
trace id 1b5f0f07 ip nat KUBE-PORTALS-CONTAINER verdict continue 
trace id 1b5f0f07 ip nat PREROUTING verdict continue 
trace id 1b5f0f07 ip nat PREROUTING policy accept 
trace id 1b5f0f07 ip filter FORWARD packet: iif "tun0" oif "tun0" ether saddr 0a:58:0a:81:02:08 ether daddr d2:2f:8c:3b:f1:af ip saddr 10.129.2.8 ip daddr 172.30.112.100 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 38623 ip length 60 tcp sport 45994 tcp dport http tcp flags == syn tcp window 26733 
trace id 1b5f0f07 ip filter FORWARD rule  counter packets 207427 bytes 700613558 jump KUBE-FORWARD (verdict jump KUBE-FORWARD)
trace id 1b5f0f07 ip filter KUBE-FORWARD verdict continue 
trace id 1b5f0f07 ip filter FORWARD rule  counter packets 207395 bytes 700609910 jump OPENSHIFT-FIREWALL-FORWARD (verdict jump OPENSHIFT-FIREWALL-FORWARD)
trace id 1b5f0f07 ip filter OPENSHIFT-FIREWALL-FORWARD rule ip saddr 10.128.0.0/14  counter packets 96019 bytes 19277735 accept (verdict accept)
trace id 1b5f0f07 ip nat POSTROUTING packet: oif "tun0" @ll,0,112 4263073140774338806410333311731712 ip saddr 10.129.2.8 ip daddr 172.30.112.100 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 38623 ip length 60 tcp sport 45994 tcp dport http tcp flags == syn tcp window 26733 
trace id 1b5f0f07 ip nat POSTROUTING rule  counter packets 58858 bytes 13201355 jump OPENSHIFT-MASQUERADE (verdict jump OPENSHIFT-MASQUERADE)
trace id 1b5f0f07 ip nat OPENSHIFT-MASQUERADE rule ip saddr 10.128.0.0/14  counter packets 16928 bytes 2059373 jump OPENSHIFT-MASQUERADE-2 (verdict jump OPENSHIFT-MASQUERADE-2)
trace id 1b5f0f07 ip nat OPENSHIFT-MASQUERADE-2 rule counter packets 923 bytes 79061 masquerade  (verdict accept)

Comment 5 Casey Callendrello 2019-05-20 16:40:15 UTC
I'm not the only one who thought this was a bug. Thockin fixed this in https://github.com/kubernetes/kubernetes/pull/72534, which is in v1.14.0.

Comment 6 Clayton Coleman 2019-05-20 16:48:29 UTC
Good backport candidate, but agree not blocking GA

Comment 7 Casey Callendrello 2019-07-15 20:19:03 UTC
So, the discussion of this has also moved to 1711538, where it was noticed that the upstream test that fixed this in 1.14 is still failing in OpenShift.

Ricardo has made some progress in fixing 1711438, so I'm assigning this to him. Ricky, when you like, you can close 1711538 as a dup of this bug.

Comment 8 Ricardo Carrillo Cruz 2019-07-29 14:48:39 UTC

*** This bug has been marked as a duplicate of bug 1711538 ***