2078691 – [OVN] Node to service traffic is blocked if service is "internalTrafficPolicy: Local" even backed pod is on the same node

Bug 2078691 - [OVN] Node to service traffic is blocked if service is "internalTrafficPolicy: Local" even backed pod is on the same node

Summary: [OVN] Node to service traffic is blocked if service is "internalTrafficPolicy...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.11
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	4.12.0
Assignee:	Surya Seetharaman
QA Contact:	huirwang
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	2082663 (view as bug list)
Depends On:	2082663
Blocks:	2105444
TreeView+	depends on / blocked

Reported:	2022-04-26 03:07 UTC by huirwang
Modified:	2023-01-17 19:48 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Clones:	2105444 (view as bug list)
Environment:
Last Closed:	2023-01-17 19:48:11 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	openshift ovn-kubernetes pull 1210	None	open	Bug 2078691: [Downstream Merge] 22-07-2022	2022-07-22 10:37:06 UTC
Github	ovn-org ovn-kubernetes pull 3006	None	Merged	Change rp_filter value for ovn-k8s-mp0	2022-07-20 21:15:37 UTC
Red Hat Product Errata	RHSA-2022:7399	None	None	None	2023-01-17 19:48:35 UTC

Description huirwang 2022-04-26 03:07:38 UTC

Description of problem:
Node to service traffic is blocked if service is "internalTrafficPolicy: Local" even backed pod is on the same node

Version-Release number of selected component (if applicable):
 4.11.0-0.nightly-2022-04-25-171513

How reproducible:
Always

Steps to Reproduce:
1. Create a service and configure "internalTrafficPolicy: Local" 
oc get svc -n test3 -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: "2022-04-26T02:31:09Z"
    labels:
      name: test-service
    name: test-service
    namespace: test3
    resourceVersion: "51643"
    uid: cf919da5-1c1f-47e7-8001-3297f0dee76d
  spec:
    clusterIP: 172.30.144.104
    clusterIPs:
    - 172.30.144.104
    internalTrafficPolicy: Local
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: http
      port: 27017
      protocol: TCP
      targetPort: 8080
    selector:
      name: test-pods
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

oc get svc -n test3
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
test-service   ClusterIP   172.30.144.104   <none>        27017/TCP   32m
$ oc get pods -n test3 -o wide
NAME            READY   STATUS    RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
hello-pod       1/1     Running   0          26m   10.128.2.31   ip-10-0-213-127.us-east-2.compute.internal   <none>           <none>
test-rc-rg45r   1/1     Running   0          32m   10.131.0.20   ip-10-0-145-176.us-east-2.compute.internal   <none>           <none>
$ oc get ep -n test3
NAME           ENDPOINTS          AGE
test-service   10.131.0.20:8080   32m

2. From node ip-10-0-145-176.us-east-2.compute.internal to access the service

3.

Actual results:
The traffic from node ip-10-0-145-176.us-east-2.compute.internal to service was blocked.

$oc debug node/ip-10-0-145-176.us-east-2.compute.internal
Starting pod/ip-10-0-145-176us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.145.176
If you don't see a command prompt, try pressing enter.
sh-4.4# curl 172.30.144.104:27017 --connect-timeout 5
curl: (28) Connection timed out after 5000 milliseconds

Expected results:
Should be able to access service from ip-10-0-145-176.us-east-2.compute.internal  as the endpoint pod is located on the same node.

Additional info:
The OUTPUT chain on node ip-10-0-145-176.us-east-2.compute.internal
sh-4.4#  iptables -S OUTPUT
-P OUTPUT ACCEPT
-A OUTPUT -p tcp -m tcp --dport 22624 --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -p tcp -m tcp --dport 22623 --tcp-flags FIN,SYN,RST,ACK SYN -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j KUBE-FIREWALL

Comment 2 Surya Seetharaman 2022-05-03 12:59:12 UTC

hmm on first screen I'm not sure what's wrong here:

I am able to reproduce the bug.

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:OVN-KUBE-ITP - [0:0]
[339157:263486942] -A OUTPUT -j OVN-KUBE-ITP
[4:240] -A OVN-KUBE-ITP -d 172.30.198.99/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1745ec/0xffffffff

The correct IPTable rule is being hit

We also have the correct routes setup:
sh-4.4# ip rule ls
0:	from all lookup local
30:	from all fwmark 0x1745ec lookup 7
32766:	from all lookup main
32767:	from all lookup default

sh-4.4# ip r show table 7
172.30.0.0/16 via 10.129.2.1 dev ovn-k8s-mp0 
sh-4.4# ip r
default via 10.0.128.1 dev br-ex proto dhcp metric 48 
10.0.128.0/18 dev br-ex proto kernel scope link src 10.0.184.157 metric 48 
10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0 
10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2 
169.254.169.0/30 via 10.0.128.1 dev br-ex 
169.254.169.3 via 10.129.2.1 dev ovn-k8s-mp0 
172.30.0.0/16 via 10.0.128.1 dev br-ex mtu 8901 

I am able to see packet entering ovn-k8s-mp0 and response is coming out:

sh-4.4# tcpdump -i ovn-k8s-mp0 -neep | grep 172.30.198.99
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ovn-k8s-mp0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:51:21.926439 fa:83:d1:e3:8b:60 > 0a:58:0a:81:02:01, ethertype IPv4 (0x0800), length 74: 10.129.2.2.58502 > 172.30.198.99.http: Flags [S], seq 1530622593, win 26583, options [mss 8861,sackOK,TS val 3987064920 ecr 0,nop,wscale 7], length 0
12:51:21.927243 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823408405 ecr 3987064920,nop,wscale 7], length 0
12:51:22.990351 fa:83:d1:e3:8b:60 > 0a:58:0a:81:02:01, ethertype IPv4 (0x0800), length 74: 10.129.2.2.58502 > 172.30.198.99.http: Flags [S], seq 1530622593, win 26583, options [mss 8861,sackOK,TS val 3987065984 ecr 0,nop,wscale 7], length 0
12:51:22.990336 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823409469 ecr 3987064920,nop,wscale 7], length 0
12:51:22.990829 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823409469 ecr 3987064920,nop,wscale 7], length 0
12:51:25.038346 fa:83:d1:e3:8b:60 > 0a:58:0a:81:02:01, ethertype IPv4 (0x0800), length 74: 10.129.2.2.58502 > 172.30.198.99.http: Flags [S], seq 1530622593, win 26583, options [mss 8861,sackOK,TS val 3987068032 ecr 0,nop,wscale 7], length 0
12:51:25.038335 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823411517 ecr 3987064920,nop,wscale 7], length 0
12:51:25.038407 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823411517 ecr 3987064920,nop,wscale 7], length 0

Definitely the backend pod is responding back:

12:54:18.251169 0a:58:0a:81:02:01 > 0a:58:0a:81:02:0a, ethertype IPv4 (0x0800), length 74: 10.129.2.2.59580 > 10.129.2.10.webcache: Flags [S], seq 2794852057, win 26583, options [mss 8861,sackOK,TS val 3987241244 ecr 0,nop,wscale 7], length 0
12:54:18.251199 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 10.129.2.10.webcache > 10.129.2.2.59580: Flags [S.], seq 2270704619, ack 2794852058, win 26547, options [mss 8861,sackOK,TS val 2823584729 ecr 3987241244,nop,wscale 7], length 0
12:54:19.310336 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 10.129.2.10.webcache > 10.129.2.2.59580: Flags [S.], seq 2270704619, ack 2794852058, win 26547, options [mss 8861,sackOK,TS val 2823585789 ecr 3987241244,nop,wscale 7], length 0
12:54:19.310980 0a:58:0a:81:02:01 > 0a:58:0a:81:02:0a, ethertype IPv4 (0x0800), length 74: 10.129.2.2.59580 > 10.129.2.10.webcache: Flags [S], seq 2794852057, win 26583, options [mss 8861,sackOK,TS val 3987242304 ecr 0,nop,wscale 7], length 0
12:54:19.311003 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 10.129.2.10.webcache > 10.129.2.2.59580: Flags [S.], seq 2270704619, ack 2794852058, win 26547, options [mss 8861,sackOK,TS val 2823585789 ecr 3987241244,nop,wscale 7], length 0
12:54:21.358339 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 10.129.2.10.webcache > 10.129.2.2.59580: Flags [S.], seq 2270704619, ack 2794852058, win 26547, options [mss 8861,sackOK,TS val 2823587837 ecr 3987241244,nop,wscale 7], length 0
12:54:21.358373 0a:58:0a:81:02:01 > 0a:58:0a:81:02:0a, ethertype IPv4 (0x0800), length 74: 10.129.2.2.59580 > 10.129.2.10.webcache: Flags [S], seq 2794852057, win 26583, options [mss 8861,sackOK,TS val 3987244352 ecr 0,nop,wscale 7], length 0


but the destination is 10.129.2.2 and not the hostIP...so the packet returnes safely? I am not sure why we are doing an SNAT downstream at OVN-KUBE-SNAT-MGMTPORT while I didn't have this problem on a KIND cluster.


note-to-self: BTW I think I found another bug here:

[9236:554160] -A OVN-KUBE-SNAT-MGMTPORT -o ovn-k8s-mp0 -m comment --comment "OVN SNAT to Management Port" -j SNAT --to-source 10.129.2.2
[0:0] -A OVN-KUBE-SNAT-MGMTPORT -p tcp -m tcp --dport 30348 -j RETURN
[0:0] -A OVN-KUBE-SNAT-MGMTPORT -p tcp -m tcp --dport 30212 -j RETURN

the order.... is looking wrong for sure. Need to change this.

Comment 3 Surya Seetharaman 2022-05-03 18:28:18 UTC

hmm all the right iptable rules are being hit

PACKET: 2 70c323cb OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN 
 TRACE: 2 70c323cb raw:OUTPUT:rule:0x8:CONTINUE  -4 -t raw -A OUTPUT -d 172.30.92.176/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 1/sec -j TRACE
 TRACE: 2 70c323cb raw:OUTPUT:return:
 TRACE: 2 70c323cb raw:OUTPUT:policy:ACCEPT 
PACKET: 2 70c323cb OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN 
 TRACE: 2 70c323cb mangle:OUTPUT:rule:0x8:JUMP:OVN-KUBE-ITP  -4 -t mangle -A OUTPUT -j OVN-KUBE-ITP
 TRACE: 2 70c323cb mangle:OVN-KUBE-ITP:rule:0x9:CONTINUE  -4 -t mangle -A OVN-KUBE-ITP -d 172.30.92.176/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1745ec/0xffffffff
 TRACE: 2 70c323cb mangle:OVN-KUBE-ITP:return:
 TRACE: 2 70c323cb mangle:OUTPUT:return:
 TRACE: 2 70c323cb mangle:OUTPUT:policy:ACCEPT 
PACKET: 2 70c323cb OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN MARK=0x1745ec 
 TRACE: 2 70c323cb nat:OUTPUT:rule:0x1c:JUMP:OVN-KUBE-EXTERNALIP  -4 -t nat -A OUTPUT -j OVN-KUBE-EXTERNALIP
 TRACE: 2 70c323cb nat:OVN-KUBE-EXTERNALIP:return:
 TRACE: 2 70c323cb nat:OUTPUT:rule:0x1a:JUMP:OVN-KUBE-NODEPORT  -4 -t nat -A OUTPUT -j OVN-KUBE-NODEPORT
 TRACE: 2 70c323cb nat:OVN-KUBE-NODEPORT:return:
 TRACE: 2 70c323cb nat:OUTPUT:rule:0x18:JUMP:OVN-KUBE-ITP  -4 -t nat -A OUTPUT -j OVN-KUBE-ITP
 TRACE: 2 70c323cb nat:OVN-KUBE-ITP:return:
 TRACE: 2 70c323cb nat:OUTPUT:return:
 TRACE: 2 70c323cb nat:OUTPUT:policy:ACCEPT 
PACKET: 2 70c323cb OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN MARK=0x1745ec 
 TRACE: 2 70c323cb filter:OUTPUT:rule:0x7:JUMP:KUBE-FIREWALL  -4 -t filter -A OUTPUT -j KUBE-FIREWALL
 TRACE: 2 70c323cb filter:KUBE-FIREWALL:return:
 TRACE: 2 70c323cb filter:OUTPUT:return:
 TRACE: 2 70c323cb filter:OUTPUT:policy:ACCEPT 
 TRACE: 2 70c323cb mangle:POSTROUTING:return:
 TRACE: 2 70c323cb mangle:POSTROUTING:policy:ACCEPT 
PACKET: 2 70c323cb OUT=ovn-k8s-mp0 SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN MARK=0x1745ec 
 TRACE: 2 70c323cb nat:POSTROUTING:rule:0x12:JUMP:OVN-KUBE-SNAT-MGMTPORT  -4 -t nat -A POSTROUTING -o ovn-k8s-mp0 -j OVN-KUBE-SNAT-MGMTPORT
 TRACE: 2 70c323cb nat:OVN-KUBE-SNAT-MGMTPORT:rule:0x2a:ACCEPT  -4 -t nat -A OVN-KUBE-SNAT-MGMTPORT -o ovn-k8s-mp0 -m comment --comment "OVN SNAT to Management Port" -j SNAT --to-source 10.131.0.2
PACKET: 2 a727a6df OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40858DF SPORT=58420 DPORT=80 SYN 
 TRACE: 2 a727a6df raw:OUTPUT:rule:0x8:CONTINUE  -4 -t raw -A OUTPUT -d 172.30.92.176/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 1/sec -j TRACE
 TRACE: 2 a727a6df raw:OUTPUT:return:
 TRACE: 2 a727a6df raw:OUTPUT:policy:ACCEPT 
PACKET: 2 a727a6df OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40858DF SPORT=58420 DPORT=80 SYN 
 TRACE: 2 a727a6df mangle:OUTPUT:rule:0x8:JUMP:OVN-KUBE-ITP  -4 -t mangle -A OUTPUT -j OVN-KUBE-ITP
 TRACE: 2 a727a6df mangle:OVN-KUBE-ITP:rule:0x9:CONTINUE  -4 -t mangle -A OVN-KUBE-ITP -d 172.30.92.176/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1745ec/0xffffffff
 TRACE: 2 a727a6df mangle:OVN-KUBE-ITP:return:
 TRACE: 2 a727a6df mangle:OUTPUT:return:
 TRACE: 2 a727a6df mangle:OUTPUT:policy:ACCEPT 
PACKET: 2 a727a6df OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40858DF SPORT=58420 DPORT=80 SYN MARK=0x1745ec 
 TRACE: 2 a727a6df filter:OUTPUT:rule:0x7:JUMP:KUBE-FIREWALL  -4 -t filter -A OUTPUT -j KUBE-FIREWALL
 TRACE: 2 a727a6df filter:KUBE-FIREWALL:return:
 TRACE: 2 a727a6df filter:OUTPUT:return:
 TRACE: 2 a727a6df filter:OUTPUT:policy:ACCEPT 
 TRACE: 2 a727a6df mangle:POSTROUTING:return:
 TRACE: 2 a727a6df mangle:POSTROUTING:policy:ACCEPT

Comment 4 Surya Seetharaman 2022-05-03 19:03:22 UTC

sh-4.4# curl --local-port 35353 172.30.92.176:80
curl: (7) Failed to connect to 172.30.92.176 port 80: Connection timed out


sh-4.4# conntrack -E | grep 35353
    [NEW] tcp      6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353
    [NEW] tcp      6 120 SYN_SENT src=10.131.0.2 dst=172.30.92.176 sport=35353 dport=80 [UNREPLIED] src=10.131.0.18 dst=10.131.0.2 sport=8080 dport=35353 zone=9
    [NEW] tcp      6 120 SYN_SENT src=10.131.0.2 dst=10.131.0.18 sport=35353 dport=8080 [UNREPLIED] src=10.131.0.18 dst=10.131.0.2 sport=8080 dport=35353 zone=26
 [UPDATE] tcp      6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 51 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 43 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 27 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353
[DESTROY] tcp      6 src=10.131.0.2 dst=10.131.0.18 sport=35353 dport=8080 src=10.131.0.18 dst=10.131.0.2 sport=8080 dport=35353 zone=26
[DESTROY] tcp      6 src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353
[DESTROY] tcp      6 src=10.131.0.2 dst=172.30.92.176 sport=35353 dport=80 src=10.131.0.18 dst=10.131.0.2 sport=8080 dport=35353 zone=9


this is funny, am I messing with conntrack zones somehow? since conntrack is clearly not receiving the packets...


sh-4.4# tcpdump -neep -i any port 35353
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
18:54:40.508851 Out 3a:24:36:01:35:0b ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 172.30.92.176.http: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163480049 ecr 0,nop,wscale 7], length 0
18:54:40.509523 Out 0a:58:0a:83:00:01 ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 10.131.0.18.webcache: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163480049 ecr 0,nop,wscale 7], length 0
18:54:40.509564   P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991695001 ecr 4163480049,nop,wscale 7], length 0
18:54:40.509810  In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991695001 ecr 4163480049,nop,wscale 7], length 0
18:54:41.523411   P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991696015 ecr 4163480049,nop,wscale 7], length 0
18:54:41.523439 Out 3a:24:36:01:35:0b ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 172.30.92.176.http: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163481064 ecr 0,nop,wscale 7], length 0
18:54:41.523411  In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991696015 ecr 4163480049,nop,wscale 7], length 0
18:54:41.524048 Out 0a:58:0a:83:00:01 ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 10.131.0.18.webcache: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163481064 ecr 0,nop,wscale 7], length 0
18:54:41.524079   P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991696015 ecr 4163480049,nop,wscale 7], length 0
18:54:41.524079  In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991696015 ecr 4163480049,nop,wscale 7], length 0
18:54:43.571399   P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991698063 ecr 4163480049,nop,wscale 7], length 0
18:54:43.571432 Out 3a:24:36:01:35:0b ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 172.30.92.176.http: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163483112 ecr 0,nop,wscale 7], length 0
18:54:43.571399  In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991698063 ecr 4163480049,nop,wscale 7], length 0
18:54:43.571477 Out 0a:58:0a:83:00:01 ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 10.131.0.18.webcache: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163483112 ecr 0,nop,wscale 7], length 0
18:54:43.571522   P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991698063 ecr 4163480049,nop,wscale 7], length 0
18:54:43.571522  In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991698063 ecr 4163480049,nop,wscale 7], length 0
18:54:47.603408   P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991702095 ecr 4163480049,nop,wscale 7], length 0
18:54:47.603425 Out 3a:24:36:01:35:0b ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 172.30.92.176.http: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163487144 ecr 0,nop,wscale 7], length 0
18:54:47.603408  In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991702095 ecr 4163480049,nop,wscale 7], length 0
18:54:47.603476 Out 0a:58:0a:83:00:01 ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 10.131.0.18.webcache: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163487144 ecr 0,nop,wscale 7], length 0
18:54:47.603499   P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991702095 ecr 4163480049,nop,wscale 7], length 0
18:54:47.603499  In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991702095 ecr 4163480049,nop,wscale 7], length 0

Comment 5 Surya Seetharaman 2022-05-03 19:22:09 UTC

ok now I am seeing very very weird issues which suggest I should stop for today:

sh-4.4# curl --local-port 35356 172.30.92.176:80
curl: (7) Failed to connect to 172.30.92.176 port 80: Connection timed out
sh-4.4# curl --local-port 35356 172.30.92.176:80


sh-4.4# conntrack -E | grep 35356
    [NEW] tcp      6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675
 [UPDATE] tcp      6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675
 [UPDATE] tcp      6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675
 [UPDATE] tcp      6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675
 [UPDATE] tcp      6 51 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675
 [UPDATE] tcp      6 43 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675
 [UPDATE] tcp      6 27 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675
[DESTROY] tcp      6 src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675

    [NEW] tcp      6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488
 [UPDATE] tcp      6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488
 [UPDATE] tcp      6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488
 [UPDATE] tcp      6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488
 [UPDATE] tcp      6 51 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488


who are 47675 & 45488 ? maybe I am reusing 35356?


I retried with other ports...

sh-4.4# curl --local-port 35350 172.30.92.176:80

sh-4.4# conntrack -E | grep 35350
    [NEW] tcp      6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35350 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=2695
 [UPDATE] tcp      6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35350 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=2695
 [UPDATE] tcp      6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35350 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=2695
 [UPDATE] tcp      6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35350 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=2695

This time even 35353 doesn't work :/

sh-4.4# conntrack -E | grep 35353
    [NEW] tcp      6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456
 [UPDATE] tcp      6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456
 [UPDATE] tcp      6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456
 [UPDATE] tcp      6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456
 [UPDATE] tcp      6 51 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456

Comment 6 Surya Seetharaman 2022-05-04 11:09:33 UTC

sh-4.4# ovs-appctl -t /var/run/ovn/ovn-controller.3397.ctl ct-zone-list
openshift-ingress_router-default-7bd676c647-655rz 17
openshift-multus_network-metrics-daemon-7rrt6 13
openshift-network-diagnostics_network-check-source-c499d84b5-2jx78 18
48e638c0-61e2-45f9-95fb-e5d995df7d44_snat 0
75401857-2f15-40ba-8cd0-0567d89edfed_snat 4
0176ad2f-b659-44c5-9bf2-f840c9593d93_snat 8
openshift-monitoring_thanos-querier-fdf9b9bdc-krmrr 15
openshift-image-registry_image-registry-56cf9ff474-pltd8 26
openshift-monitoring_openshift-state-metrics-6897bc8b6b-9dwc8 21
openshift-monitoring_kube-state-metrics-8b7fdd9d6-bhrm7 20
k8s-ip-10-0-189-71.us-west-2.compute.internal 7
0176ad2f-b659-44c5-9bf2-f840c9593d93_dnat 9
openshift-console_downloads-8656667bd6-2qtb4 24
29a2b52d-6587-4d64-ae45-3f198626bfd5_dnat 6
29a2b52d-6587-4d64-ae45-3f198626bfd5_snat 5
openshift-network-diagnostics_network-check-target-n2qnd 12
openshift-monitoring_alertmanager-main-1 14
7cd14d33-7858-4867-b718-71dafd1540ee_snat 3
75401857-2f15-40ba-8cd0-0567d89edfed_dnat 1
7cd14d33-7858-4867-b718-71dafd1540ee_dnat 2
6ac33542-9c18-4574-a149-399a805e1752_snat 23
6ac33542-9c18-4574-a149-399a805e1752_dnat 25
openshift-e2e-loki_loki-promtail-v989l 11
openshift-dns_dns-default-f8dkm 22
surya_hello-world-2-5c87676b7-cps2p 27
openshift-ingress-canary_ingress-canary-5lcc4 16
48e638c0-61e2-45f9-95fb-e5d995df7d44_dnat 10
openshift-monitoring_telemeter-client-5d457bf857-52dfn 19

    [NEW] tcp      6 120 SYN_SENT src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 [UNREPLIED] src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353
    [NEW] tcp      6 120 SYN_SENT src=10.131.0.2 dst=172.30.51.103 sport=35353 dport=80 [UNREPLIED] src=10.131.0.21 dst=10.131.0.2 sport=8080 dport=35353 zone=7
    [NEW] tcp      6 120 SYN_SENT src=10.131.0.2 dst=10.131.0.21 sport=35353 dport=8080 [UNREPLIED] src=10.131.0.21 dst=10.131.0.2 sport=8080 dport=35353 zone=27
 [UPDATE] tcp      6 58 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 57 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 55 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 51 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 43 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353
 [UPDATE] tcp      6 27 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353
[DESTROY] tcp      6 src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353
[DESTROY] tcp      6 src=10.131.0.2 dst=10.131.0.21 sport=35353 dport=8080 src=10.131.0.21 dst=10.131.0.2 sport=8080 dport=35353 zone=27
[DESTROY] tcp      6 src=10.131.0.2 dst=172.30.51.103 sport=35353 dport=80 src=10.131.0.21 dst=10.131.0.2 sport=8080 dport=35353 zone=7

11:04:32.279714 8a:93:43:a7:46:38 > 0a:58:0a:83:00:01, ethertype IPv4 (0x0800), length 74: 10.131.0.2.35353 > 172.30.51.103.http: Flags [S], seq 565827876, win 26583, options [mss 8861,sackOK,TS val 96881880 ecr 0,nop,wscale 7], length 0
11:04:32.279673 0a:58:0a:83:00:15 > 8a:93:43:a7:46:38, ethertype IPv4 (0x0800), length 74: 172.30.51.103.http > 10.131.0.2.35353: Flags [S.], seq 3519086944, ack 565827877, win 26547, options [mss 8861,sackOK,TS val 4001253272 ecr 96874772,nop,wscale 7], length 0
11:04:32.279782 0a:58:0a:83:00:15 > 8a:93:43:a7:46:38, ethertype IPv4 (0x0800), length 74: 172.30.51.103.http > 10.131.0.2.35353: Flags [S.], seq 3519086944, ack 565827877, win 26547, options [mss 8861,sackOK,TS val 4001253272 ecr 96874772,nop,wscale 7], length 0
11:04:40.599698 8a:93:43:a7:46:38 > 0a:58:0a:83:00:01, ethertype IPv4 (0x0800), length 74: 10.131.0.2.35353 > 172.30.51.103.http: Flags [S], seq 565827876, win 26583, options [mss 8861,sackOK,TS val 96890200 ecr 0,nop,wscale 7], length 0
11:04:40.599737 0a:58:0a:83:00:15 > 8a:93:43:a7:46:38, ethertype IPv4 (0x0800), length 74: 172.30.51.103.http > 10.131.0.2.35353: Flags [S.], seq 3519086944, ack 565827877, win 26547, options [mss 8861,sackOK,TS val 4001261592 ecr 96874772,nop,wscale 7], length 0
11:04:40.599752 0a:58:0a:83:00:15 > 8a:93:43:a7:46:38, ethertype IPv4 (0x0800), length 74: 172.30.51.103.http > 10.131.0.2.35353: Flags [S.], seq 3519086944, ack 565827877, win 26547, options [mss 8861,sackOK,TS val 4001261592 ecr 96874772,nop,wscale 7], length 0

Everything looks normal I can't understand why conntrack is unable to match the response packet to the tuple?

Comment 7 Surya Seetharaman 2022-05-06 12:41:41 UTC

https://coreos.slack.com/archives/CDCP2LA9L/p1651832420485449?thread_ts=1651672510.640529&cid=CDCP2LA9L

Comment 8 Surya Seetharaman 2022-05-06 16:58:37 UTC

We feel it might be a netfilter issue, opened https://bugzilla.redhat.com/show_bug.cgi?id=2082663 to track this and see what response we get.

Comment 9 Surya Seetharaman 2022-05-30 17:27:43 UTC

*** Bug 2082663 has been marked as a duplicate of this bug. ***

Comment 10 Surya Seetharaman 2022-07-08 18:53:58 UTC

rp filtering thing is only for IPV4: https://github.com/ovn-org/ovn-kubernetes/pull/3006#discussion_r917062397; V6 doesn't have this problem

Comment 11 Surya Seetharaman 2022-07-22 10:40:48 UTC

The specific commit that will fix this bug is https://github.com/openshift/ovn-kubernetes/pull/1210/commits/944b5c2f606b06c0a82624a9dea5a9fc4e7d4124

Hey Huiran: Please check on v4 and v6 cluster if the host -> internal traffic policy work flow works fine in OCP after this merges.

Comment 17 errata-xmlrpc 2023-01-17 19:48:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399

Note You need to log in before you can comment on or make changes to this bug.