Bug 2078691
Summary: | [OVN] Node to service traffic is blocked if service is "internalTrafficPolicy: Local" even backed pod is on the same node | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | huirwang | |
Component: | Networking | Assignee: | Surya Seetharaman <surya> | |
Networking sub component: | ovn-kubernetes | QA Contact: | huirwang | |
Status: | CLOSED ERRATA | Docs Contact: | ||
Severity: | high | |||
Priority: | unspecified | CC: | ffernand, surya | |
Version: | 4.11 | |||
Target Milestone: | --- | |||
Target Release: | 4.12.0 | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | No Doc Update | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 2105444 (view as bug list) | Environment: | ||
Last Closed: | 2023-01-17 19:48:11 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 2082663 | |||
Bug Blocks: | 2105444 |
Description
huirwang
2022-04-26 03:07:38 UTC
hmm on first screen I'm not sure what's wrong here: I am able to reproduce the bug. *mangle :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :KUBE-KUBELET-CANARY - [0:0] :OVN-KUBE-ITP - [0:0] [339157:263486942] -A OUTPUT -j OVN-KUBE-ITP [4:240] -A OVN-KUBE-ITP -d 172.30.198.99/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1745ec/0xffffffff The correct IPTable rule is being hit We also have the correct routes setup: sh-4.4# ip rule ls 0: from all lookup local 30: from all fwmark 0x1745ec lookup 7 32766: from all lookup main 32767: from all lookup default sh-4.4# ip r show table 7 172.30.0.0/16 via 10.129.2.1 dev ovn-k8s-mp0 sh-4.4# ip r default via 10.0.128.1 dev br-ex proto dhcp metric 48 10.0.128.0/18 dev br-ex proto kernel scope link src 10.0.184.157 metric 48 10.128.0.0/14 via 10.129.2.1 dev ovn-k8s-mp0 10.129.2.0/23 dev ovn-k8s-mp0 proto kernel scope link src 10.129.2.2 169.254.169.0/30 via 10.0.128.1 dev br-ex 169.254.169.3 via 10.129.2.1 dev ovn-k8s-mp0 172.30.0.0/16 via 10.0.128.1 dev br-ex mtu 8901 I am able to see packet entering ovn-k8s-mp0 and response is coming out: sh-4.4# tcpdump -i ovn-k8s-mp0 -neep | grep 172.30.198.99 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ovn-k8s-mp0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:51:21.926439 fa:83:d1:e3:8b:60 > 0a:58:0a:81:02:01, ethertype IPv4 (0x0800), length 74: 10.129.2.2.58502 > 172.30.198.99.http: Flags [S], seq 1530622593, win 26583, options [mss 8861,sackOK,TS val 3987064920 ecr 0,nop,wscale 7], length 0 12:51:21.927243 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823408405 ecr 3987064920,nop,wscale 7], length 0 12:51:22.990351 fa:83:d1:e3:8b:60 > 0a:58:0a:81:02:01, ethertype IPv4 (0x0800), length 74: 10.129.2.2.58502 > 172.30.198.99.http: Flags [S], seq 1530622593, win 26583, options [mss 8861,sackOK,TS val 3987065984 ecr 0,nop,wscale 7], length 0 12:51:22.990336 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823409469 ecr 3987064920,nop,wscale 7], length 0 12:51:22.990829 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823409469 ecr 3987064920,nop,wscale 7], length 0 12:51:25.038346 fa:83:d1:e3:8b:60 > 0a:58:0a:81:02:01, ethertype IPv4 (0x0800), length 74: 10.129.2.2.58502 > 172.30.198.99.http: Flags [S], seq 1530622593, win 26583, options [mss 8861,sackOK,TS val 3987068032 ecr 0,nop,wscale 7], length 0 12:51:25.038335 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823411517 ecr 3987064920,nop,wscale 7], length 0 12:51:25.038407 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 172.30.198.99.http > 10.129.2.2.58502: Flags [S.], seq 3766486950, ack 1530622594, win 26547, options [mss 8861,sackOK,TS val 2823411517 ecr 3987064920,nop,wscale 7], length 0 Definitely the backend pod is responding back: 12:54:18.251169 0a:58:0a:81:02:01 > 0a:58:0a:81:02:0a, ethertype IPv4 (0x0800), length 74: 10.129.2.2.59580 > 10.129.2.10.webcache: Flags [S], seq 2794852057, win 26583, options [mss 8861,sackOK,TS val 3987241244 ecr 0,nop,wscale 7], length 0 12:54:18.251199 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 10.129.2.10.webcache > 10.129.2.2.59580: Flags [S.], seq 2270704619, ack 2794852058, win 26547, options [mss 8861,sackOK,TS val 2823584729 ecr 3987241244,nop,wscale 7], length 0 12:54:19.310336 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 10.129.2.10.webcache > 10.129.2.2.59580: Flags [S.], seq 2270704619, ack 2794852058, win 26547, options [mss 8861,sackOK,TS val 2823585789 ecr 3987241244,nop,wscale 7], length 0 12:54:19.310980 0a:58:0a:81:02:01 > 0a:58:0a:81:02:0a, ethertype IPv4 (0x0800), length 74: 10.129.2.2.59580 > 10.129.2.10.webcache: Flags [S], seq 2794852057, win 26583, options [mss 8861,sackOK,TS val 3987242304 ecr 0,nop,wscale 7], length 0 12:54:19.311003 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 10.129.2.10.webcache > 10.129.2.2.59580: Flags [S.], seq 2270704619, ack 2794852058, win 26547, options [mss 8861,sackOK,TS val 2823585789 ecr 3987241244,nop,wscale 7], length 0 12:54:21.358339 0a:58:0a:81:02:0a > fa:83:d1:e3:8b:60, ethertype IPv4 (0x0800), length 74: 10.129.2.10.webcache > 10.129.2.2.59580: Flags [S.], seq 2270704619, ack 2794852058, win 26547, options [mss 8861,sackOK,TS val 2823587837 ecr 3987241244,nop,wscale 7], length 0 12:54:21.358373 0a:58:0a:81:02:01 > 0a:58:0a:81:02:0a, ethertype IPv4 (0x0800), length 74: 10.129.2.2.59580 > 10.129.2.10.webcache: Flags [S], seq 2794852057, win 26583, options [mss 8861,sackOK,TS val 3987244352 ecr 0,nop,wscale 7], length 0 but the destination is 10.129.2.2 and not the hostIP...so the packet returnes safely? I am not sure why we are doing an SNAT downstream at OVN-KUBE-SNAT-MGMTPORT while I didn't have this problem on a KIND cluster. note-to-self: BTW I think I found another bug here: [9236:554160] -A OVN-KUBE-SNAT-MGMTPORT -o ovn-k8s-mp0 -m comment --comment "OVN SNAT to Management Port" -j SNAT --to-source 10.129.2.2 [0:0] -A OVN-KUBE-SNAT-MGMTPORT -p tcp -m tcp --dport 30348 -j RETURN [0:0] -A OVN-KUBE-SNAT-MGMTPORT -p tcp -m tcp --dport 30212 -j RETURN the order.... is looking wrong for sure. Need to change this. hmm all the right iptable rules are being hit PACKET: 2 70c323cb OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN TRACE: 2 70c323cb raw:OUTPUT:rule:0x8:CONTINUE -4 -t raw -A OUTPUT -d 172.30.92.176/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 1/sec -j TRACE TRACE: 2 70c323cb raw:OUTPUT:return: TRACE: 2 70c323cb raw:OUTPUT:policy:ACCEPT PACKET: 2 70c323cb OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN TRACE: 2 70c323cb mangle:OUTPUT:rule:0x8:JUMP:OVN-KUBE-ITP -4 -t mangle -A OUTPUT -j OVN-KUBE-ITP TRACE: 2 70c323cb mangle:OVN-KUBE-ITP:rule:0x9:CONTINUE -4 -t mangle -A OVN-KUBE-ITP -d 172.30.92.176/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1745ec/0xffffffff TRACE: 2 70c323cb mangle:OVN-KUBE-ITP:return: TRACE: 2 70c323cb mangle:OUTPUT:return: TRACE: 2 70c323cb mangle:OUTPUT:policy:ACCEPT PACKET: 2 70c323cb OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN MARK=0x1745ec TRACE: 2 70c323cb nat:OUTPUT:rule:0x1c:JUMP:OVN-KUBE-EXTERNALIP -4 -t nat -A OUTPUT -j OVN-KUBE-EXTERNALIP TRACE: 2 70c323cb nat:OVN-KUBE-EXTERNALIP:return: TRACE: 2 70c323cb nat:OUTPUT:rule:0x1a:JUMP:OVN-KUBE-NODEPORT -4 -t nat -A OUTPUT -j OVN-KUBE-NODEPORT TRACE: 2 70c323cb nat:OVN-KUBE-NODEPORT:return: TRACE: 2 70c323cb nat:OUTPUT:rule:0x18:JUMP:OVN-KUBE-ITP -4 -t nat -A OUTPUT -j OVN-KUBE-ITP TRACE: 2 70c323cb nat:OVN-KUBE-ITP:return: TRACE: 2 70c323cb nat:OUTPUT:return: TRACE: 2 70c323cb nat:OUTPUT:policy:ACCEPT PACKET: 2 70c323cb OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN MARK=0x1745ec TRACE: 2 70c323cb filter:OUTPUT:rule:0x7:JUMP:KUBE-FIREWALL -4 -t filter -A OUTPUT -j KUBE-FIREWALL TRACE: 2 70c323cb filter:KUBE-FIREWALL:return: TRACE: 2 70c323cb filter:OUTPUT:return: TRACE: 2 70c323cb filter:OUTPUT:policy:ACCEPT TRACE: 2 70c323cb mangle:POSTROUTING:return: TRACE: 2 70c323cb mangle:POSTROUTING:policy:ACCEPT PACKET: 2 70c323cb OUT=ovn-k8s-mp0 SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40857DF SPORT=58420 DPORT=80 SYN MARK=0x1745ec TRACE: 2 70c323cb nat:POSTROUTING:rule:0x12:JUMP:OVN-KUBE-SNAT-MGMTPORT -4 -t nat -A POSTROUTING -o ovn-k8s-mp0 -j OVN-KUBE-SNAT-MGMTPORT TRACE: 2 70c323cb nat:OVN-KUBE-SNAT-MGMTPORT:rule:0x2a:ACCEPT -4 -t nat -A OVN-KUBE-SNAT-MGMTPORT -o ovn-k8s-mp0 -m comment --comment "OVN SNAT to Management Port" -j SNAT --to-source 10.131.0.2 PACKET: 2 a727a6df OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40858DF SPORT=58420 DPORT=80 SYN TRACE: 2 a727a6df raw:OUTPUT:rule:0x8:CONTINUE -4 -t raw -A OUTPUT -d 172.30.92.176/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 1/sec -j TRACE TRACE: 2 a727a6df raw:OUTPUT:return: TRACE: 2 a727a6df raw:OUTPUT:policy:ACCEPT PACKET: 2 a727a6df OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40858DF SPORT=58420 DPORT=80 SYN TRACE: 2 a727a6df mangle:OUTPUT:rule:0x8:JUMP:OVN-KUBE-ITP -4 -t mangle -A OUTPUT -j OVN-KUBE-ITP TRACE: 2 a727a6df mangle:OVN-KUBE-ITP:rule:0x9:CONTINUE -4 -t mangle -A OVN-KUBE-ITP -d 172.30.92.176/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1745ec/0xffffffff TRACE: 2 a727a6df mangle:OVN-KUBE-ITP:return: TRACE: 2 a727a6df mangle:OUTPUT:return: TRACE: 2 a727a6df mangle:OUTPUT:policy:ACCEPT PACKET: 2 a727a6df OUT=br-ex SRC=10.0.158.184 DST=172.30.92.176 LEN=60 TOS=0x0 TTL=64 ID=40858DF SPORT=58420 DPORT=80 SYN MARK=0x1745ec TRACE: 2 a727a6df filter:OUTPUT:rule:0x7:JUMP:KUBE-FIREWALL -4 -t filter -A OUTPUT -j KUBE-FIREWALL TRACE: 2 a727a6df filter:KUBE-FIREWALL:return: TRACE: 2 a727a6df filter:OUTPUT:return: TRACE: 2 a727a6df filter:OUTPUT:policy:ACCEPT TRACE: 2 a727a6df mangle:POSTROUTING:return: TRACE: 2 a727a6df mangle:POSTROUTING:policy:ACCEPT sh-4.4# curl --local-port 35353 172.30.92.176:80 curl: (7) Failed to connect to 172.30.92.176 port 80: Connection timed out sh-4.4# conntrack -E | grep 35353 [NEW] tcp 6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353 [NEW] tcp 6 120 SYN_SENT src=10.131.0.2 dst=172.30.92.176 sport=35353 dport=80 [UNREPLIED] src=10.131.0.18 dst=10.131.0.2 sport=8080 dport=35353 zone=9 [NEW] tcp 6 120 SYN_SENT src=10.131.0.2 dst=10.131.0.18 sport=35353 dport=8080 [UNREPLIED] src=10.131.0.18 dst=10.131.0.2 sport=8080 dport=35353 zone=26 [UPDATE] tcp 6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 51 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 43 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 27 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353 [DESTROY] tcp 6 src=10.131.0.2 dst=10.131.0.18 sport=35353 dport=8080 src=10.131.0.18 dst=10.131.0.2 sport=8080 dport=35353 zone=26 [DESTROY] tcp 6 src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=35353 [DESTROY] tcp 6 src=10.131.0.2 dst=172.30.92.176 sport=35353 dport=80 src=10.131.0.18 dst=10.131.0.2 sport=8080 dport=35353 zone=9 this is funny, am I messing with conntrack zones somehow? since conntrack is clearly not receiving the packets... sh-4.4# tcpdump -neep -i any port 35353 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes 18:54:40.508851 Out 3a:24:36:01:35:0b ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 172.30.92.176.http: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163480049 ecr 0,nop,wscale 7], length 0 18:54:40.509523 Out 0a:58:0a:83:00:01 ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 10.131.0.18.webcache: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163480049 ecr 0,nop,wscale 7], length 0 18:54:40.509564 P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991695001 ecr 4163480049,nop,wscale 7], length 0 18:54:40.509810 In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991695001 ecr 4163480049,nop,wscale 7], length 0 18:54:41.523411 P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991696015 ecr 4163480049,nop,wscale 7], length 0 18:54:41.523439 Out 3a:24:36:01:35:0b ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 172.30.92.176.http: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163481064 ecr 0,nop,wscale 7], length 0 18:54:41.523411 In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991696015 ecr 4163480049,nop,wscale 7], length 0 18:54:41.524048 Out 0a:58:0a:83:00:01 ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 10.131.0.18.webcache: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163481064 ecr 0,nop,wscale 7], length 0 18:54:41.524079 P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991696015 ecr 4163480049,nop,wscale 7], length 0 18:54:41.524079 In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991696015 ecr 4163480049,nop,wscale 7], length 0 18:54:43.571399 P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991698063 ecr 4163480049,nop,wscale 7], length 0 18:54:43.571432 Out 3a:24:36:01:35:0b ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 172.30.92.176.http: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163483112 ecr 0,nop,wscale 7], length 0 18:54:43.571399 In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991698063 ecr 4163480049,nop,wscale 7], length 0 18:54:43.571477 Out 0a:58:0a:83:00:01 ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 10.131.0.18.webcache: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163483112 ecr 0,nop,wscale 7], length 0 18:54:43.571522 P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991698063 ecr 4163480049,nop,wscale 7], length 0 18:54:43.571522 In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991698063 ecr 4163480049,nop,wscale 7], length 0 18:54:47.603408 P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991702095 ecr 4163480049,nop,wscale 7], length 0 18:54:47.603425 Out 3a:24:36:01:35:0b ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 172.30.92.176.http: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163487144 ecr 0,nop,wscale 7], length 0 18:54:47.603408 In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991702095 ecr 4163480049,nop,wscale 7], length 0 18:54:47.603476 Out 0a:58:0a:83:00:01 ethertype IPv4 (0x0800), length 76: 10.131.0.2.35353 > 10.131.0.18.webcache: Flags [S], seq 2230914970, win 26583, options [mss 8861,sackOK,TS val 4163487144 ecr 0,nop,wscale 7], length 0 18:54:47.603499 P 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 10.131.0.18.webcache > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991702095 ecr 4163480049,nop,wscale 7], length 0 18:54:47.603499 In 0a:58:0a:83:00:12 ethertype IPv4 (0x0800), length 76: 172.30.92.176.http > 10.131.0.2.35353: Flags [S.], seq 1594570653, ack 2230914971, win 26547, options [mss 8861,sackOK,TS val 991702095 ecr 4163480049,nop,wscale 7], length 0 ok now I am seeing very very weird issues which suggest I should stop for today: sh-4.4# curl --local-port 35356 172.30.92.176:80 curl: (7) Failed to connect to 172.30.92.176 port 80: Connection timed out sh-4.4# curl --local-port 35356 172.30.92.176:80 sh-4.4# conntrack -E | grep 35356 [NEW] tcp 6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675 [UPDATE] tcp 6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675 [UPDATE] tcp 6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675 [UPDATE] tcp 6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675 [UPDATE] tcp 6 51 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675 [UPDATE] tcp 6 43 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675 [UPDATE] tcp 6 27 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675 [DESTROY] tcp 6 src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=47675 [NEW] tcp 6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488 [UPDATE] tcp 6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488 [UPDATE] tcp 6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488 [UPDATE] tcp 6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488 [UPDATE] tcp 6 51 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35356 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=45488 who are 47675 & 45488 ? maybe I am reusing 35356? I retried with other ports... sh-4.4# curl --local-port 35350 172.30.92.176:80 sh-4.4# conntrack -E | grep 35350 [NEW] tcp 6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35350 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=2695 [UPDATE] tcp 6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35350 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=2695 [UPDATE] tcp 6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35350 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=2695 [UPDATE] tcp 6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35350 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=2695 This time even 35353 doesn't work :/ sh-4.4# conntrack -E | grep 35353 [NEW] tcp 6 120 SYN_SENT src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 [UNREPLIED] src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456 [UPDATE] tcp 6 58 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456 [UPDATE] tcp 6 57 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456 [UPDATE] tcp 6 55 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456 [UPDATE] tcp 6 51 SYN_RECV src=10.0.158.184 dst=172.30.92.176 sport=35353 dport=80 src=172.30.92.176 dst=10.131.0.2 sport=80 dport=7456 sh-4.4# ovs-appctl -t /var/run/ovn/ovn-controller.3397.ctl ct-zone-list openshift-ingress_router-default-7bd676c647-655rz 17 openshift-multus_network-metrics-daemon-7rrt6 13 openshift-network-diagnostics_network-check-source-c499d84b5-2jx78 18 48e638c0-61e2-45f9-95fb-e5d995df7d44_snat 0 75401857-2f15-40ba-8cd0-0567d89edfed_snat 4 0176ad2f-b659-44c5-9bf2-f840c9593d93_snat 8 openshift-monitoring_thanos-querier-fdf9b9bdc-krmrr 15 openshift-image-registry_image-registry-56cf9ff474-pltd8 26 openshift-monitoring_openshift-state-metrics-6897bc8b6b-9dwc8 21 openshift-monitoring_kube-state-metrics-8b7fdd9d6-bhrm7 20 k8s-ip-10-0-189-71.us-west-2.compute.internal 7 0176ad2f-b659-44c5-9bf2-f840c9593d93_dnat 9 openshift-console_downloads-8656667bd6-2qtb4 24 29a2b52d-6587-4d64-ae45-3f198626bfd5_dnat 6 29a2b52d-6587-4d64-ae45-3f198626bfd5_snat 5 openshift-network-diagnostics_network-check-target-n2qnd 12 openshift-monitoring_alertmanager-main-1 14 7cd14d33-7858-4867-b718-71dafd1540ee_snat 3 75401857-2f15-40ba-8cd0-0567d89edfed_dnat 1 7cd14d33-7858-4867-b718-71dafd1540ee_dnat 2 6ac33542-9c18-4574-a149-399a805e1752_snat 23 6ac33542-9c18-4574-a149-399a805e1752_dnat 25 openshift-e2e-loki_loki-promtail-v989l 11 openshift-dns_dns-default-f8dkm 22 surya_hello-world-2-5c87676b7-cps2p 27 openshift-ingress-canary_ingress-canary-5lcc4 16 48e638c0-61e2-45f9-95fb-e5d995df7d44_dnat 10 openshift-monitoring_telemeter-client-5d457bf857-52dfn 19 [NEW] tcp 6 120 SYN_SENT src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 [UNREPLIED] src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353 [NEW] tcp 6 120 SYN_SENT src=10.131.0.2 dst=172.30.51.103 sport=35353 dport=80 [UNREPLIED] src=10.131.0.21 dst=10.131.0.2 sport=8080 dport=35353 zone=7 [NEW] tcp 6 120 SYN_SENT src=10.131.0.2 dst=10.131.0.21 sport=35353 dport=8080 [UNREPLIED] src=10.131.0.21 dst=10.131.0.2 sport=8080 dport=35353 zone=27 [UPDATE] tcp 6 58 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 57 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 55 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 51 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 43 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353 [UPDATE] tcp 6 27 SYN_RECV src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353 [DESTROY] tcp 6 src=10.0.189.71 dst=172.30.51.103 sport=35353 dport=80 src=172.30.51.103 dst=10.131.0.2 sport=80 dport=35353 [DESTROY] tcp 6 src=10.131.0.2 dst=10.131.0.21 sport=35353 dport=8080 src=10.131.0.21 dst=10.131.0.2 sport=8080 dport=35353 zone=27 [DESTROY] tcp 6 src=10.131.0.2 dst=172.30.51.103 sport=35353 dport=80 src=10.131.0.21 dst=10.131.0.2 sport=8080 dport=35353 zone=7 11:04:32.279714 8a:93:43:a7:46:38 > 0a:58:0a:83:00:01, ethertype IPv4 (0x0800), length 74: 10.131.0.2.35353 > 172.30.51.103.http: Flags [S], seq 565827876, win 26583, options [mss 8861,sackOK,TS val 96881880 ecr 0,nop,wscale 7], length 0 11:04:32.279673 0a:58:0a:83:00:15 > 8a:93:43:a7:46:38, ethertype IPv4 (0x0800), length 74: 172.30.51.103.http > 10.131.0.2.35353: Flags [S.], seq 3519086944, ack 565827877, win 26547, options [mss 8861,sackOK,TS val 4001253272 ecr 96874772,nop,wscale 7], length 0 11:04:32.279782 0a:58:0a:83:00:15 > 8a:93:43:a7:46:38, ethertype IPv4 (0x0800), length 74: 172.30.51.103.http > 10.131.0.2.35353: Flags [S.], seq 3519086944, ack 565827877, win 26547, options [mss 8861,sackOK,TS val 4001253272 ecr 96874772,nop,wscale 7], length 0 11:04:40.599698 8a:93:43:a7:46:38 > 0a:58:0a:83:00:01, ethertype IPv4 (0x0800), length 74: 10.131.0.2.35353 > 172.30.51.103.http: Flags [S], seq 565827876, win 26583, options [mss 8861,sackOK,TS val 96890200 ecr 0,nop,wscale 7], length 0 11:04:40.599737 0a:58:0a:83:00:15 > 8a:93:43:a7:46:38, ethertype IPv4 (0x0800), length 74: 172.30.51.103.http > 10.131.0.2.35353: Flags [S.], seq 3519086944, ack 565827877, win 26547, options [mss 8861,sackOK,TS val 4001261592 ecr 96874772,nop,wscale 7], length 0 11:04:40.599752 0a:58:0a:83:00:15 > 8a:93:43:a7:46:38, ethertype IPv4 (0x0800), length 74: 172.30.51.103.http > 10.131.0.2.35353: Flags [S.], seq 3519086944, ack 565827877, win 26547, options [mss 8861,sackOK,TS val 4001261592 ecr 96874772,nop,wscale 7], length 0 Everything looks normal I can't understand why conntrack is unable to match the response packet to the tuple? We feel it might be a netfilter issue, opened https://bugzilla.redhat.com/show_bug.cgi?id=2082663 to track this and see what response we get. *** Bug 2082663 has been marked as a duplicate of this bug. *** rp filtering thing is only for IPV4: https://github.com/ovn-org/ovn-kubernetes/pull/3006#discussion_r917062397; V6 doesn't have this problem The specific commit that will fix this bug is https://github.com/openshift/ovn-kubernetes/pull/1210/commits/944b5c2f606b06c0a82624a9dea5a9fc4e7d4124 Hey Huiran: Please check on v4 and v6 cluster if the host -> internal traffic policy work flow works fine in OCP after this merges. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399 |