Created attachment 1799934 [details] ovn-sbdb Created attachment 1799934 [details] ovn-sbdb Description of problem: ovn-trace does not show table 13 getting hit and ct_lb(backends=10.244.0.4:8080). Version-Release number of selected component (if applicable): sh-5.0# ovn-nbctl --version ovn-nbctl 21.03.1 Open vSwitch Library 2.15.90 DB Schema 5.31.0 How reproducible: Always Steps to Reproduce: 1. Create a client pod and clusterIP svc (any object that will need the lb dnat) 2. curling from client to service works which means the load balancing and dnating work fine. 3. Run the ovn-trace: "ovn-trace --ct new 'inport=="default_client-on-ovn-worker" && eth.src==0a:58:0a:f4:00:09 && eth.dst==0a:58:0a:f4:00:01 && tcp && tcp.src==80 && tcp.dst==80 && ip4.src==10.244.0.9 && ip4.dst==10.96.245.163 && ip.ttl==64' --lb-dst 10.244.0.10:80" Actual results: Trace shows: ingress(dp="ovn-worker", inport="default_client-on-ovn-worker") --------------------------------------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:4802): inport == "default_client-on-ovn-worker" && eth.src == {0a:58:0a:f4:00:09}, priority 50, uuid 308b2fd0 next; 1. ls_in_port_sec_ip (ovn-northd.c:4450): inport == "default_client-on-ovn-worker" && eth.src == 0a:58:0a:f4:00:09 && ip4.src == {10.244.0.9}, priority 90, uuid 0134ca82 next; 5. ls_in_pre_acl (ovn-northd.c:5004): ip, priority 100, uuid a27e108e reg0[0] = 1; next; 6. ls_in_pre_lb (ovn-northd.c:5172): ip, priority 100, uuid ac3d55f5 reg0[2] = 1; next; 7. ls_in_pre_stateful (ovn-northd.c:5199): reg0[2] == 1 && ip4 && tcp, priority 120, uuid dcf60d09 reg1 = ip4.dst; reg2[0..15] = tcp.dst; ct_lb; ct_lb ----- 8. ls_in_acl_hint (ovn-northd.c:5290): !ct.trk, priority 5, uuid 71d49e33 reg0[8] = 1; reg0[9] = 1; next; 22. ls_in_l2_lkup (ovn-northd.c:7374): eth.dst == 0a:58:0a:f4:00:01, priority 50, uuid d75fc9d6 outport = "stor-ovn-worker"; output; on the ingress pipeline of ovn logical switch Expected results: Trace should show packet matching on flow with prio 120 in table 13: example snippet from a trace on older ovn-trace versions: old-trace (ovn-trace --db tcp:172.18.0.3:6642 ovn-worker --ct=new 'inport=="default_web-client-pod-5xh74" && eth.dst==0a:58:0a:f4:00:04 && eth.src==0a:58:0a:f4:00:05 && ip4.dst==10.96.6.91 && ip4.src==10.244.0.5 && ip.ttl==64 && tcp.dst==8080 && tcp.src==52888' --lb-dst 10.244.0.4:8080) ingress(dp="ovn-worker", inport="default_web-client-pod-5xh74") --------------------------------------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:5015): inport == "default_web-client-pod-5xh74" && eth.src == {0a:58:0a:f4:00:05}, priority 50, uuid 5935f131 next; 1. ls_in_port_sec_ip (ovn-northd.c:4674): inport == "default_web-client-pod-5xh74" && eth.src == 0a:58:0a:f4:00:05 && ip4.src == {10.244.0.5}, priority 90, uuid 97d649eb next; 5. ls_in_pre_acl (ovn-northd.c:5217): ip, priority 100, uuid 57ceb751 reg0[0] = 1; next; 6. ls_in_pre_lb (ovn-northd.c:5385): ip, priority 100, uuid 14c4aa4d reg0[0] = 1; next; 7. ls_in_pre_stateful (ovn-northd.c:5402): reg0[0] == 1, priority 100, uuid bb85fc82 ct_next; ct_next(ct_state=new|trk) ------------------------- 8. ls_in_acl_hint (ovn-northd.c:5443): ct.new && !ct.est, priority 7, uuid a6ef3200 reg0[7] = 1; reg0[9] = 1; next; 9. ls_in_acl (ovn-northd.c:5860): ip && (!ct.est || (ct.est && ct_label.blocked == 1)), priority 1, uuid 1240afe2 reg0[1] = 1; next; 13. ls_in_stateful (ovn-northd.c:6165): ct.new && ip4.dst == 10.96.6.91 && tcp.dst == 8080, priority 120, uuid 10ac6868 reg1 = 10.96.6.91; reg2[0..15] = 8080; ct_lb(backends=10.244.0.4:8080); ct_lb ----- 14. ls_in_pre_hairpin (ovn-northd.c:6269): ip && ct.trk, priority 100, uuid e77ecfe7 reg0[6] = chk_lb_hairpin(); reg0[12] = chk_lb_hairpin_reply(); *** chk_lb_hairpin_reply action not implemented next; 23. ls_in_l2_lkup (ovn-northd.c:7606): eth.dst == 0a:58:0a:f4:00:04, priority 50, uuid a65e4023 outport = "default_web-server-v4"; output; Additional info: Seems like the match on table7: ls_in_pre_stateful for this kind (pod2svc) of traffic has changed?
Created attachment 1799935 [details] ovn-nbdb
Looks like the NB/SB attached files are actually OVS conf.db databases. We'd need the OVN Northbound and Southbound databases instead (running in the ovnkube-master pod, nbdb/sbdb containers, likely in /etc/ovn/ovnnb_db.db, /etc/ovn/ovnsb_db.db). Thanks, Dumitru
My bad, I have recreated the scenario on a new cluster. Re-attaching the db's. Some more info: $ oc get pods -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client-on-ovn-worker 1/1 Running 0 3m28s 10.244.0.6 ovn-worker <none> <none> server-on-ovn-worker 1/1 Running 0 2m58s 10.244.0.7 ovn-worker <none> <none> $ oc get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ovn-control-plane Ready control-plane,master 5h36m v1.20.0 172.18.0.2 <none> Ubuntu Groovy Gorilla (development branch) 5.11.22-100.fc32.x86_64 containerd://1.4.0 ovn-worker Ready <none> 5h36m v1.20.0 172.18.0.4 <none> Ubuntu Groovy Gorilla (development branch) 5.11.22-100.fc32.x86_64 containerd://1.4.0 ovn-worker2 Ready <none> 5h36m v1.20.0 172.18.0.3 <none> Ubuntu Groovy Gorilla (development branch) 5.11.22-100.fc32.x86_64 containerd://1.4.0 $ oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h36m server-on-ovn-worker ClusterIP 10.96.174.37 <none> 80/TCP 53s sh-5.0# ovn-trace --ct new 'inport=="default_client-on-ovn-worker" && eth.src==0a:58:0a:f4:00:06 && eth.dst==0a:58:0a:f4:00:01 && ip4.src==10.244.0.6 && ip4.dst==10.96.174.37 && ip.ttl==64 && tcp && tcp.dst==80' --lb-dst 10.244.0.7:80 # tcp,reg14=0x6,vlan_tci=0x0000,dl_src=0a:58:0a:f4:00:06,dl_dst=0a:58:0a:f4:00:01,nw_src=10.244.0.6,nw_dst=10.96.174.37,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=80,tcp_flags=0 ingress(dp="ovn-worker", inport="default_client-on-ovn-worker") --------------------------------------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:4827): inport == "default_client-on-ovn-worker" && eth.src == {0a:58:0a:f4:00:06}, priority 50, uuid 57d50463 next; 1. ls_in_port_sec_ip (ovn-northd.c:4475): inport == "default_client-on-ovn-worker" && eth.src == 0a:58:0a:f4:00:06 && ip4.src == {10.244.0.6}, priority 90, uuid 50b41b14 next; 5. ls_in_pre_acl (ovn-northd.c:5077): ip, priority 100, uuid ab454b0f reg0[0] = 1; next; 6. ls_in_pre_lb (ovn-northd.c:5245): ip, priority 100, uuid 47be6105 reg0[2] = 1; next; 7. ls_in_pre_stateful (ovn-northd.c:5272): reg0[2] == 1 && ip4 && tcp, priority 120, uuid 1c5f89e6 reg1 = ip4.dst; reg2[0..15] = tcp.dst; ct_lb; ct_lb ----- 8. ls_in_acl_hint (ovn-northd.c:5363): !ct.trk, priority 5, uuid f05025b9 reg0[8] = 1; reg0[9] = 1; next; 22. ls_in_l2_lkup (ovn-northd.c:7471): eth.dst == 0a:58:0a:f4:00:01, priority 50, uuid 0c8bf241 outport = "stor-ovn-worker"; output; egress(dp="ovn-worker", inport="default_client-on-ovn-worker", outport="stor-ovn-worker") ----------------------------------------------------------------------------------------- 0. ls_out_pre_lb (ovn-northd.c:4973): ip && outport == "stor-ovn-worker", priority 110, uuid 47334b4e next; 1. ls_out_pre_acl (ovn-northd.c:4973): ip && outport == "stor-ovn-worker", priority 110, uuid d890d0cd next; 3. ls_out_acl_hint (ovn-northd.c:5363): !ct.trk, priority 5, uuid 3e3b34ae reg0[8] = 1; reg0[9] = 1; next; 9. ls_out_port_sec_l2 (ovn-northd.c:4922): outport == "stor-ovn-worker", priority 50, uuid a1a31184 output; /* output to "stor-ovn-worker", type "patch" */ ingress(dp="ovn_cluster_router", inport="rtos-ovn-worker") ---------------------------------------------------------- 0. lr_in_admission (ovn-northd.c:9541): eth.dst == 0a:58:0a:f4:00:01 && inport == "rtos-ovn-worker", priority 50, uuid 93e6a59e xreg0[0..47] = 0a:58:0a:f4:00:01; next; 1. lr_in_lookup_neighbor (ovn-northd.c:9621): 1, priority 0, uuid 4b1b2d99 reg9[2] = 1; next; 2. lr_in_learn_neighbor (ovn-northd.c:9630): reg9[2] == 1, priority 100, uuid 6a92cd55 next; 10. lr_in_ip_routing (ovn-northd.c:8586): ip4.dst == 10.244.0.0/24, priority 49, uuid 7061d0f3 ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 10.244.0.1; eth.src = 0a:58:0a:f4:00:01; outport = "rtos-ovn-worker"; flags.loopback = 1; next; 11. lr_in_ip_routing_ecmp (ovn-northd.c:9888): reg8[0..15] == 0, priority 150, uuid 46d08480 next; 12. lr_in_policy (ovn-northd.c:7917): ip4.src == 10.244.0.0/16 && ip4.dst == 10.244.0.0/16, priority 101, uuid b256ac9e reg8[0..15] = 0; next; 13. lr_in_policy_ecmp (ovn-northd.c:10015): reg8[0..15] == 0, priority 150, uuid fa75253b next; 14. lr_in_arp_resolve (ovn-northd.c:10191): outport == "rtos-ovn-worker" && reg0 == 10.244.0.7, priority 100, uuid bb7ca931 eth.dst = 0a:58:0a:f4:00:07; next; 18. lr_in_arp_request (ovn-northd.c:10639): 1, priority 0, uuid 4f5f9187 output; egress(dp="ovn_cluster_router", inport="rtos-ovn-worker", outport="rtos-ovn-worker") ------------------------------------------------------------------------------------ 3. lr_out_delivery (ovn-northd.c:10686): outport == "rtos-ovn-worker", priority 100, uuid 676d3764 output; /* output to "rtos-ovn-worker", type "patch" */ ingress(dp="ovn-worker", inport="stor-ovn-worker") -------------------------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:4827): inport == "stor-ovn-worker", priority 50, uuid c80ac971 next; 5. ls_in_pre_acl (ovn-northd.c:4970): ip && inport == "stor-ovn-worker", priority 110, uuid a15cb04e next; 6. ls_in_pre_lb (ovn-northd.c:4970): ip && inport == "stor-ovn-worker", priority 110, uuid 3083e138 next; 8. ls_in_acl_hint (ovn-northd.c:5363): !ct.trk, priority 5, uuid f05025b9 reg0[8] = 1; reg0[9] = 1; next; 22. ls_in_l2_lkup (ovn-northd.c:7471): eth.dst == 0a:58:0a:f4:00:07, priority 50, uuid ef1869fb outport = "default_server-on-ovn-worker"; output; egress(dp="ovn-worker", inport="stor-ovn-worker", outport="default_server-on-ovn-worker") ----------------------------------------------------------------------------------------- 0. ls_out_pre_lb (ovn-northd.c:5247): ip, priority 100, uuid 47086058 reg0[2] = 1; next; 1. ls_out_pre_acl (ovn-northd.c:5079): ip, priority 100, uuid a62be758 reg0[0] = 1; next; 2. ls_out_pre_stateful (ovn-northd.c:5292): reg0[2] == 1, priority 110, uuid 6766ee89 ct_lb; ct_lb ----- 3. ls_out_acl_hint (ovn-northd.c:5363): !ct.trk, priority 5, uuid 3e3b34ae reg0[8] = 1; reg0[9] = 1; next; 8. ls_out_port_sec_ip (ovn-northd.c:4475): outport == "default_server-on-ovn-worker" && eth.dst == 0a:58:0a:f4:00:07 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.244.0.7}, priority 90, uuid 4d25c5e0 next; 9. ls_out_port_sec_l2 (ovn-northd.c:4922): outport == "default_server-on-ovn-worker" && eth.dst == {0a:58:0a:f4:00:07}, priority 50, uuid e37128ae output; /* output to "default_server-on-ovn-worker", type "" */ sh-5.0# sh-5.0# ovn-nbctl show ovn-worker switch ac11954d-4a58-4872-af54-ec9e15fae048 (ovn-worker) port stor-ovn-worker type: router addresses: ["0a:58:0a:f4:00:01"] router-port: rtos-ovn-worker port k8s-ovn-worker addresses: ["6a:c3:85:31:c3:fa 10.244.0.2"] port kube-system_coredns-74ff55c5b-ckw7h addresses: ["0a:58:0a:f4:00:05 10.244.0.5"] port int-ovn-worker addresses: ["0a:58:0a:f4:00:03"] port default_server-on-ovn-worker addresses: ["0a:58:0a:f4:00:07 10.244.0.7"] port local-path-storage_local-path-provisioner-78776bfc44-lqbxq addresses: ["0a:58:0a:f4:00:04 10.244.0.4"] port default_client-on-ovn-worker addresses: ["0a:58:0a:f4:00:06 10.244.0.6"]
upstream patch: https://patchwork.ozlabs.org/project/ovn/patch/e41f5799af7d1f84c804830184c6f05c790c7965.1638393565.git.lorenzo.bianconi@redhat.com/
topo: vm1-----s2------r1-------s3-----vm2 create a lb for s2 and s3,an acl with allow-related . reproduced on version: # rpm -qa|grep ovn ovn-2021-21.09.0-12.el8fdp.x86_64 ovn-2021-host-21.09.0-12.el8fdp.x86_64 ovn-2021-central-21.09.0-12.el8fdp.x86_64 # ovn-trace --ct new 'inport=="hv1_vm00_vnet1" && eth.src==00:de:ad:01:01:01 && eth.dst==00:de:ad:01:00:01 && tcp && ip4.src==172.16.103.11 && ip4.dst==172.16.102.11 && ip.ttl==64' --lb-dst 172.16.102.100 # ovn-trace --ct new 'inport=="hv1_vm00_vnet1" && eth.src==00:de:ad:01:01:01 && eth.dst==00:de:ad:01:00:01 && tcp && ip4.src==172.16.103.11 && ip4.dst==172.16.102.11 && ip.ttl==64' --lb-dst 172.16.102.100 # tcp,reg14=0x2,vlan_tci=0x0000,dl_src=00:de:ad:01:01:01,dl_dst=00:de:ad:01:00:01,nw_src=172.16.103.11,nw_dst=172.16.102.11,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=0,tcp_flags=0 ingress(dp="s2", inport="hv1_vm00_vnet1") ----------------------------------------- 0. ls_in_port_sec_l2 (northd.c:5228): inport == "hv1_vm00_vnet1", priority 50, uuid 50a55725 next; 5. ls_in_pre_acl (northd.c:5485): ip, priority 100, uuid 6595c1bb reg0[0] = 1; next; 6. ls_in_pre_lb (northd.c:5637): ip, priority 100, uuid d8ed668b reg0[2] = 1; next; 7. ls_in_pre_stateful (northd.c:5666): reg0[2] == 1 && ip4 && tcp, priority 120, uuid 89a09ea9 reg1 = ip4.dst; reg2[0..15] = tcp.dst; ct_lb; ct_lb ----- 8. ls_in_acl_hint (northd.c:5759): !ct.trk, priority 5, uuid f39ef5de reg0[8] = 1; reg0[9] = 1; next; 22. ls_in_l2_lkup (northd.c:7987): eth.dst == 00:de:ad:01:00:01, priority 50, uuid 73bf1d49 outport = "hv1_vm00_vnet1"; output; verified on version: # rpm -qa|grep ovn ovn-2021-central-21.12.0-11.el8fdp.x86_64 ovn-2021-host-21.12.0-11.el8fdp.x86_64 ovn-2021-21.12.0-11.el8fdp.x86_64 # ovn-trace --ct new 'inport=="hv1_vm00_vnet1" && eth.src==00:de:ad:01:01:01 && eth.dst==00:de:ad:01:00:01 && tcp && ip4.src==172.16.103.11 && ip4.dst==172.16.102.11 && ip.ttl==64' --lb-dst 172.16.102.100 # tcp,reg14=0x2,vlan_tci=0x0000,dl_src=00:de:ad:01:01:01,dl_dst=00:de:ad:01:00:01,nw_src=172.16.103.11,nw_dst=172.16.102.11,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=0,tcp_flags=0 ingress(dp="s2", inport="hv1_vm00_vnet1") ----------------------------------------- 0. ls_in_port_sec_l2 (northd.c:5493): inport == "hv1_vm00_vnet1", priority 50, uuid bf0c0563 next; 5. ls_in_pre_acl (northd.c:5753): ip, priority 100, uuid 79bf660b reg0[0] = 1; next; 6. ls_in_pre_lb (northd.c:5885): ip, priority 100, uuid fd121cd0 reg0[2] = 1; next; 7. ls_in_pre_stateful (northd.c:5912): reg0[2] == 1 && ip4 && tcp, priority 120, uuid f52f27ad reg1 = ip4.dst; reg2[0..15] = tcp.dst; ct_lb; ct_lb ----- 8. ls_in_acl_hint (northd.c:5985): ct.new && !ct.est, priority 7, uuid decfdebb reg0[7] = 1; reg0[9] = 1; next; 9. ls_in_acl (northd.c:6432): ip && (!ct.est || (ct.est && ct_label.blocked == 1)), priority 1, uuid eb3f40e1 reg0[1] = 1; next; 12. ls_in_stateful (northd.c:6736): ct.new && ip4.dst == 172.16.102.100, priority 110, uuid d9a04360 reg1 = 172.16.102.100; ct_lb(backends=172.16.102.11); ct_lb /* default (use --ct to customize) */ ------------------------------------------- 13. ls_in_pre_hairpin (northd.c:6800): ip && ct.trk, priority 100, uuid 5e9047e5 reg0[6] = chk_lb_hairpin(); reg0[12] = chk_lb_hairpin_reply(); *** chk_lb_hairpin_reply action not implemented next; 22. ls_in_l2_lkup (northd.c:8218): eth.dst == 00:de:ad:01:00:01, priority 50, uuid 34e97b50 outport = "hv1_vm00_vnet1"; output;
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0674