Bug 1980702

Summary: [ovn] [ovn-trace] Does not show the load balancer backends getting hit
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Surya Seetharaman <surya>
Component: OVNAssignee: lorenzo bianconi <lorenzo.bianconi>
Status: CLOSED ERRATA QA Contact: ying xu <yinxu>
Severity: unspecified Docs Contact:
Priority: medium    
Version: FDP 21.CCC: ctrautma, dceara, jiji, lorenzo.bianconi, mmichels, mori, pliu
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn21.12-21.12.0-25.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-02-24 17:47:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ovn-sbdb none

Description Surya Seetharaman 2021-07-09 10:21:05 UTC 
Created attachment 1799934 [details]
ovn-sbdb

Created attachment 1799934 [details]
ovn-sbdb

Description of problem:

ovn-trace does not show table 13 getting hit and ct_lb(backends=10.244.0.4:8080).


Version-Release number of selected component (if applicable):

sh-5.0# ovn-nbctl --version
ovn-nbctl 21.03.1
Open vSwitch Library 2.15.90
DB Schema 5.31.0


How reproducible:
Always

Steps to Reproduce:
1. Create a client pod and clusterIP svc (any object that will need the lb dnat)
2. curling from client to service works which means the load balancing and dnating work fine.
3. Run the ovn-trace: "ovn-trace --ct new 'inport=="default_client-on-ovn-worker" && eth.src==0a:58:0a:f4:00:09 && eth.dst==0a:58:0a:f4:00:01 && tcp && tcp.src==80 && tcp.dst==80 && ip4.src==10.244.0.9 && ip4.dst==10.96.245.163 && ip.ttl==64' --lb-dst 10.244.0.10:80"

Actual results:

Trace shows:

ingress(dp="ovn-worker", inport="default_client-on-ovn-worker")
---------------------------------------------------------------
 0. ls_in_port_sec_l2 (ovn-northd.c:4802): inport == "default_client-on-ovn-worker" && eth.src == {0a:58:0a:f4:00:09}, priority 50, uuid 308b2fd0
    next;
 1. ls_in_port_sec_ip (ovn-northd.c:4450): inport == "default_client-on-ovn-worker" && eth.src == 0a:58:0a:f4:00:09 && ip4.src == {10.244.0.9}, priority 90, uuid 0134ca82
    next;
 5. ls_in_pre_acl (ovn-northd.c:5004): ip, priority 100, uuid a27e108e
    reg0[0] = 1;
    next;
 6. ls_in_pre_lb (ovn-northd.c:5172): ip, priority 100, uuid ac3d55f5
    reg0[2] = 1;
    next;
 7. ls_in_pre_stateful (ovn-northd.c:5199): reg0[2] == 1 && ip4 && tcp, priority 120, uuid dcf60d09
    reg1 = ip4.dst;
    reg2[0..15] = tcp.dst;
    ct_lb;

ct_lb
-----
 8. ls_in_acl_hint (ovn-northd.c:5290): !ct.trk, priority 5, uuid 71d49e33
    reg0[8] = 1;
    reg0[9] = 1;
    next;
22. ls_in_l2_lkup (ovn-northd.c:7374): eth.dst == 0a:58:0a:f4:00:01, priority 50, uuid d75fc9d6
    outport = "stor-ovn-worker";
    output;

on the ingress pipeline of ovn logical switch

Expected results:

Trace should show packet matching on flow with prio 120 in table 13:
example snippet from a trace on older ovn-trace versions:

old-trace (ovn-trace  --db tcp:172.18.0.3:6642  ovn-worker --ct=new  'inport=="default_web-client-pod-5xh74" && eth.dst==0a:58:0a:f4:00:04 && eth.src==0a:58:0a:f4:00:05 && ip4.dst==10.96.6.91 && ip4.src==10.244.0.5 && ip.ttl==64 && tcp.dst==8080 && tcp.src==52888' --lb-dst 10.244.0.4:8080)

ingress(dp="ovn-worker", inport="default_web-client-pod-5xh74")
---------------------------------------------------------------
 0. ls_in_port_sec_l2 (ovn-northd.c:5015): inport == "default_web-client-pod-5xh74" && eth.src == {0a:58:0a:f4:00:05}, priority 50, uuid 5935f131
    next;
 1. ls_in_port_sec_ip (ovn-northd.c:4674): inport == "default_web-client-pod-5xh74" && eth.src == 0a:58:0a:f4:00:05 && ip4.src == {10.244.0.5}, priority 90, uuid 97d649eb
    next;
 5. ls_in_pre_acl (ovn-northd.c:5217): ip, priority 100, uuid 57ceb751
    reg0[0] = 1;
    next;
 6. ls_in_pre_lb (ovn-northd.c:5385): ip, priority 100, uuid 14c4aa4d
    reg0[0] = 1;
    next;
 7. ls_in_pre_stateful (ovn-northd.c:5402): reg0[0] == 1, priority 100, uuid bb85fc82
    ct_next;
ct_next(ct_state=new|trk)
-------------------------
 8. ls_in_acl_hint (ovn-northd.c:5443): ct.new && !ct.est, priority 7, uuid a6ef3200
    reg0[7] = 1;
    reg0[9] = 1;
    next;
 9. ls_in_acl (ovn-northd.c:5860): ip && (!ct.est || (ct.est && ct_label.blocked == 1)), priority 1, uuid 1240afe2
    reg0[1] = 1;
    next;
13. ls_in_stateful (ovn-northd.c:6165): ct.new && ip4.dst == 10.96.6.91 && tcp.dst == 8080, priority 120, uuid 10ac6868
    reg1 = 10.96.6.91;
    reg2[0..15] = 8080;
    ct_lb(backends=10.244.0.4:8080);
ct_lb
-----
14. ls_in_pre_hairpin (ovn-northd.c:6269): ip && ct.trk, priority 100, uuid e77ecfe7
    reg0[6] = chk_lb_hairpin();
    reg0[12] = chk_lb_hairpin_reply();
    *** chk_lb_hairpin_reply action not implemented
    next;
23. ls_in_l2_lkup (ovn-northd.c:7606): eth.dst == 0a:58:0a:f4:00:04, priority 50, uuid a65e4023
    outport = "default_web-server-v4";
    output;


Additional info:

Seems like the match on table7: ls_in_pre_stateful for this kind (pod2svc) of traffic has changed?
Comment 1 Surya Seetharaman 2021-07-09 10:22:00 UTC 
Created attachment 1799935 [details]
ovn-nbdb
Comment 3 Dumitru Ceara 2021-07-27 18:55:39 UTC 
Looks like the NB/SB attached files are actually OVS conf.db databases.  We'd need the OVN Northbound and Southbound databases instead (running in the ovnkube-master pod, nbdb/sbdb containers, likely in /etc/ovn/ovnnb_db.db, /etc/ovn/ovnsb_db.db).

Thanks,
Dumitru
Comment 4 Surya Seetharaman 2021-07-28 17:32:01 UTC 
My bad, I have recreated the scenario on a new cluster. Re-attaching the db's. Some more info:

$ oc get pods -owide
NAME                   READY   STATUS    RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
client-on-ovn-worker   1/1     Running   0          3m28s   10.244.0.6   ovn-worker   <none>           <none>
server-on-ovn-worker   1/1     Running   0          2m58s   10.244.0.7   ovn-worker   <none>           <none>

$ oc get nodes -owide
NAME                STATUS   ROLES                  AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                     KERNEL-VERSION            CONTAINER-RUNTIME
ovn-control-plane   Ready    control-plane,master   5h36m   v1.20.0   172.18.0.2    <none>        Ubuntu Groovy Gorilla (development branch)   5.11.22-100.fc32.x86_64   containerd://1.4.0
ovn-worker          Ready    <none>                 5h36m   v1.20.0   172.18.0.4    <none>        Ubuntu Groovy Gorilla (development branch)   5.11.22-100.fc32.x86_64   containerd://1.4.0
ovn-worker2         Ready    <none>                 5h36m   v1.20.0   172.18.0.3    <none>        Ubuntu Groovy Gorilla (development branch)   5.11.22-100.fc32.x86_64   containerd://1.4.0

$ oc get svc 
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes             ClusterIP   10.96.0.1      <none>        443/TCP   5h36m
server-on-ovn-worker   ClusterIP   10.96.174.37   <none>        80/TCP    53s


sh-5.0# ovn-trace --ct new 'inport=="default_client-on-ovn-worker" && eth.src==0a:58:0a:f4:00:06 && eth.dst==0a:58:0a:f4:00:01 && ip4.src==10.244.0.6 && ip4.dst==10.96.174.37 && ip.ttl==64 && tcp && tcp.dst==80' --lb-dst 10.244.0.7:80
# tcp,reg14=0x6,vlan_tci=0x0000,dl_src=0a:58:0a:f4:00:06,dl_dst=0a:58:0a:f4:00:01,nw_src=10.244.0.6,nw_dst=10.96.174.37,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=80,tcp_flags=0

ingress(dp="ovn-worker", inport="default_client-on-ovn-worker")
---------------------------------------------------------------
 0. ls_in_port_sec_l2 (ovn-northd.c:4827): inport == "default_client-on-ovn-worker" && eth.src == {0a:58:0a:f4:00:06}, priority 50, uuid 57d50463
    next;
 1. ls_in_port_sec_ip (ovn-northd.c:4475): inport == "default_client-on-ovn-worker" && eth.src == 0a:58:0a:f4:00:06 && ip4.src == {10.244.0.6}, priority 90, uuid 50b41b14
    next;
 5. ls_in_pre_acl (ovn-northd.c:5077): ip, priority 100, uuid ab454b0f
    reg0[0] = 1;
    next;
 6. ls_in_pre_lb (ovn-northd.c:5245): ip, priority 100, uuid 47be6105
    reg0[2] = 1;
    next;
 7. ls_in_pre_stateful (ovn-northd.c:5272): reg0[2] == 1 && ip4 && tcp, priority 120, uuid 1c5f89e6
    reg1 = ip4.dst;
    reg2[0..15] = tcp.dst;
    ct_lb;

ct_lb
-----
 8. ls_in_acl_hint (ovn-northd.c:5363): !ct.trk, priority 5, uuid f05025b9
    reg0[8] = 1;
    reg0[9] = 1;
    next;
22. ls_in_l2_lkup (ovn-northd.c:7471): eth.dst == 0a:58:0a:f4:00:01, priority 50, uuid 0c8bf241
    outport = "stor-ovn-worker";
    output;

egress(dp="ovn-worker", inport="default_client-on-ovn-worker", outport="stor-ovn-worker")
-----------------------------------------------------------------------------------------
 0. ls_out_pre_lb (ovn-northd.c:4973): ip && outport == "stor-ovn-worker", priority 110, uuid 47334b4e
    next;
 1. ls_out_pre_acl (ovn-northd.c:4973): ip && outport == "stor-ovn-worker", priority 110, uuid d890d0cd
    next;
 3. ls_out_acl_hint (ovn-northd.c:5363): !ct.trk, priority 5, uuid 3e3b34ae
    reg0[8] = 1;
    reg0[9] = 1;
    next;
 9. ls_out_port_sec_l2 (ovn-northd.c:4922): outport == "stor-ovn-worker", priority 50, uuid a1a31184
    output;
    /* output to "stor-ovn-worker", type "patch" */

ingress(dp="ovn_cluster_router", inport="rtos-ovn-worker")
----------------------------------------------------------
 0. lr_in_admission (ovn-northd.c:9541): eth.dst == 0a:58:0a:f4:00:01 && inport == "rtos-ovn-worker", priority 50, uuid 93e6a59e
    xreg0[0..47] = 0a:58:0a:f4:00:01;
    next;
 1. lr_in_lookup_neighbor (ovn-northd.c:9621): 1, priority 0, uuid 4b1b2d99
    reg9[2] = 1;
    next;
 2. lr_in_learn_neighbor (ovn-northd.c:9630): reg9[2] == 1, priority 100, uuid 6a92cd55
    next;
10. lr_in_ip_routing (ovn-northd.c:8586): ip4.dst == 10.244.0.0/24, priority 49, uuid 7061d0f3
    ip.ttl--;
    reg8[0..15] = 0;
    reg0 = ip4.dst;
    reg1 = 10.244.0.1;
    eth.src = 0a:58:0a:f4:00:01;
    outport = "rtos-ovn-worker";
    flags.loopback = 1;
    next;
11. lr_in_ip_routing_ecmp (ovn-northd.c:9888): reg8[0..15] == 0, priority 150, uuid 46d08480
    next;
12. lr_in_policy (ovn-northd.c:7917): ip4.src == 10.244.0.0/16 && ip4.dst == 10.244.0.0/16, priority 101, uuid b256ac9e
    reg8[0..15] = 0;
    next;
13. lr_in_policy_ecmp (ovn-northd.c:10015): reg8[0..15] == 0, priority 150, uuid fa75253b
    next;
14. lr_in_arp_resolve (ovn-northd.c:10191): outport == "rtos-ovn-worker" && reg0 == 10.244.0.7, priority 100, uuid bb7ca931
    eth.dst = 0a:58:0a:f4:00:07;
    next;
18. lr_in_arp_request (ovn-northd.c:10639): 1, priority 0, uuid 4f5f9187
    output;

egress(dp="ovn_cluster_router", inport="rtos-ovn-worker", outport="rtos-ovn-worker")
------------------------------------------------------------------------------------
 3. lr_out_delivery (ovn-northd.c:10686): outport == "rtos-ovn-worker", priority 100, uuid 676d3764
    output;
    /* output to "rtos-ovn-worker", type "patch" */

ingress(dp="ovn-worker", inport="stor-ovn-worker")
--------------------------------------------------
 0. ls_in_port_sec_l2 (ovn-northd.c:4827): inport == "stor-ovn-worker", priority 50, uuid c80ac971
    next;
 5. ls_in_pre_acl (ovn-northd.c:4970): ip && inport == "stor-ovn-worker", priority 110, uuid a15cb04e
    next;
 6. ls_in_pre_lb (ovn-northd.c:4970): ip && inport == "stor-ovn-worker", priority 110, uuid 3083e138
    next;
 8. ls_in_acl_hint (ovn-northd.c:5363): !ct.trk, priority 5, uuid f05025b9
    reg0[8] = 1;
    reg0[9] = 1;
    next;
22. ls_in_l2_lkup (ovn-northd.c:7471): eth.dst == 0a:58:0a:f4:00:07, priority 50, uuid ef1869fb
    outport = "default_server-on-ovn-worker";
    output;

egress(dp="ovn-worker", inport="stor-ovn-worker", outport="default_server-on-ovn-worker")
-----------------------------------------------------------------------------------------
 0. ls_out_pre_lb (ovn-northd.c:5247): ip, priority 100, uuid 47086058
    reg0[2] = 1;
    next;
 1. ls_out_pre_acl (ovn-northd.c:5079): ip, priority 100, uuid a62be758
    reg0[0] = 1;
    next;
 2. ls_out_pre_stateful (ovn-northd.c:5292): reg0[2] == 1, priority 110, uuid 6766ee89
    ct_lb;

ct_lb
-----
 3. ls_out_acl_hint (ovn-northd.c:5363): !ct.trk, priority 5, uuid 3e3b34ae
    reg0[8] = 1;
    reg0[9] = 1;
    next;
 8. ls_out_port_sec_ip (ovn-northd.c:4475): outport == "default_server-on-ovn-worker" && eth.dst == 0a:58:0a:f4:00:07 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.244.0.7}, priority 90, uuid 4d25c5e0
    next;
 9. ls_out_port_sec_l2 (ovn-northd.c:4922): outport == "default_server-on-ovn-worker" && eth.dst == {0a:58:0a:f4:00:07}, priority 50, uuid e37128ae
    output;
    /* output to "default_server-on-ovn-worker", type "" */
sh-5.0# 


sh-5.0# ovn-nbctl show ovn-worker
switch ac11954d-4a58-4872-af54-ec9e15fae048 (ovn-worker)
    port stor-ovn-worker
        type: router
        addresses: ["0a:58:0a:f4:00:01"]
        router-port: rtos-ovn-worker
    port k8s-ovn-worker
        addresses: ["6a:c3:85:31:c3:fa 10.244.0.2"]
    port kube-system_coredns-74ff55c5b-ckw7h
        addresses: ["0a:58:0a:f4:00:05 10.244.0.5"]
    port int-ovn-worker
        addresses: ["0a:58:0a:f4:00:03"]
    port default_server-on-ovn-worker
        addresses: ["0a:58:0a:f4:00:07 10.244.0.7"]
    port local-path-storage_local-path-provisioner-78776bfc44-lqbxq
        addresses: ["0a:58:0a:f4:00:04 10.244.0.4"]
    port default_client-on-ovn-worker
        addresses: ["0a:58:0a:f4:00:06 10.244.0.6"]
Comment 8 lorenzo bianconi 2021-12-01 22:19:56 UTC 
upstream patch: https://patchwork.ozlabs.org/project/ovn/patch/e41f5799af7d1f84c804830184c6f05c790c7965.1638393565.git.lorenzo.bianconi@redhat.com/
Comment 12 ying xu 2022-01-26 09:01:37 UTC 
topo:

vm1-----s2------r1-------s3-----vm2

create a lb for s2 and s3,an acl with allow-related .

reproduced on version:
# rpm -qa|grep ovn
ovn-2021-21.09.0-12.el8fdp.x86_64
ovn-2021-host-21.09.0-12.el8fdp.x86_64
ovn-2021-central-21.09.0-12.el8fdp.x86_64

# ovn-trace --ct new 'inport=="hv1_vm00_vnet1" && eth.src==00:de:ad:01:01:01 && eth.dst==00:de:ad:01:00:01 && tcp && ip4.src==172.16.103.11 && ip4.dst==172.16.102.11 && ip.ttl==64' --lb-dst 172.16.102.100
# ovn-trace --ct new 'inport=="hv1_vm00_vnet1" && eth.src==00:de:ad:01:01:01 && eth.dst==00:de:ad:01:00:01 && tcp && ip4.src==172.16.103.11 && ip4.dst==172.16.102.11 && ip.ttl==64' --lb-dst 172.16.102.100
# tcp,reg14=0x2,vlan_tci=0x0000,dl_src=00:de:ad:01:01:01,dl_dst=00:de:ad:01:00:01,nw_src=172.16.103.11,nw_dst=172.16.102.11,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=0,tcp_flags=0

ingress(dp="s2", inport="hv1_vm00_vnet1")
-----------------------------------------
 0. ls_in_port_sec_l2 (northd.c:5228): inport == "hv1_vm00_vnet1", priority 50, uuid 50a55725
    next;
 5. ls_in_pre_acl (northd.c:5485): ip, priority 100, uuid 6595c1bb
    reg0[0] = 1;
    next;
 6. ls_in_pre_lb (northd.c:5637): ip, priority 100, uuid d8ed668b
    reg0[2] = 1;
    next;
 7. ls_in_pre_stateful (northd.c:5666): reg0[2] == 1 && ip4 && tcp, priority 120, uuid 89a09ea9
    reg1 = ip4.dst;
    reg2[0..15] = tcp.dst;
    ct_lb;

ct_lb
-----
 8. ls_in_acl_hint (northd.c:5759): !ct.trk, priority 5, uuid f39ef5de
    reg0[8] = 1;
    reg0[9] = 1;
    next;
22. ls_in_l2_lkup (northd.c:7987): eth.dst == 00:de:ad:01:00:01, priority 50, uuid 73bf1d49
    outport = "hv1_vm00_vnet1";
    output;


verified on version:
# rpm  -qa|grep ovn
ovn-2021-central-21.12.0-11.el8fdp.x86_64
ovn-2021-host-21.12.0-11.el8fdp.x86_64
ovn-2021-21.12.0-11.el8fdp.x86_64

# ovn-trace --ct new 'inport=="hv1_vm00_vnet1" && eth.src==00:de:ad:01:01:01 && eth.dst==00:de:ad:01:00:01 && tcp && ip4.src==172.16.103.11 && ip4.dst==172.16.102.11 && ip.ttl==64' --lb-dst 172.16.102.100
# tcp,reg14=0x2,vlan_tci=0x0000,dl_src=00:de:ad:01:01:01,dl_dst=00:de:ad:01:00:01,nw_src=172.16.103.11,nw_dst=172.16.102.11,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=0,tcp_flags=0

ingress(dp="s2", inport="hv1_vm00_vnet1")
-----------------------------------------
 0. ls_in_port_sec_l2 (northd.c:5493): inport == "hv1_vm00_vnet1", priority 50, uuid bf0c0563
    next;
 5. ls_in_pre_acl (northd.c:5753): ip, priority 100, uuid 79bf660b
    reg0[0] = 1;
    next;
 6. ls_in_pre_lb (northd.c:5885): ip, priority 100, uuid fd121cd0
    reg0[2] = 1;
    next;
 7. ls_in_pre_stateful (northd.c:5912): reg0[2] == 1 && ip4 && tcp, priority 120, uuid f52f27ad
    reg1 = ip4.dst;
    reg2[0..15] = tcp.dst;
    ct_lb;

ct_lb
-----
 8. ls_in_acl_hint (northd.c:5985): ct.new && !ct.est, priority 7, uuid decfdebb
    reg0[7] = 1;
    reg0[9] = 1;
    next;
 9. ls_in_acl (northd.c:6432): ip && (!ct.est || (ct.est && ct_label.blocked == 1)), priority 1, uuid eb3f40e1
    reg0[1] = 1;
    next;
12. ls_in_stateful (northd.c:6736): ct.new && ip4.dst == 172.16.102.100, priority 110, uuid d9a04360
    reg1 = 172.16.102.100;
    ct_lb(backends=172.16.102.11);

ct_lb /* default (use --ct to customize) */
-------------------------------------------
13. ls_in_pre_hairpin (northd.c:6800): ip && ct.trk, priority 100, uuid 5e9047e5
    reg0[6] = chk_lb_hairpin();
    reg0[12] = chk_lb_hairpin_reply();
    *** chk_lb_hairpin_reply action not implemented
    next;
22. ls_in_l2_lkup (northd.c:8218): eth.dst == 00:de:ad:01:00:01, priority 50, uuid 34e97b50
    outport = "hv1_vm00_vnet1";
    output;
Comment 14 errata-xmlrpc 2022-02-24 17:47:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0674