Description of problem: ovn-trace fails with the following error, when tracing a packet to a load balancer with empty backends 2020-12-18T19:00:00Z|00030|ovntrace|WARN|trigger_event(event = "empty_lb_backends", meter = "", vip = "172.30.134.205:80", protocol = "tcp", load_balancer = "02660873-b845-4242-b130-b0a7104f45f4");: parsing actions failed (Syntax error at `vip' expecting empty_lb_backends option name.) Steps to Reproduce: 1. Create a load balancer with a VIP and no backends 2. Trace a packet to the VIP It happens with ovn-kubernetes, both upstream and downstream
I have a coredump if I reproduce it using KIND and ovn-kubernetes upstream code. These are the stesp: 1. from the ovn-kubernetes repo https://github.com/ovn-org/ovn-kubernetes 2. cd contrib 3. Create a cluster: ./kind.sh 4. Once it is created create 1 pod and expose it with a service kubectl run test --image=httpd --port 80 kubectl expose pod/test kubectl get services -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h43m <none> test ClusterIP 10.96.57.175 <none> 80/TCP 12m run=test the ovn loadbalancer is configured _uuid : 079c5238-e32a-47a5-ac44-08c4b2a3a66e external_ids : {k8s-cluster-lb-tcp=yes} health_check : [] ip_port_mappings : {} name : "" protocol : tcp selection_fields : [] vips : {"10.96.0.10:53"="10.244.1.3:53,10.244.2.3:53", "10.96.0.10:9153"="10.244.1.3:9153,10.244.2.3:9153", "10.96. 0.1:443"="172.19.0.6:6443", "10.96.57.175:80"="10.244.2.6:80"} 5. Create another pod to test the service kubectl run testpod --image=httpd --port 80 6. Using nbctl we can get the testpod details for the trace: switch 83ff7243-3bd0-4335-a50e-be445e2d7303 (ovn-worker) port default_testpod addresses: ["0a:58:0a:f4:02:05 10.244.2.5"] port local-path-storage_local-path-provisioner-78776bfc44-k7mtd addresses: ["0a:58:0a:f4:02:04 10.244.2.4"] port k8s-ovn-worker 7. and the trace works ovn-trace 83ff7243-3bd0-4335-a50e-be445e2d7303 'inport == "default_testpod" && eth.src == 0a:58:0a:f4:02:05 && ip4.src==10.244.2.5 && ip4.dst==10.96.57.175 && tcp && tcp.dst==80' # tcp,reg14=0x5,vlan_tci=0x0000,dl_src=0a:58:0a:f4:02:05,dl_dst=00:00:00:00:00:00,nw_src=10.244.2.5,nw_dst=10.96.57.175,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=80,tcp_flags=0 ingress(dp="ovn-worker", inport="default_testpod") -------------------------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:4834): inport == "default_testpod" && eth.src == {0a:58:0a:f4:02:05}, priority 50, uuid a821cf82 next; 1. ls_in_port_sec_ip (ovn-northd.c:4490): inport == "default_testpod" && eth.src == 0a:58:0a:f4:02:05 && ip4.src == {10.244.2.5}, priority 90, uuid dd801c92 next; 3. ls_in_pre_acl (ovn-northd.c:5016): ip, priority 100, uuid bfb729d1 reg0[0] = 1; next; 4. ls_in_pre_lb (ovn-northd.c:5214): ip, priority 100, uuid 07d97413 reg0[0] = 1; next; 5. ls_in_pre_stateful (ovn-northd.c:5231): reg0[0] == 1, priority 100, uuid fa43a719 ct_next; ct_next(ct_state=est|trk /* default (use --ct to customize) */) --------------------------------------------------------------- 6. ls_in_acl_hint (ovn-northd.c:5298): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid 4ba8c00e reg0[8] = 1; reg0[10] = 1; next; 20. ls_in_l2_lkup: no match (implicit drop) 8. we delete the backend of the service, so the OVN loadbalancer doesn´t have backends kubectl delete pod test and the ovn loadbalancer doesn´t have backends for that vip _uuid : 079c5238-e32a-47a5-ac44-08c4b2a3a66e external_ids : {k8s-cluster-lb-tcp=yes} health_check : [] ip_port_mappings : {} name : "" protocol : tcp selection_fields : [] vips : {"10.96.0.10:53"="10.244.1.3:53,10.244.2.3:53", "10.96.0.10:9153"="10.244.1.3:9153,10.244.2.3:9153", "10.96. 0.1:443"="172.19.0.6:6443", "10.96.57.175:80"=""} 9. the trace fails and core dumps ovn-trace 83ff7243-3bd0-4335-a50e-be445e2d7303 'inport == "default_testpod" && eth.src == 0a:58:0a: ovn-trace: utilities/ovn-trace.c:1891: assertion next->pipeline == OVNACT_P_INGRESS failed in execute_next() Aborted (core dumped)
fixes posted upstream: - https://mail.openvswitch.org/pipermail/ovs-dev/2020-December/379014.html - https://mail.openvswitch.org/pipermail/ovs-dev/2020-December/379016.html
reproduced this bug on the version: # rpm -qa|grep ovn ovn2.13-20.12.0-1.el8fdp.x86_64 ovn2.13-central-20.12.0-1.el8fdp.x86_64 ovn2.13-host-20.12.0-1.el8fdp.x86_64 # ovn-nbctl show switch 11a5fba8-0450-43b4-a6fd-506833dbc848 (s2) port vm10 addresses: ["00:de:ad:01:00:01 172.16.102.11 2001:db8:102::11"] port s2_r1 type: router addresses: ["00:de:ad:ff:01:02 172.16.102.1 2001:db8:102::1"] router-port: r1_s2 port vm11 addresses: ["00:de:ad:01:01:01 172.16.102.12 2001:db8:102::12"] # ovn-nbctl list load_balancer _uuid : 82a86a8b-5eb5-4ba2-9fbe-8d17c2590416 external_ids : {} health_check : [] ip_port_mappings : {} name : lb1 options : {} protocol : tcp selection_fields : [] vips : {"172.16.103.10:8000"="", "[2001:db8:103::10]:8000"=""} # ovn-trace s2 "inport==\"vm10\" && eth.src==00:de:ad:01:00:01 && ip4.src==172.16.102.11 && ip4.dst==172.16.103.10 && ip.ttl==64 && tcp && tcp.dst==8000" 2021-01-29T02:21:25Z|00001|ovntrace|WARN|trigger_event(event = "empty_lb_backends", meter = "", vip = "172.16.103.10:8000", protocol = "tcp", load_balancer = "82a86a8b-5eb5-4ba2-9fbe-8d17c2590416");: parsing actions failed (Syntax error at `vip' expecting empty_lb_backends option name.) 2021-01-29T02:21:25Z|00002|ovntrace|WARN|trigger_event(event = "empty_lb_backends", meter = "", vip = "[2001:db8:103::10]:8000", protocol = "tcp", load_balancer = "82a86a8b-5eb5-4ba2-9fbe-8d17c2590416");: parsing actions failed (Syntax error at `vip' expecting empty_lb_backends option name.) 2021-01-29T02:21:25Z|00003|ovntrace|WARN|trigger_event(event = "empty_lb_backends", meter = "", vip = "[2001:db8:103::10]:8000", protocol = "tcp", load_balancer = "82a86a8b-5eb5-4ba2-9fbe-8d17c2590416");: parsing actions failed (Syntax error at `vip' expecting empty_lb_backends option name.) 2021-01-29T02:21:25Z|00004|ovntrace|WARN|trigger_event(event = "empty_lb_backends", meter = "", vip = "172.16.103.10:8000", protocol = "tcp", load_balancer = "82a86a8b-5eb5-4ba2-9fbe-8d17c2590416");: parsing actions failed (Syntax error at `vip' expecting empty_lb_backends option name.) 2021-01-29T02:21:25Z|00005|ovntrace|WARN|trigger_event(event = "empty_lb_backends", meter = "", vip = "172.16.103.10:8000", protocol = "tcp", load_balancer = "82a86a8b-5eb5-4ba2-9fbe-8d17c2590416");: parsing actions failed (Syntax error at `vip' expecting empty_lb_backends option name.) 2021-01-29T02:21:25Z|00006|ovntrace|WARN|trigger_event(event = "empty_lb_backends", meter = "", vip = "[2001:db8:103::10]:8000", protocol = "tcp", load_balancer = "82a86a8b-5eb5-4ba2-9fbe-8d17c2590416");: parsing actions failed (Syntax error at `vip' expecting empty_lb_backends option name.) # tcp,reg14=0x2,vlan_tci=0x0000,dl_src=00:de:ad:01:00:01,dl_dst=00:00:00:00:00:00,nw_src=172.16.102.11,nw_dst=172.16.103.10,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=8000,tcp_flags=0 ingress(dp="s2", inport="vm10") ------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:4898): inport == "vm10", priority 50, uuid b712a274 next; 4. ls_in_pre_lb (ovn-northd.c:5238): ip, priority 100, uuid 39ad8cd6 reg0[0] = 1; next; 5. ls_in_pre_stateful (ovn-northd.c:5255): reg0[0] == 1, priority 100, uuid b33918d3 ct_next; ct_next(ct_state=est|trk /* default (use --ct to customize) */) --------------------------------------------------------------- 6. ls_in_acl_hint (ovn-northd.c:5325): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid 27851865 reg0[8] = 1; reg0[10] = 1; next; 12. ls_in_pre_hairpin (ovn-northd.c:6078): ip, priority 90, uuid 600ab3d6 reg0[6] = chk_lb_hairpin_reply(); *** chk_lb_hairpin_reply action not implemented next; 21. ls_in_l2_lkup: no match (implicit drop)
and I tested on the latest version: # rpm -qa|grep ovn ovn2.13-20.12.0-9.el8fdp.x86_64 kernel-kernel-networking-openvswitch-ovn-common-1.0-15.noarch ovn2.13-central-20.12.0-9.el8fdp.x86_64 ovn2.13-host-20.12.0-9.el8fdp.x86_64 it worked, and no error reproted.
verified on version: # rpm -qa|grep ovn ovn2.13-20.12.0-15.el8fdp.x86_64 ovn2.13-central-20.12.0-15.el8fdp.x86_64 ovn2.13-host-20.12.0-15.el8fdp.x86_64 # ovn-trace s2 "inport==\"vm10\" && eth.src==00:de:ad:01:00:01 && ip4.src==172.16.102.11 && ip4.dst==172.16.103.10 && ip.ttl==64 && tcp && tcp.dst==8000" # tcp,reg14=0x2,vlan_tci=0x0000,dl_src=00:de:ad:01:00:01,dl_dst=00:00:00:00:00:00,nw_src=172.16.102.11,nw_dst=172.16.103.10,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=8000,tcp_flags=0 ingress(dp="s2", inport="vm10") ------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:4927): inport == "vm10", priority 50, uuid b63c5c48 next; 4. ls_in_pre_lb (ovn-northd.c:5156): ip4.dst == 172.16.103.10 && tcp && tcp.dst == 8000, priority 130, uuid 2ceb247b trigger_event(event = "empty_lb_backends", vip = "172.16.103.10:8000", protocol = "tcp", load_balancer = "79da0445-62a3-4ea5-8669-d28d159bb93e"); no errors.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0836