Description of problem: Today in ovn-k8s we add reject ACLs to force OVN to send a TCP RST when a client attempts to connect to a service (load balancer) that has no endpoints. Rather than managing these ourselves in ovn-k8s, it would be nice if OVN could build this in as an option to a load balancer. With this feature, when we create the load balancer we could give the option of "default action reject", and OVN would automatically install flows for a load balancer with no endpoints to reject.
scripts: server: # r1 ovn-nbctl lr-add r1 ovn-nbctl lrp-add r1 r1_s2 00:de:ad:ff:01:02 172.16.102.1/24 2001:db8:102::1/64 ovn-nbctl lrp-add r1 r1_s3 00:de:ad:ff:01:03 172.16.103.1/24 2001:db8:103::1/64 # s2 ovn-nbctl ls-add s2 ovn-nbctl set Logical-switch s2 other_config:subnet=172.16.102.0/24 ovn-nbctl set Logical-switch s2 other_config:ipv6_prefix=2001:db8:102::0 # s2 - r1 ovn-nbctl lsp-add s2 s2_r1 ovn-nbctl lsp-set-type s2_r1 router ovn-nbctl lsp-set-addresses s2_r1 "00:de:ad:ff:01:02 172.16.102.1 2001:db8:102::1" ovn-nbctl lsp-set-options s2_r1 router-port=r1_s2 # s2 - hv1_vm00_vnet1 ovn-nbctl lsp-add s2 vm10 ovn-nbctl lsp-set-addresses vm10 "00:de:ad:01:00:01 172.16.102.11 2001:db8:102::11" # s2 - hv1_vm01_vnet1 ovn-nbctl lsp-add s2 vm11 ovn-nbctl lsp-set-addresses vm11 "00:de:ad:01:01:01 172.16.102.12 2001:db8:102::12" # s3 ovn-nbctl ls-add s3 ovn-nbctl set Logical-switch s3 other_config:subnet=172.16.103.0/24 ovn-nbctl set Logical-switch s3 other_config:ipv6_prefix=2001:db8:103::0 # s3 - r1 ovn-nbctl lsp-add s3 s3_r1 ovn-nbctl lsp-set-type s3_r1 router ovn-nbctl lsp-set-addresses s3_r1 "00:de:ad:ff:01:03 172.16.103.1 2001:db8:103::1" ovn-nbctl lsp-set-options s3_r1 router-port=r1_s3 # s3 - hv0_vm00_vnet1 ovn-nbctl lsp-add s3 vm00 ovn-nbctl lsp-set-addresses vm00 "00:de:ad:00:00:01 172.16.103.11 2001:db8:103::11" # s3 - hv0_vm01_vnet1 ovn-nbctl lsp-add s3 vm01 ovn-nbctl lsp-set-addresses vm01 "00:de:ad:00:01:01 172.16.103.12 2001:db8:103::12" ip netns add vm10 ovs-vsctl add-port br-int vm10 -- set interface vm10 type=internal ip link set vm10 netns vm10 ip netns exec vm10 ip link set vm10 address 00:de:ad:01:00:01 ip netns exec vm10 ip addr add 172.16.102.11/24 dev vm10 ip netns exec vm10 ip link set vm10 up ip netns exec vm10 ip link set lo up ip netns exec vm10 ip route add default via 172.16.102.1 ovs-vsctl set Interface vm10 external_ids:iface-id=vm10 ip netns add vm11 ovs-vsctl add-port br-int vm11 -- set interface vm11 type=internal ip link set vm11 netns vm11 ip netns exec vm11 ip link set vm11 address 00:de:ad:01:01:01 ip netns exec vm11 ip addr add 172.16.102.12/24 dev vm11 ip netns exec vm11 ip link set vm11 up ip netns exec vm11 ip link set lo up ip netns exec vm11 ip route add default via 172.16.102.1 ovs-vsctl set Interface vm11 external_ids:iface-id=vm11 ovn-nbctl set Logical_Router r1 options:chassis="hv1" ovn-nbctl show ovn-sbctl show sync_set client $FUNCNAME sync_wait client $FUNCNAME rlRun "ip netns exec vm10 ping -c3 172.16.102.12" rlRun "ip netns exec vm10 ping -c3 172.16.102.11" rlRun "ip netns exec vm10 ping -c3 172.16.103.11" rlRun "ip netns exec vm10 ping -c3 172.16.103.12" #add load balance on logical switch uuid=`ovn-nbctl create load_balancer vips:30.0.0.1="172.16.103.11,172.16.103.12"` rlRun "ovn-nbctl set load_balancer $uuid vips:'\"30.0.0.1:8000\"'='\"172.16.103.11:80,172.16.103.12:80\"' options:reject=true" #create load balance check uuid3=`ovn-nbctl --id=@hc create Load_Balancer_Health_Check vip="30.0.0.1\:8000" -- add Load_Balancer $uuid health_check @hc` ovn-nbctl set Load_Balancer_Health_Check $uuid3 options:interval=5 options:timeout=20 options:success_count=3 options:failure_count=3 ovn-nbctl set logical_switch s2 load_balancer=$uuid ovn-nbctl set logical_router r1 load_balancer=$uuid rlRun "ovn-nbctl --wait=sb set load_balancer $uuid ip_port_mappings:172.16.103.12=vm01:172.16.103.1" rlRun "ovn-nbctl --wait=sb set load_balancer $uuid ip_port_mappings:172.16.103.11=vm00:172.16.103.1" client: ip netns add vm00 ovs-vsctl add-port br-int vm00 -- set interface vm00 type=internal ip link set vm00 netns vm00 ip netns exec vm00 ip link set vm00 address 00:de:ad:00:00:01 ip netns exec vm00 ip addr add 172.16.103.11/24 dev vm00 ip netns exec vm00 ip link set vm00 up ip netns exec vm00 ip link set lo up ip netns exec vm00 ip route add default via 172.16.103.1 ovs-vsctl set Interface vm00 external_ids:iface-id=vm00 ip netns add vm01 ovs-vsctl add-port br-int vm01 -- set interface vm01 type=internal ip link set vm01 netns vm01 ip netns exec vm01 ip link set vm01 address 00:de:ad:00:01:01 ip netns exec vm01 ip addr add 172.16.103.12/24 dev vm01 ip netns exec vm01 ip link set vm01 up ip netns exec vm01 ip link set lo up ip netns exec vm01 ip route add default via 172.16.103.1 ovs-vsctl set Interface vm01 external_ids:iface-id=vm01 verified on version: # rpm -qa|grep ovn ovn2.13-host-20.12.0-1.el8fdp.x86_64 ovn2.13-central-20.12.0-1.el8fdp.x86_64 ovn2.13-20.12.0-1.el8fdp.x86_64 no service start on server(no ncat listening on vm01 and vm00), health check status offline # ovn-sbctl list service_monitor _uuid : 06d26bad-10d6-44fa-98eb-eccacc3f9954 external_ids : {} ip : "172.16.103.12" logical_port : vm01 options : {failure_count="3", interval="5", success_count="3", timeout="20"} port : 80 protocol : tcp src_ip : "172.16.103.1" src_mac : "8a:02:c1:22:a3:1b" status : offline _uuid : d67861bf-95b2-48bc-b6b6-ad6366fb58ab external_ids : {} ip : "172.16.103.11" logical_port : vm00 options : {failure_count="3", interval="5", success_count="3", timeout="20"} port : 80 protocol : tcp src_ip : "172.16.103.1" src_mac : "8a:02:c1:22:a3:1b" status : offline then start ncat client on vm10 # ip netns exec vm10 ncat 30.0.0.1 8000 <<< h 00:07:55.231187 IP (tos 0x0, ttl 64, id 12573, offset 0, flags [DF], proto TCP (6), length 60) 172.16.102.11.41392 > 30.0.0.1.8000: Flags [S], cksum 0x304b (incorrect -> 0xf1e3), seq 1276970488, win 29200, options [mss 1460,sackOK,TS val 1163090838 ecr 0,nop,wscale 7], length 0 00:07:55.231718 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 40) 30.0.0.1.8000 > 172.16.102.11.41392: Flags [R.], cksum 0x6cad (correct), seq 0, ack 1276970489, win 0, length 0 -----------RST Ncat: Connection refused. # ovn-nbctl set load_balancer 2d184157-d122-4295-9846-c9dc315365cf options:reject=false # ip netns exec vm10 ncat 30.0.0.1 8000 <<< h 00:07:08.084636 IP (tos 0x0, ttl 64, id 44216, offset 0, flags [DF], proto TCP (6), length 60) 172.16.102.11.41388 > 30.0.0.1.8000: Flags [S], cksum 0x304b (incorrect -> 0x14d2), seq 639615282, win 29200, options [mss 1460,sackOK,TS val 1163043696 ecr 0,nop,wscale 7], length 0 00:07:09.085880 IP (tos 0x0, ttl 64, id 44217, offset 0, flags [DF], proto TCP (6), length 60) 172.16.102.11.41388 > 30.0.0.1.8000: Flags [S], cksum 0x304b (incorrect -> 0x10e9), seq 639615282, win 29200, options [mss 1460,sackOK,TS val 1163044697 ecr 0,nop,wscale 7], length 0 00:07:11.134016 IP (tos 0x0, ttl 64, id 44218, offset 0, flags [DF], proto TCP (6), length 60) 172.16.102.11.41388 > 30.0.0.1.8000: Flags [S], cksum 0x304b (incorrect -> 0x08e9), seq 639615282, win 29200, options [mss 1460,sackOK,TS val 1163046745 ecr 0,nop,wscale 7], length 0 00:07:15.166015 IP (tos 0x0, ttl 64, id 44219, offset 0, flags [DF], proto TCP (6), length 60) 172.16.102.11.41388 > 30.0.0.1.8000: Flags [S], cksum 0x304b (incorrect -> 0xf928), seq 639615282, win 29200, options [mss 1460,sackOK,TS val 1163050777 ecr 0,nop,wscale 7], length 0 ------------no RST packet. Ncat: Connection timed out.
I´m sorry Ying XU, but #3 is not the correct test. A good test, based on your setup (don´t use the loadbalancer health check), can be: - Configure the loadbalancer VIP ONLY (no endpoints) `ovn-nbctl create load_balancer vips:30.0.0.1:8000=""` - Check that connecting from the client to the VIP blackhole the packet (no answer) - Set the "options:reject=true" to the loadbalancer (no endpoints configured yet) - Check that connecting to from the client to the VIP we receive a RST - Configured endpoints to the loadbalancer ovn-nbctl set load_balancer $uuid vips:'\"30.0.0.1:8000\"'='\"172.16.103.11:80,172.16.103.12:80\"' options:reject=true" - Check that the client reaches the endpoints - Remove the endpoints again from the loadbalance rovn-nbctl set load_balancer $uuid vips:'\"30.0.0.1:8000\"'='\"\"' options:reject=true" - Check that connecting to from the client to the VIP we receive a RST
(In reply to Antonio Ojea from comment #4) > I´m sorry Ying XU, but #3 is not the correct test. > > A good test, based on your setup (don´t use the loadbalancer health check), > can be: > > - Configure the loadbalancer VIP ONLY (no endpoints) `ovn-nbctl create > load_balancer vips:30.0.0.1:8000=""` But I can't set this command successfully ,it will report error.so I don't know how to create lb with no backends. # ovn-nbctl create load_balancer vips:30.0.0.1:8000="" ovn-nbctl: vips:30.0.0.1:8000=: argument does not end in "=" followed by a value. > - Check that connecting from the client to the VIP blackhole the packet (no > answer) > - Set the "options:reject=true" to the loadbalancer (no endpoints configured > yet) > - Check that connecting to from the client to the VIP we receive a RST > - Configured endpoints to the loadbalancer ovn-nbctl set load_balancer $uuid > vips:'\"30.0.0.1:8000\"'='\"172.16.103.11:80,172.16.103.12:80\"' > options:reject=true" > - Check that the client reaches the endpoints > - Remove the endpoints again from the loadbalance rovn-nbctl set > load_balancer $uuid vips:'\"30.0.0.1:8000\"'='\"\"' options:reject=true" > - Check that connecting to from the client to the VIP we receive a RST
Try ovn-nbctl lb-add test 30.0.0.10:80 "" it works for me _uuid : c25ced35-7944-48fd-ab08-5775b6e31aa4 external_ids : {} health_check : [] ip_port_mappings : {} name : test protocol : tcp selection_fields : [] vips : {"30.0.0.10:80"=""}
Thanks to Antonio Ojea, and I test it as Antonio Ojea said in comment 4 # ovn-nbctl list load_balancer _uuid : 606713c6-f294-4512-895c-d3443a0ac86d external_ids : {} health_check : [] ip_port_mappings : {} name : lb1 options : {reject="true"} protocol : tcp selection_fields : [] vips : {"172.16.103.10:8000"=""} --------------create a load_balancer with no backends # ovn-nbctl list logical-switch _uuid : 94be45cc-4f8d-4d5f-9bee-e6b06a44f7de acls : [] dns_records : [] external_ids : {} forwarding_groups : [] load_balancer : [606713c6-f294-4512-895c-d3443a0ac86d] name : s3 other_config : {} ports : [3c156838-bfd9-4e9b-9bdc-cc53d9c92f34, 6273a634-e885-42c7-ad67-87362fe9cf31, 9659a9e8-8d1a-445c-9a71-c282c4590259] qos_rules : [] _uuid : 5bee5761-cd67-43ff-a515-fc1057d566e7 acls : [] dns_records : [] external_ids : {} forwarding_groups : [] load_balancer : [606713c6-f294-4512-895c-d3443a0ac86d] name : s2 other_config : {} ports : [76b8c12f-1a27-4aff-8b67-cd74ce7d7639, bbbe0076-affc-44e5-9b54-4a422a97a73a, f461d39b-ba83-49fe-be2a-7885885ac15f] qos_rules : [] # ovn-nbctl list load_balancer_health_check ---------------no health check # then send tcp connect to 172.16.103.10:8000 # ip netns exec vm10 ncat 172.16.103.10 8000 <<< h Ncat: Connection refused. ------------------------refused set option to false # ovn-nbctl set load_balancer 606713c6-f294-4512-895c-d3443a0ac86d options:reject=false # ip netns exec vm11 ncat 172.16.103.10 8000 <<< h Ncat: Connection timed out. ---------------------------time out add the endpoints: # ovn-nbctl list load_balancer _uuid : 606713c6-f294-4512-895c-d3443a0ac86d external_ids : {} health_check : [] ip_port_mappings : {} name : lb1 options : {reject="false"} protocol : tcp selection_fields : [] vips : {"172.16.103.10:8000"="172.16.102.12:80,172.16.103.12:80"} # ip netns exec vm11 ncat 172.16.103.10 8000 <<< h start server and server can recieve the packets: # ip netns exec vm01 ncat -l 80& [1] 1119939 [root@dell-per740-54 load_balance]# h remove the backends again: # ovn-nbctl list load_balancer 606713c6-f294-4512-895c-d3443a0ac86d _uuid : 606713c6-f294-4512-895c-d3443a0ac86d external_ids : {} health_check : [] ip_port_mappings : {} name : lb1 options : {reject="true"} protocol : tcp selection_fields : [] vips : {"172.16.103.10:8000"=""} # ip netns exec vm11 ncat 172.16.103.10 8000 <<< h Ncat: Connection refused. ----------------refused again
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0407