Description of problem: If a load balancer is applied to a logical switch that contains localnet ports, no traffic received from the localnet ports will be load balanced. This is due to two reasons: a. conntrack (for ACL and LB) is skipped in the ingress logical switch pipeline for localnet ports. b. even if conntrack wouldn't be skipped, localnet ports don't get a conntrack zone assignment in ovn-controller. Version-Release number of selected component (if applicable): How reproducible: Always. Steps to Reproduce: # Topology: # br-phys (localnet) --- ls-pub --- rtr -- ls-priv -- vm1 # - a load balancer applied on ls-pub with backend vm1 # - try to access vm1 via LB from br-phys ovn-nbctl lr-add rtr ovn-nbctl lrp-add rtr rtr-ls-pub 00:00:00:00:01:00 41.41.41.1/24 ovn-nbctl lrp-add rtr rtr-ls-priv 00:00:00:00:02:00 42.42.42.1/24 ovn-nbctl ls-add ls-pub ovn-nbctl ls-add ls-priv # Localnet ovn-nbctl lsp-add ls-pub ln ovn-nbctl lsp-set-type ln localnet ovn-nbctl lsp-set-addresses ln unknown ovn-nbctl lsp-set-options ln network_name=phys ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys ovs-vsctl add-br br-phys # Simulate external host on localnet ip a a dev br-phys 41.41.41.2/24 ip link set dev br-phys up ovn-nbctl lsp-add ls-pub ls-pub-rtr ovn-nbctl lsp-set-addresses ls-pub-rtr 00:00:00:00:01:00 ovn-nbctl lsp-set-type ls-pub-rtr router ovn-nbctl lsp-set-options ls-pub-rtr router-port=rtr-ls-pub ovn-nbctl lsp-add ls-priv ls-priv-rtr ovn-nbctl lsp-set-addresses ls-priv-rtr 00:00:00:00:02:00 ovn-nbctl lsp-set-type ls-priv-rtr router ovn-nbctl lsp-set-options ls-priv-rtr router-port=rtr-ls-priv # Simulate backend attached to router ovn-nbctl lsp-add ls-priv vm1 ovn-nbctl lsp-set-addresses vm1 00:00:00:00:00:01 ip netns add vm1 ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal ip link set vm1 netns vm1 ip netns exec vm1 ip link set vm1 address 00:00:00:00:00:01 ip netns exec vm1 ip addr add 42.42.42.2/24 dev vm1 ip netns exec vm1 ip link set vm1 up ip netns exec vm1 ip r a default via 42.42.42.1 ovs-vsctl set Interface vm1 external_ids:iface-id=vm1 # Add LB to switch and router ovn-nbctl lb-add lb-test 66.66.66.66:666 42.42.42.2:4242 tcp ovn-nbctl ls-lb-add ls-pub lb-test # Add route to 66.66.66.66 and 42.42.42.2 ip r a 66.66.66.66 via 41.41.41.1 ip r a 42.42.42.0/24 via 41.41.41.1 # Start a listener on vm1 ip netns exec vm1 nc -v -k -l 42.42.42.2 4242 # Try to connect from the localnet port: nc -v -z 66.66.66.66 666 Note: With the following patch to force conntrack zones to be allocated to localnet ports and to skip bypassing conntrack for localnet ports traffic is load balanced correctly. This doesn't mean however that the patch is the right way to fix the problem, it's merely a way of confirming the two reasons why traffic doesn't flow properly. diff --git a/controller/ovn-controller.c b/controller/ovn-controller.c index 265740cabc..dcbf65a149 100644 --- a/controller/ovn-controller.c +++ b/controller/ovn-controller.c @@ -711,7 +711,7 @@ get_snat_ct_zone(const struct sbrec_datapath_binding *dp) } static void -update_ct_zones(const struct shash *binding_lports, +update_ct_zones(const struct sset *local_lports, const struct hmap *local_datapaths, struct simap *ct_zones, unsigned long *ct_zone_bitmap, struct shash *pending_ct_zones) @@ -724,9 +724,9 @@ update_ct_zones(const struct shash *binding_lports, unsigned long unreq_snat_zones_map[BITMAP_N_LONGS(MAX_CT_ZONES)]; struct simap unreq_snat_zones = SIMAP_INITIALIZER(&unreq_snat_zones); - struct shash_node *shash_node; - SHASH_FOR_EACH (shash_node, binding_lports) { - sset_add(&all_users, shash_node->name); + const char *local_lport; + SSET_FOR_EACH (local_lport, local_lports) { + sset_add(&all_users, local_lport); } /* Local patched datapath (gateway routers) need zones assigned. */ @@ -2373,7 +2373,7 @@ en_ct_zones_run(struct engine_node *node, void *data) EN_OVSDB_GET(engine_get_input("OVS_bridge", node)); restore_ct_zones(bridge_table, ovs_table, ct_zones_data); - update_ct_zones(&rt_data->lbinding_data.lports, &rt_data->local_datapaths, + update_ct_zones(&rt_data->local_lports, &rt_data->local_datapaths, &ct_zones_data->current, ct_zones_data->bitmap, &ct_zones_data->pending); @@ -2463,13 +2463,15 @@ ct_zones_runtime_data_handler(struct engine_node *node, void *data) SHASH_FOR_EACH (shash_node, &tdp->lports) { struct tracked_lport *t_lport = shash_node->data; if (strcmp(t_lport->pb->type, "") - && strcmp(t_lport->pb->type, "localport")) { + && strcmp(t_lport->pb->type, "localport") + && strcmp(t_lport->pb->type, "localnet")) { /* We allocate zone-id's only to VIF and localport lports. */ continue; } if (t_lport->tracked_type == TRACKED_RESOURCE_NEW || t_lport->tracked_type == TRACKED_RESOURCE_UPDATED) { + VLOG_INFO("DEBUG DEBUG trying to allocate ct zone id for port %s", t_lport->pb->logical_port); if (!simap_contains(&ct_zones_data->current, t_lport->pb->logical_port)) { alloc_id_to_ct_zone(t_lport->pb->logical_port, diff --git a/northd/northd.c b/northd/northd.c index 0944a7b567..fac6a15411 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -5843,11 +5843,6 @@ build_pre_acls(struct ovn_datapath *od, const struct hmap *port_groups, S_SWITCH_IN_PRE_ACL, S_SWITCH_OUT_PRE_ACL, 110, lflows); } - for (size_t i = 0; i < od->n_localnet_ports; i++) { - skip_port_from_conntrack(od, od->localnet_ports[i], - S_SWITCH_IN_PRE_ACL, S_SWITCH_OUT_PRE_ACL, - 110, lflows); - } /* stateless filters always take precedence over stateful ACLs. */ build_stateless_filters(od, port_groups, lflows); @@ -6014,11 +6009,6 @@ build_pre_lb(struct ovn_datapath *od, const struct shash *meter_groups, S_SWITCH_IN_PRE_LB, S_SWITCH_OUT_PRE_LB, 110, lflows); } - for (size_t i = 0; i < od->n_localnet_ports; i++) { - skip_port_from_conntrack(od, od->localnet_ports[i], - S_SWITCH_IN_PRE_LB, S_SWITCH_OUT_PRE_LB, - 110, lflows); - } /* Do not sent statless flows via conntrack */ ovn_lflow_add(lflows, od, S_SWITCH_IN_PRE_LB, 110,
ovn23.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208730 ovn23.03 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208731 ovn23.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208732 ovn22.12 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208733 ovn22.12 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208734 ovn22.03 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208735 ovn22.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208736
verified on version: # rpm -qa|grep ovn ovn23.06-central-23.06.1-60.el8fdp.x86_64 ovn23.06-23.06.1-60.el8fdp.x86_64 ovn23.06-host-23.06.1-60.el8fdp.x86_64 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: ovn_test_bz2164652 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ 19:21:55 ] :: [ PASS ] :: Command 'systemctl start openvswitch' (Expected 0, got 0) :: [ 19:21:55 ] :: [ PASS ] :: Command 'systemctl start ovn-northd' (Expected 0, got 0) :: [ 19:21:55 ] :: [ PASS ] :: Command 'ovn-nbctl set-connection ptcp:6641' (Expected 0, got 0) :: [ 19:21:55 ] :: [ PASS ] :: Command 'ovn-sbctl set-connection ptcp:6642' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command 'ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:127.0.0.1:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command 'systemctl restart ovn-controller' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl lr-add rtr' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl lrp-add rtr rtr-ls-pub 00:00:00:00:01:00 41.41.41.1/24' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl lrp-add rtr rtr-ls-priv 00:00:00:00:02:00 42.42.42.1/24' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl ls-add ls-pub' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl ls-add ls-priv' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-add ls-pub ln' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-type ln localnet' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-addresses ln unknown' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-options ln network_name=phys' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys' (Expected 0, got 0) :: [ 19:21:56 ] :: [ PASS ] :: Command ' ovs-vsctl add-br br-phys' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ip a a dev br-phys 41.41.41.2/24' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ip link set dev br-phys up' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-add ls-pub ls-pub-rtr' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-addresses ls-pub-rtr 00:00:00:00:01:00' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-type ls-pub-rtr router' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-options ls-pub-rtr router-port=rtr-ls-pub' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-add ls-priv ls-priv-rtr' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-addresses ls-priv-rtr 00:00:00:00:02:00' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-type ls-priv-rtr router' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-options ls-priv-rtr router-port=rtr-ls-priv' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-add ls-priv vm1' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovn-nbctl lsp-set-addresses vm1 00:00:00:00:00:01' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ip netns add vm1' (Expected 0, got 0) :: [ 19:21:57 ] :: [ PASS ] :: Command ' ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ip link set vm1 netns vm1' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ip netns exec vm1 ip link set vm1 address 00:00:00:00:00:01' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ip netns exec vm1 ip addr add 42.42.42.2/24 dev vm1' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ip netns exec vm1 ip link set vm1 up' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ip netns exec vm1 ip r a default via 42.42.42.1' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ovs-vsctl set Interface vm1 external_ids:iface-id=vm1' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ovn-nbctl lb-add lb-test 66.66.66.66:666 42.42.42.2:4242 tcp' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ovn-nbctl ls-lb-add ls-pub lb-test' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ip r a 66.66.66.66 via 41.41.41.1' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command ' ip r a 42.42.42.0/24 via 41.41.41.1' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command 'ip netns exec vm1 nc -v -k -l 42.42.42.2 4242&' (Expected 0, got 0) :: [ 19:21:58 ] :: [ PASS ] :: Command 'nc -v -z 66.66.66.66 666' (Expected 0, got 0) :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: Duration: 15s :: Assertions: 43 good, 0 bad :: RESULT: PASS (ovn_test_bz2164652) set verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn23.06 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:0388