The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.
Bug 2164652 - Localnet port traffic is not load balanced on the localnet switch.
Summary: Localnet port traffic is not load balanced on the localnet switch.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: ovn23.06
Version: FDP 23.A
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
: ---
Assignee: Mark Michelson
QA Contact: ying xu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-01-25 22:17 UTC by Dumitru Ceara
Modified: 2024-01-24 11:05 UTC (History)
5 users (show)

Fixed In Version: ovn23.06-23.06.0-148.el8fdp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-01-24 11:05:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-2655 0 None None None 2023-01-25 22:21:02 UTC
Red Hat Product Errata RHBA-2024:0388 0 None None None 2024-01-24 11:05:17 UTC

Description Dumitru Ceara 2023-01-25 22:17:41 UTC
Description of problem:

If a load balancer is applied to a logical switch that contains localnet ports, no traffic received from the localnet ports will be load balanced.

This is due to two reasons:
a. conntrack (for ACL and LB) is skipped in the ingress logical switch pipeline for localnet ports.
b. even if conntrack wouldn't be skipped, localnet ports don't get a conntrack zone assignment in ovn-controller.

Version-Release number of selected component (if applicable):


How reproducible:
Always.

Steps to Reproduce:

# Topology:
# br-phys (localnet) --- ls-pub --- rtr -- ls-priv -- vm1
# - a load balancer applied on ls-pub with backend vm1
# - try to access vm1 via LB from br-phys
ovn-nbctl lr-add rtr
ovn-nbctl lrp-add rtr rtr-ls-pub 00:00:00:00:01:00 41.41.41.1/24
ovn-nbctl lrp-add rtr rtr-ls-priv 00:00:00:00:02:00 42.42.42.1/24
ovn-nbctl ls-add ls-pub
ovn-nbctl ls-add ls-priv

# Localnet
ovn-nbctl lsp-add ls-pub ln
ovn-nbctl lsp-set-type ln localnet
ovn-nbctl lsp-set-addresses ln unknown
ovn-nbctl lsp-set-options ln network_name=phys
ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
ovs-vsctl add-br br-phys

# Simulate external host on localnet
ip a a dev br-phys 41.41.41.2/24
ip link set dev br-phys up

ovn-nbctl lsp-add ls-pub ls-pub-rtr
ovn-nbctl lsp-set-addresses ls-pub-rtr 00:00:00:00:01:00
ovn-nbctl lsp-set-type ls-pub-rtr router
ovn-nbctl lsp-set-options ls-pub-rtr router-port=rtr-ls-pub

ovn-nbctl lsp-add ls-priv ls-priv-rtr
ovn-nbctl lsp-set-addresses ls-priv-rtr 00:00:00:00:02:00
ovn-nbctl lsp-set-type ls-priv-rtr router
ovn-nbctl lsp-set-options ls-priv-rtr router-port=rtr-ls-priv

# Simulate backend attached to router
ovn-nbctl lsp-add ls-priv vm1
ovn-nbctl lsp-set-addresses vm1 00:00:00:00:00:01
ip netns add vm1
ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal
ip link set vm1 netns vm1
ip netns exec vm1 ip link set vm1 address 00:00:00:00:00:01
ip netns exec vm1 ip addr add 42.42.42.2/24 dev vm1
ip netns exec vm1 ip link set vm1 up
ip netns exec vm1 ip r a default via 42.42.42.1
ovs-vsctl set Interface vm1 external_ids:iface-id=vm1

# Add LB to switch and router
ovn-nbctl lb-add lb-test 66.66.66.66:666 42.42.42.2:4242 tcp
ovn-nbctl ls-lb-add ls-pub lb-test

# Add route to 66.66.66.66 and 42.42.42.2
ip r a 66.66.66.66 via 41.41.41.1
ip r a 42.42.42.0/24 via 41.41.41.1

# Start a listener on vm1
ip netns exec vm1 nc -v -k -l 42.42.42.2 4242

# Try to connect from the localnet port:
nc -v -z 66.66.66.66 666

Note:
With the following patch to force conntrack zones to be allocated to localnet ports and to skip bypassing conntrack for localnet ports traffic is load balanced correctly.  This doesn't mean however that the patch is the right way to fix the problem, it's merely a way of confirming the two reasons why traffic doesn't flow properly.

diff --git a/controller/ovn-controller.c b/controller/ovn-controller.c
index 265740cabc..dcbf65a149 100644
--- a/controller/ovn-controller.c
+++ b/controller/ovn-controller.c
@@ -711,7 +711,7 @@ get_snat_ct_zone(const struct sbrec_datapath_binding *dp)
 }
 
 static void
-update_ct_zones(const struct shash *binding_lports,
+update_ct_zones(const struct sset *local_lports,
                 const struct hmap *local_datapaths,
                 struct simap *ct_zones, unsigned long *ct_zone_bitmap,
                 struct shash *pending_ct_zones)
@@ -724,9 +724,9 @@ update_ct_zones(const struct shash *binding_lports,
     unsigned long unreq_snat_zones_map[BITMAP_N_LONGS(MAX_CT_ZONES)];
     struct simap unreq_snat_zones = SIMAP_INITIALIZER(&unreq_snat_zones);
 
-    struct shash_node *shash_node;
-    SHASH_FOR_EACH (shash_node, binding_lports) {
-        sset_add(&all_users, shash_node->name);
+    const char *local_lport;
+    SSET_FOR_EACH (local_lport, local_lports) {
+        sset_add(&all_users, local_lport);
     }
 
     /* Local patched datapath (gateway routers) need zones assigned. */
@@ -2373,7 +2373,7 @@ en_ct_zones_run(struct engine_node *node, void *data)
         EN_OVSDB_GET(engine_get_input("OVS_bridge", node));
 
     restore_ct_zones(bridge_table, ovs_table, ct_zones_data);
-    update_ct_zones(&rt_data->lbinding_data.lports, &rt_data->local_datapaths,
+    update_ct_zones(&rt_data->local_lports, &rt_data->local_datapaths,
                     &ct_zones_data->current, ct_zones_data->bitmap,
                     &ct_zones_data->pending);
 
@@ -2463,13 +2463,15 @@ ct_zones_runtime_data_handler(struct engine_node *node, void *data)
         SHASH_FOR_EACH (shash_node, &tdp->lports) {
             struct tracked_lport *t_lport = shash_node->data;
             if (strcmp(t_lport->pb->type, "")
-                && strcmp(t_lport->pb->type, "localport")) {
+                && strcmp(t_lport->pb->type, "localport")
+                && strcmp(t_lport->pb->type, "localnet")) {
                 /* We allocate zone-id's only to VIF and localport lports. */
                 continue;
             }
 
             if (t_lport->tracked_type == TRACKED_RESOURCE_NEW ||
                 t_lport->tracked_type == TRACKED_RESOURCE_UPDATED) {
+                VLOG_INFO("DEBUG DEBUG trying to allocate ct zone id for port %s", t_lport->pb->logical_port);
                 if (!simap_contains(&ct_zones_data->current,
                                     t_lport->pb->logical_port)) {
                     alloc_id_to_ct_zone(t_lport->pb->logical_port,
diff --git a/northd/northd.c b/northd/northd.c
index 0944a7b567..fac6a15411 100644
--- a/northd/northd.c
+++ b/northd/northd.c
@@ -5843,11 +5843,6 @@ build_pre_acls(struct ovn_datapath *od, const struct hmap *port_groups,
                                      S_SWITCH_IN_PRE_ACL, S_SWITCH_OUT_PRE_ACL,
                                      110, lflows);
         }
-        for (size_t i = 0; i < od->n_localnet_ports; i++) {
-            skip_port_from_conntrack(od, od->localnet_ports[i],
-                                     S_SWITCH_IN_PRE_ACL, S_SWITCH_OUT_PRE_ACL,
-                                     110, lflows);
-        }
 
         /* stateless filters always take precedence over stateful ACLs. */
         build_stateless_filters(od, port_groups, lflows);
@@ -6014,11 +6009,6 @@ build_pre_lb(struct ovn_datapath *od, const struct shash *meter_groups,
                                  S_SWITCH_IN_PRE_LB, S_SWITCH_OUT_PRE_LB,
                                  110, lflows);
     }
-    for (size_t i = 0; i < od->n_localnet_ports; i++) {
-        skip_port_from_conntrack(od, od->localnet_ports[i],
-                                 S_SWITCH_IN_PRE_LB, S_SWITCH_OUT_PRE_LB,
-                                 110, lflows);
-    }
 
     /* Do not sent statless flows via conntrack */
     ovn_lflow_add(lflows, od, S_SWITCH_IN_PRE_LB, 110,

Comment 1 OVN Bot 2023-05-20 04:10:18 UTC
ovn23.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208730
ovn23.03 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208731
ovn23.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208732
ovn22.12 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208733
ovn22.12 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208734
ovn22.03 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208735
ovn22.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208736

Comment 4 ying xu 2023-12-02 01:03:26 UTC
verified on version:
# rpm -qa|grep ovn
ovn23.06-central-23.06.1-60.el8fdp.x86_64
ovn23.06-23.06.1-60.el8fdp.x86_64
ovn23.06-host-23.06.1-60.el8fdp.x86_64



::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   ovn_test_bz2164652
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ 19:21:55 ] :: [   PASS   ] :: Command 'systemctl start openvswitch' (Expected 0, got 0)
:: [ 19:21:55 ] :: [   PASS   ] :: Command 'systemctl start ovn-northd' (Expected 0, got 0)
:: [ 19:21:55 ] :: [   PASS   ] :: Command 'ovn-nbctl set-connection ptcp:6641' (Expected 0, got 0)
:: [ 19:21:55 ] :: [   PASS   ] :: Command 'ovn-sbctl set-connection ptcp:6642' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command 'ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:127.0.0.1:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=127.0.0.1' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command 'systemctl restart ovn-controller' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl lr-add rtr' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl lrp-add rtr rtr-ls-pub 00:00:00:00:01:00 41.41.41.1/24' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl lrp-add rtr rtr-ls-priv 00:00:00:00:02:00 42.42.42.1/24' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl ls-add ls-pub' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl ls-add ls-priv' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-add ls-pub ln' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-type ln localnet' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-addresses ln unknown' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-options ln network_name=phys' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys' (Expected 0, got 0)
:: [ 19:21:56 ] :: [   PASS   ] :: Command '	ovs-vsctl add-br br-phys' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ip a a dev br-phys 41.41.41.2/24' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ip link set dev br-phys up' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-add ls-pub ls-pub-rtr' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-addresses ls-pub-rtr 00:00:00:00:01:00' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-type ls-pub-rtr router' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-options ls-pub-rtr router-port=rtr-ls-pub' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-add ls-priv ls-priv-rtr' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-addresses ls-priv-rtr 00:00:00:00:02:00' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-type ls-priv-rtr router' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-options ls-priv-rtr router-port=rtr-ls-priv' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-add ls-priv vm1' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovn-nbctl lsp-set-addresses vm1 00:00:00:00:00:01' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ip netns add vm1' (Expected 0, got 0)
:: [ 19:21:57 ] :: [   PASS   ] :: Command '	ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ip link set vm1 netns vm1' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ip netns exec vm1 ip link set vm1 address 00:00:00:00:00:01' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ip netns exec vm1 ip addr add 42.42.42.2/24 dev vm1' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ip netns exec vm1 ip link set vm1 up' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ip netns exec vm1 ip r a default via 42.42.42.1' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ovs-vsctl set Interface vm1 external_ids:iface-id=vm1' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ovn-nbctl lb-add lb-test 66.66.66.66:666 42.42.42.2:4242 tcp' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ovn-nbctl ls-lb-add ls-pub lb-test' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ip r a 66.66.66.66 via 41.41.41.1' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command '	ip r a 42.42.42.0/24 via 41.41.41.1' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command 'ip netns exec vm1 nc -v -k -l 42.42.42.2 4242&' (Expected 0, got 0)
:: [ 19:21:58 ] :: [   PASS   ] :: Command 'nc -v -z 66.66.66.66 666' (Expected 0, got 0)
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Duration: 15s
::   Assertions: 43 good, 0 bad
::   RESULT: PASS (ovn_test_bz2164652)

set verified.

Comment 6 errata-xmlrpc 2024-01-24 11:05:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn23.06 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:0388


Note You need to log in before you can comment on or make changes to this bug.