Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1707513

Summary: [RFE] Consistent-hashing capability for load balancing
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Carlos Goncalves <cgoncalves>
Component: OVNAssignee: Numan Siddique <nusiddiq>
Status: CLOSED ERRATA QA Contact: ying xu <yinxu>
Severity: medium Docs Contact:
Priority: medium    
Version: FDP 19.ECC: atragler, ctrautma, dcbw, jishi, nusiddiq, rkhan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn2.13-2.13.0-28.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-15 13:00:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1825073    

Description Carlos Goncalves 2019-05-07 16:56:50 UTC
Consistent-hashing algorithm in load balancing enables ingress traffic to be distributed among backend servers according to a hash scheme that is computed based on given fields (single or compound). For example: source, destination or both IP addresses, transport protocol, requested URL, etc.

OVS supports a set of OpenFlow "select" groups that could facilitate the programming of flows. Particularly, the selection_method=hash. From the OVS documentation, it can hash fields like "source and destination Ethernet address, VLAN ID, Ethernet type, IPv4/v6 source and destination address and protocol, and for TCP and SCTP only, the source and destination ports."

Also interesting to explore would be the field "weight=" under bucket parameters to enable connection drainage of server backends (e.g. to stop forwarding new flows to given server backends).

A more advanced use-case: active-active load balancing topology. OVN load balancers on Tier 1 (outer tier), Octavia amphora or 3rd party vendors load balancers on Tier 2 (inner tier).

There's a patch open in Octavia implementing an active-active topology where the distributor (Tier 1) is backed by OpenFlow: https://review.opendev.org/#/c/427858/

Comment 2 Dan Williams 2020-04-09 16:30:45 UTC
OpenShift would also like slightly different hashing capability for LBs in UDP scenarios. It would like to use [srcip, dstip, dstport] (eg *not* srcport) for specific LB VIP entries, but allow other LB VIPs to use the normal 4-tuple.

Comment 3 Dan Williams 2020-04-09 16:32:03 UTC
Related affinity issues/bugs from the OpenShift side:

https://issues.redhat.com/browse/SDN-510
https://bugzilla.redhat.com/show_bug.cgi?id=1707513

Comment 4 Dan Williams 2020-04-21 14:26:27 UTC
Clarification: OpenShift would like to optionally use [srcip, dstip, dstport] for both TCP and UDP for a given LB VIP. If traffic matching that 3-tuple has not been seen within X seconds the affinity can be cleared and recalculated when new traffic comes to the VIP.

Comment 5 Numan Siddique 2020-04-21 17:20:35 UTC
OVN load balancer is using dp_hash as a selection method for choosing the bucket (which translates to a backend).
Although the commit message in the ovs says, dp_hash uses 5-tuple hash, I didn't see that in my testing. For a given L4 pkt,
the dp_hash changes and this results in bucket selection non deterministic.

 There are few issues with the current OVN implementation. More details here - https://mail.openvswitch.org/pipermail/ovs-discuss/2020-April/049940.html

If we change from dp_hash to "hash", then ovs-vswitchd will use 5-tuple hash. I did some testing and I can confirm that with this for a given L4 packet,
the bucket selection is deterministic i.e the same bucket is chosen all the time.

Is opensift OK changing the selection_method from dp_hash to hash ?

Also I think we can most likely support a hash of srcip, dstip and dst port. I'm looking into this and I'm planning to expose the hash method
to CMS so that it can chose - i.e SRC_IP_PORT, OR SRC_IP_DST_IP_DST_PORT etc.

Comment 6 Dan Williams 2020-04-22 03:22:18 UTC
I don't think ovnkube/OpenShift care what the hash algorithm is, but I think it's useful to have both options. For example, OpenShift is OK with the more random nature of the current hash as that has randomness advantages. But for specific LB VIPs it would like to use a more deterministic algorithm like SRC_IP_DST_IP_DST_PORT, but only certain ones chosen by the cluster users.

Comment 7 Numan Siddique 2020-04-22 06:59:26 UTC
Ok. Thanks. I'll not change from dp_hash to hash. Instead provide the option for CMS to choose with the default being dp_hash.

Comment 8 Numan Siddique 2020-05-28 17:02:34 UTC
Fix available in 2.13.0-28

Comment 12 ying xu 2020-06-11 07:46:49 UTC
I create a env as follow script:
ovn-nbctl ls-add ls1
                ovn-nbctl lsp-add ls1 ls1p1

                ovn-nbctl lsp-set-addresses ls1p1 00:01:02:01:01:01
                ovn-nbctl lsp-add ls1 ls1p2

                ovn-nbctl lsp-set-addresses ls1p2 00:01:02:01:01:02

                ovn-nbctl lsp-add ls1 ls1p3
                ovn-nbctl lsp-set-addresses ls1p3 00:01:02:01:01:04

                ovn-nbctl ls-add ls2
                ovn-nbctl lsp-add ls2 ls2p1
                ovn-nbctl lsp-set-addresses ls2p1 00:01:02:01:01:03

                ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal
                ip netns add server0
                ip link set vm1 netns server0
                ip netns exec server0 ip link set lo up
                ip netns exec server0 ip link set vm1 up
                ip netns exec server0 ip link set vm1 address 00:01:02:01:01:01
                ip netns exec server0 ip addr add 192.168.0.1/24 dev vm1
                ip netns exec server0 ip addr add 3001::1/64 dev vm1
                ip netns exec server0 ip route add default via 192.168.0.254 dev vm1
                ip netns exec server0 ip -6 route add default via 3001::a dev vm1
                ovs-vsctl set Interface vm1 external_ids:iface-id=ls1p1
		ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal
                ip netns add server1
                ip link set vm2 netns server1
                ip netns exec server1 ip link set lo up
                ip netns exec server1 ip link set vm2 up
                ip netns exec server1 ip link set vm2 address 00:01:02:01:01:02
                ip netns exec server1 ip addr add 192.168.0.2/24 dev vm2
                ip netns exec server1 ip addr add 3001::2/64 dev vm2
                ip netns exec server1 ip route add default via 192.168.0.254 dev vm2
                ip netns exec server1 ip -6 route add default via 3001::a dev vm2
                ovs-vsctl set Interface vm2 external_ids:iface-id=ls1p2

                ovn-nbctl lr-add lr1
                ovn-nbctl lrp-add lr1 lr1ls1 00:01:02:0d:01:01 192.168.0.254/24 3001::a/64
                ovn-nbctl lrp-add lr1 lr1ls2 00:01:02:0d:01:02 192.168.1.254/24 3001:1::a/64

                ovn-nbctl lsp-add ls1 ls1lr1
                ovn-nbctl lsp-set-type ls1lr1 router
                ovn-nbctl lsp-set-options ls1lr1 router-port=lr1ls1
                ovn-nbctl lsp-set-addresses ls1lr1 "00:01:02:0d:01:01 192.168.0.254 3001::a"
                ovn-nbctl lsp-add ls2 ls2lr1
                ovn-nbctl lsp-set-type ls2lr1 router
                ovn-nbctl lsp-set-options ls2lr1 router-port=lr1ls2
                ovn-nbctl lsp-set-addresses ls2lr1 "00:01:02:0d:01:02 192.168.1.254 3001:1::a"
		ovn-nbctl lrp-add lr1 lr1p 00:01:02:0d:0f:01 172.16.1.254/24 2002::a/64

                ovn-nbctl lb-add lb0 192.168.2.1:12345 192.168.0.1:12345,192.168.0.2:12345
                ovn-nbctl lb-add lb0 [3000::100]:12345 [3001::1]:12345,[3001::2]:12345
                 uuid=`ovn-nbctl list Load_Balancer |grep uuid|awk '{printf $3}'`

                ovn-nbctl set load_balancer $uuid selection_fields="ip_src,ip_dst"
                ovn-nbctl show
                ovn-sbctl show
                 ovn-nbctl set  Logical_Router lr1 options:chassis="hv1"

                ovn-nbctl ls-lb-add ls1 lb0


I test selection_fields as "eth_src,eth_dst","ip_src,ip_dst","ip_src,ip_dst,tp_src,tp_dst",and lb on ls and lr.
for tcp case,lb works well. so set the bug verified.
version:
# rpm -qa|grep ovn
ovn2.13-central-2.13.0-34.el7fdp.x86_64
ovn2.13-2.13.0-34.el7fdp.x86_64
ovn2.13-host-2.13.0-34.el7fdp.x86_64

performance test will be done later.

for udp and sctp case, lb works not correctly.raised bug here:https://bugzilla.redhat.com/show_bug.cgi?id=1846189

Comment 14 errata-xmlrpc 2020-07-15 13:00:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2941