Bug 1416869

Created attachment 1244807 [details]
Split Traffic UI Screenshot

Did you set the balance algorithm annotation to round-robin on the route?  It sounds like you are getting the default least connections behavior.

The only annotation attached to that route is: openshift.io/host.generated=true

The web console either needs to set, or allow to be set, the route balance algorithm.

The route annotation haproxy.router.openshift.io/balance needs to be set to roundrobin if you want the weights to have any effect.

https://docs.openshift.com/container-platform/3.4/architecture/core_concepts/routes.html#route-specific-annotations

Ben, can the router default to round robin if there is a route has alternate backends? If the user sets weights, it seems clear that they want them to work...

Is this also a problem with `oc set route-backends`?

Even if working as intended, I see it as a usability bug. It's really not obvious you need to set the annotation. Even if the web console starts adding the annotation, it won't fix any existing routes created from the web console.

At the very least, we need to mention the annotation in the topic on a/b testing:

https://docs.openshift.org/latest/dev_guide/routes.html#routes-load-balancing-for-AB-testing

The better fix is to have the router default to round-robin for routes that have alternate backends and no `haproxy.router.openshift.io/balance` annotation set.

I've confirmed `oc set route-backends` also does not add this annotation, so the problem is not specific to the web console. Updating the web console will not fix any existing routes.

Ok, that's a reasonable argument.  I'll see if we can change the template to accomodate.

Also note that if service A (25%) has 4 pods and service B (75%) has 1, the traffic is no longer balanced at 25/75. The percentage that is asked for by the user is no longer valid. This could be problematic and confusing to the end user when using multiple pods under their services.

Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/0ddebc1a464010ae5919120366d7be3500af00c5
Changed the router to default to roundrobin with multiple services

If the route is associated with multiple services then we will set the
default load balance policy to RoundRobin if no policy is set with an
annotation or as a global default with an environment variable.
Without this change the user would need to both set the services,
weights, and then set an annotation to change the default balancing
algorithm... which people almost always forgot to do.

For bug 1416869 (https://bugzilla.redhat.com/show_bug.cgi?id=1416869)

This has been merged into ocp and is in OCP v3.5.0.17 or newer.

I just test on latest OCP env v3.5.0.17
ose-haproxy-router    v3.5.0.17           6a86c1d87ea7

Seems the image doesn't contain the code change. Pls. help to check it

Sorry about that.  Still working through issues with our merge script.  Should be in the next build.

This has been merged into ocp and is in OCP v3.5.0.18 or newer.  I made sure of it this time.

Test on latest OCP v3.5.0.18
ose-haproxy-router    v3.5.0.18           109538c1aad4
The route with multiple service will set load balance policy to RoundRobin by default. Move bug to verified.

Thanks, @Troy

Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/a4815c6314f9df1d2ce8060216d0924181c48b6c
Changed the router default to roundrobin if non-zero weights are used

If the route has non-zero weights set for the services it is
associated with, then we will set the default load balance policy to
RoundRobin if no policy is set with an annotation or as a global
default with an environment variable. Without this change the user
would need to both set the weights, and then set an annotation to
change the default balancing algorithm... which people almost always
forgot to do.

For bug 1416869 (https://bugzilla.redhat.com/show_bug.cgi?id=1416869)

Weibin: Can you verify that the test cases handle the following cases:
  1 route with two services with weights set to 1 and 9, one service gets 10% the other gets 90%.  No annotations should be set on the route.

 1 route with two services with weights set to 0 and 1.  The 1 route gets 100% of the traffic.  (No annotations should be set)

The other test I want is for roundrobin vs leastconn.  In the previous case with a service with weight 0, we should be using leastconn.  As soon as the weight becomes non-zero we should swap to roundrobin in the generated haproxy.conf in the router.

(In reply to Ben Bennett from comment #19)
> Weibin: Can you verify that the test cases handle the following cases:
>   1 route with two services with weights set to 1 and 9, one service gets
> 10% the other gets 90%.  No annotations should be set on the route.

weliang: Test passed as expected.

>  1 route with two services with weights set to 0 and 1.  The 1 route gets
> 100% of the traffic.  (No annotations should be set)

weliang: Test passed as expected.

> The other test I want is for roundrobin vs leastconn.  In the previous case
> with a service with weight 0, we should be using leastconn.  As soon as the
> weight becomes non-zero we should swap to roundrobin in the generated
> haproxy.conf in the router.

weliang: still use roundrobin not leastconn

[root@ip-172-18-9-212 ~]# oc get route
NAME               HOST/PORT                                          PATH      SERVICES                                        PORT               TERMINATION   WILDCARD
docker-registry    docker-registry-default.0209-l4a.qe.rhcloud.com              docker-registry                                 5000-tcp           passthrough   None
registry-console   registry-console-default.0209-l4a.qe.rhcloud.com             registry-console                                registry-console   passthrough   None
unsecure-route     unsecure-route-default.0209-l4a.qe.rhcloud.com               service-unsecure(100%),service-unsecure-2(0%)   http                             None
[root@ip-172-18-9-212 ~]# oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
caddy-docker               1/1       Running   0          2m
caddy-docker-2             1/1       Running   0          2m
caddy-docker-3             1/1       Running   0          2m
docker-registry-2-857b4    1/1       Running   0          2h
docker-registry-2-n6b4h    1/1       Running   2          2h
registry-console-1-rfhch   1/1       Running   0          2h
router-1-3hvt4             1/1       Running   0          2h
router-1-n4b4p             1/1       Running   0          2h
[root@ip-172-18-9-212 ~]# oc rsh router-1-3hvt4
sh-4.2$ more haproxy.config 
-----------
# Plain http backend
backend be_http_default_unsecure-route
    
  mode http
  option redispatch
  option forwardfor
    
      
  balance roundrobin
      
-----------------------

Weibin: Thanks.  You need the follow-on PR that landed this morning in order to get the proper zero-weight behavior.

(In reply to Ben Bennett from comment #21)
> Weibin: Thanks.  You need the follow-on PR that landed this morning in order
> to get the proper zero-weight behavior.

Test results are:

Case1: with a service with weight 0, roundrobin is shown in haproxy.conf

Case2: Without setting the weight for a service, leastconn is shown in haproxy.conf.

Case3, Fellowing case2, then set the service with non-zero weight, roundrobin is shown in haproxy.conf

Tested in oc v3.5.0.18+9a5d1aa

Super.  Thanks!

This has been merged into ocp and is in OCP v3.5.0.19 or newer.

The scenarios in the comments are works well with latest OCP v3.5.0.19+199197c

Hi, Ben

I just tested on OCP v3.5.0.52 env and found the zero weight behaviour is a little different from before(#comment 22 and 25)

1) with a service with weight 0, leastconn is shown in haproxy.conf
$ oc set route-backends route1
NAME           KIND     TO                  WEIGHT
routes/route1  Service  service-unsecure    0 (0%)
routes/route1  Service  service-unsecure-2  1 (100%)

# Plain http backend
backend be_http_d1_route1   
  mode http
  option redispatch
  option forwardfor      
  balance leastconn

Could you please help to confirm that which behaviour is the proper one? Thanks

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0884

@yandu: https://github.com/openshift/origin/pull/12828 added more fixes, so the correct behavior is that we use LeastConn until there are >1 routes with non-zero weights in which case we use RoundRobin.