Bug 1366180 - [networking_public_107]reencrypt route will always wake up the idled service
Summary: [networking_public_107]reencrypt route will always wake up the idled service
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.3.0
Hardware: All
OS: All
medium
medium
Target Milestone: ---
: ---
Assignee: Solly Ross
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On:
Blocks: 1379552
TreeView+ depends on / blocked
 
Reported: 2016-08-11 09:01 UTC by zhaozhanqi
Modified: 2016-09-27 06:43 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1379552 (view as bug list)
Environment:
Last Closed: 2016-09-19 15:17:21 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC
Origin (Github) 10420 None None None 2016-08-17 20:23:53 UTC

Description zhaozhanqi 2016-08-11 09:01:17 UTC
Description of problem:
When created one reencrypt route. the mapping idled service will be waked up

Version-Release number of selected component (if applicable):
# openshift version
openshift v3.3.0.18
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

haproxy router images(2d96f765472c)

How reproducible:
always

Steps to Reproduce:
1. Create app pod/service
   oc create -f https://raw.githubusercontent.com/zhaozhanqi/v3-testfiles/route_validate/routing/service_pod.json
2. idle the service
  $ oc idle test-service

3. check the endpoints
  $ oc get endpoints
NAME           ENDPOINTS   AGE
test-service   <none>      24m

4. Create reencrept route
   oc create -f https://raw.githubusercontent.com/zhaozhanqi/v3-testfiles/route_validate/routing/route_reencrypt.json

5. Check the endpoints
 $oc get endpoints ; 
NAME           ENDPOINTS                     AGE
test-service   10.1.0.3:8443,10.1.0.5:8443   25m

6. idle the service again
   $ oc idle test-service

7. Given 5 second pass and check the endpoints
# oc get endpoints ; 
NAME           ENDPOINTS                     AGE
test-service   10.1.0.3:8443,10.1.0.5:8443   26m

Actual ressults:

Expected results:

should not wake up the idle service when reencrpt route is created.

Additional info:

Comment 1 Solly Ross 2016-08-15 15:08:45 UTC
We've determined that this is due to a the way HAProxy does health checks.  With a backend which communicates over HTTP, it partially opens a TCP connection, but does not finish, or send a payload.  With a backend which communicates over TLS, it tries to open a TLS connection.  Opening a TLS connection requires a full TCP connection with a payload, so we interpret that as a wake-up event (seeing the full connection).

We are currently evaluating options to solve this.

Comment 3 Andy Goldstein 2016-08-15 20:09:16 UTC
Actual PR https://github.com/openshift/origin/pull/10420

Comment 4 openshift-github-bot 2016-08-19 13:36:30 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/814255a16372be23ec07a42ca7343fa1eb6965f9
HAProxy Router: Don't health-check idled services

Previously, all "endpoints" were health-checked by HAProxy.  For
backends which connected via non-secured connections, this entailed
partial establishment of a TCP connection by HAProxy, which did not
trigger the unidler proxy.  However, for secured backends, the health
check involved attempting to start a TLS connection, which used a full
TCP connection, which caused a wakeup to occur.

This commit adds an extra field to the template data's endpoint
structure which indicates that a particular endpoint is to be considered
an "idled" endpoint, meaning that no health check is necessary.  The
default template then uses this to disable health checking on the given
backend endpoints, which actually point to service IPs backed by the
unidling proxy.

Fixes bug 1366180

Comment 5 zhaozhanqi 2016-08-23 05:30:27 UTC
tested this issue on 

# openshift version
openshift v3.3.0.24-dirty
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

with haproxy image:
openshift3/ose-haproxy-router          v3.3.0.24           c1a1a20e6301

and this issue still can be reproduced. 

When create reencrypt route. the idled service will be waked up in 2 mins in my env.

Comment 6 openshift-github-bot 2016-08-24 04:50:02 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/b08af4a7b7a9d9e216a46f4bbed4cf8f24e79bcb
HAProxy Router: Invert health-check idled check

The check for no endpoints got inverted at some point during the PR,
causing health checks to be enabled *only* when a service was idled,
instead of the other way around.  This fixes that.

Fixes bug 1366180

Comment 7 Meng Bo 2016-08-25 09:33:25 UTC
Tested on OSE build v3.3.0.25 with router image openshift3/ose-haproxy-router v3.3.0.25 d95ecbdbc039

Issue has been fixed. The reencrypt route will not bring the idle service up.


Note You need to log in before you can comment on or make changes to this bug.