Bug 1366180

Summary: [networking_public_107]reencrypt route will always wake up the idled service
Product: OpenShift Container Platform Reporter: zhaozhanqi <zzhao>
Component: NetworkingAssignee: Solly Ross <sross>
Status: CLOSED CURRENTRELEASE QA Contact: Meng Bo <bmeng>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.3.0CC: agoldste, aos-bugs, bbennett, eparis, sross, tdawson
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1379552 (view as bug list) Environment:
Last Closed: 2016-09-19 15:17:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1379552    

Description zhaozhanqi 2016-08-11 09:01:17 UTC 
Description of problem:
When created one reencrypt route. the mapping idled service will be waked up

Version-Release number of selected component (if applicable):
# openshift version
openshift v3.3.0.18
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

haproxy router images(2d96f765472c)

How reproducible:
always

Steps to Reproduce:
1. Create app pod/service
   oc create -f https://raw.githubusercontent.com/zhaozhanqi/v3-testfiles/route_validate/routing/service_pod.json
2. idle the service
  $ oc idle test-service

3. check the endpoints
  $ oc get endpoints
NAME           ENDPOINTS   AGE
test-service   <none>      24m

4. Create reencrept route
   oc create -f https://raw.githubusercontent.com/zhaozhanqi/v3-testfiles/route_validate/routing/route_reencrypt.json

5. Check the endpoints
 $oc get endpoints ; 
NAME           ENDPOINTS                     AGE
test-service   10.1.0.3:8443,10.1.0.5:8443   25m

6. idle the service again
   $ oc idle test-service

7. Given 5 second pass and check the endpoints
# oc get endpoints ; 
NAME           ENDPOINTS                     AGE
test-service   10.1.0.3:8443,10.1.0.5:8443   26m

Actual ressults:

Expected results:

should not wake up the idle service when reencrpt route is created.

Additional info:
Comment 1 Solly Ross 2016-08-15 15:08:45 UTC 
We've determined that this is due to a the way HAProxy does health checks.  With a backend which communicates over HTTP, it partially opens a TCP connection, but does not finish, or send a payload.  With a backend which communicates over TLS, it tries to open a TLS connection.  Opening a TLS connection requires a full TCP connection with a payload, so we interpret that as a wake-up event (seeing the full connection).

We are currently evaluating options to solve this.
Comment 3 Andy Goldstein 2016-08-15 20:09:16 UTC 
Actual PR https://github.com/openshift/origin/pull/10420
Comment 4 openshift-github-bot 2016-08-19 13:36:30 UTC 
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/814255a16372be23ec07a42ca7343fa1eb6965f9
HAProxy Router: Don't health-check idled services

Previously, all "endpoints" were health-checked by HAProxy.  For
backends which connected via non-secured connections, this entailed
partial establishment of a TCP connection by HAProxy, which did not
trigger the unidler proxy.  However, for secured backends, the health
check involved attempting to start a TLS connection, which used a full
TCP connection, which caused a wakeup to occur.

This commit adds an extra field to the template data's endpoint
structure which indicates that a particular endpoint is to be considered
an "idled" endpoint, meaning that no health check is necessary.  The
default template then uses this to disable health checking on the given
backend endpoints, which actually point to service IPs backed by the
unidling proxy.

Fixes bug 1366180
Comment 5 zhaozhanqi 2016-08-23 05:30:27 UTC 
tested this issue on 

# openshift version
openshift v3.3.0.24-dirty
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

with haproxy image:
openshift3/ose-haproxy-router          v3.3.0.24           c1a1a20e6301

and this issue still can be reproduced. 

When create reencrypt route. the idled service will be waked up in 2 mins in my env.
Comment 6 openshift-github-bot 2016-08-24 04:50:02 UTC 
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/b08af4a7b7a9d9e216a46f4bbed4cf8f24e79bcb
HAProxy Router: Invert health-check idled check

The check for no endpoints got inverted at some point during the PR,
causing health checks to be enabled *only* when a service was idled,
instead of the other way around.  This fixes that.

Fixes bug 1366180
Comment 7 Meng Bo 2016-08-25 09:33:25 UTC 
Tested on OSE build v3.3.0.25 with router image openshift3/ose-haproxy-router v3.3.0.25 d95ecbdbc039

Issue has been fixed. The reencrypt route will not bring the idle service up.