+++ This bug was initially created as a clone of Bug #1145810 +++ Description of problem: HAProxy does not consider a '401 unauthenticated' status code to be a valid code for health checks. When a scaled application requires Basic HTTP authentication, the HAProxy health checks will therefore fail and mark the server as down. This results in a 503 internal server error response from the application. Version-Release number of selected component (if applicable): 2.1.5 How reproducible: Steps to Reproduce: 1. rhc create-app basicauthtest jbosseap-6 --from-url=https://github.com/etsauer/openshift-quickstart-basic-auth.git -s 2. Navigate to app. Actual results: 503 Error page Expected results: Loging Pop-Up window referencing a 'Test Realm' Additional info: Need to add 401 to list of valid response codes in the HAProxy cartridge. --- Additional comment from RHEL Product and Program Management on 2014-09-23 16:06:38 EDT --- Since this issue was entered in bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. --- Additional comment from Eric Sauer on 2014-09-23 16:09:57 EDT --- Created PR at: https://github.com/openshift/origin-server/pull/5835 --- Additional comment from Miciah Dashiel Butler Masters on 2014-09-23 16:14:32 EDT --- Is this a duplicate of bug 1128242, or do we want to fix both problems (HTTP 401 should be recognised as a successful response in the health check, and the health check should authenticate if configured with credentials)? --- Additional comment from Eric Sauer on 2014-09-23 17:22:43 EDT --- A fix requiring that the user has to 'do something' in order to make this work doesn't quite fit our use case. We are deploying an OSE environment with overridden default templates for each cartridge that already contain Basic Authentication. If we require user intervention, then those applications will be created in a broken state. We would like some way of fixing this globally. Satisfactory options here would be: - Simple hard code addition to haproxy.cnf.erb, (very small change, see PR) - Some kind of global config option that would allow admins to set a list of accepted codes - Delegate a subsection of the haproxy.cnf file as a configuration file. Then admins would be able to customized behvior to include expected response, GET uri for healthchk, authentication info, etc. --- Additional comment from Eric Sauer on 2014-09-23 17:23:15 EDT --- Note, there is also a trello card for this: https://trello.com/c/0X9wpRLF/225-support-configurable-haproxy-url-response-code-for-status-checks --- Additional comment from Luke Meyer on 2014-09-24 09:26:55 EDT --- I have a feeling someone will want to both customize the health check URI and the acceptable response codes. Basically, complete customization of the health check.
I don't think we want to go w/ the PR referenced, we should make this configurable via an env variable the drives the haproxy.cnf settings, as discussed in the trello card.
I like this idea of making this configurable, but if we do it via an environment variable, how can we make that env variable configurable by an admin?
Ben, Eric, I pretty much like the idea described in the Trello card, to make both (url, status code) configurable with some reasonable default. I don't think the 401 should be included in the default, but if you as a developer know that your gear is UP when it is returning 401, then you will have ability to configure it. Eric: For admins, they can modify the haproxy cartridge and set the default to what they want to be the default (if with admin, you mean 'node' admin)
I've made some changes to the PR. The erb now pulls in environment variables to set values for the two items discussed in the trello card. However, I realized that, while this will work for admins, who can set global vars in /etc/openshift/env/, this will not work for users. If someone does a... rhc env set HTTP_CHK_URL="/httpchk.php" .. post app creation, this will have no affect because the erb has already been evaluated and the haproxy.cnf is now more or less static.
yeah you'll need to add something to start/restart that updates it appropriately. in most cases we regenerate from the erb on every start, but in this case we can't do that since we modify the file contents dynamically as we add/remove gears, so you'll have to do something slightly smarter.
Ok, I think I've come up with something. https://github.com/openshift/origin-server/pull/5835
One other concern in changing how the health check works - can we ensure this is communicated in the routing SPI? Otherwise the routing layer won't have this information correct for HA apps (though the routing layer can always be customized out of band by administrators).
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/64bb36e571be31f0c3d4c3553df0cfa2649b056b Bug 1146112 - Added 401 to default expected status codes on health check. This will allow for Basic Auth applications to work in scaled mode. Also provided ability for admins and users to change this and the health check URI values using environment variables. Bug 1146112 - Can now set http-check status codes and GET uri via ENV vars Bug 1146112 - replacing a line that was removed by accident Bug 1146112 - Control script will check for user defined haproxy.cnf variables and set them Bug 1146112 - No need for ternary operator here Bug 1146112 - Per bparees: This came up on the bug scrum and there was agreement that we should accept 401 by default, can you update this to include 401 as part of the default status code list? Bug 1146112 - Changed to double quotes for default values Bug 1146112 - Fixing more syntax issues
Just a note: as implemented, the routing layer will never receive this information and will presumably use the cartridge-specified health check unless otherwise customized. I'm not actually sure how the routing layer even could receive this information. I guess the main use case is setting env vars via the REST API which the broker could theoretically publish, except that AFAIK we don't have an "update_endpoint" event for the routing SPI yet.
Verified on devenv_5219. After create app with "rhc create-app basicauthtest jbosseap-6 --from-url=https://github.com/etsauer/openshift-quickstart-basic-auth.git -s", access it without any auth info input, 401 code will return; it returns "1" if access /health page.