Bug 1005013 - Jenkins build will fail when using haproxy as load balancer fronting multiple OpenShift brokers
Jenkins build will fail when using haproxy as load balancer fronting multiple...
Status: CLOSED NOTABUG
Product: OpenShift Container Platform
Classification: Red Hat
Component: Kubernetes (Show other bugs)
1.2.1
Unspecified Unspecified
low Severity high
: ---
: ---
Assigned To: Miciah Dashiel Butler Masters
libra bugs
:
Depends On: 1008352
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-05 23:00 EDT by Johnny Liu
Modified: 2014-05-13 13:42 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-05-12 10:04:40 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Johnny Liu 2013-09-05 23:00:33 EDT
Description of problem:
Jenkins build will fail when using haproxy as load balancer fronting multiple OpenShift brokers

Version-Release number of selected component (if applicable):
1.2/2013-08-22.2 puddle

How reproducible:
Always

Steps to Reproduce:
1. Install 2 brokers, and make srue they are using the same named,activemq,datastore.

2. Install 1 node.

3. Against the 2 brokers, create app individually. Make sure the 2 brokers are working well.

4. According to https://access.redhat.com/site/solutions/455673, modify /etc/httpd/conf.d/000002_openshift_origin_broker_proxy.conf on the two brokers.
# cat  /etc/httpd/conf.d/000002_openshift_origin_broker_proxy.conf
<VirtualHost *:80>
  # ServerName we will inherit from other config;
  # ServerAlias is to make sure "localhost" traffic goes here regardless.
  ServerAlias localhost
  ServerAdmin root@localhost
  DocumentRoot /var/www/html
  RequestHeader set X_FORWARDED_PROTO 'https'
  RequestHeader set Front-End-Https "On"
  ProxyTimeout 300
  ProxyPass /console http://127.0.0.1:8118/console
  ProxyPassReverse /console http://127.0.0.1:8118/console
  ProxyPass /broker http://127.0.0.1:8080/broker
  ProxyPassReverse /broker http://127.0.0.1:8080/broker
</VirtualHost>

5. On 2 brokers, restart httpd service.

6. On 2 brokers, access rest api via http.
# curl http://127.0.0.1/broker/rest/api

7. Install a new machine, access http://haproxy.1wt.eu/ to downoad haproxy 1.5 or 1.5+ version.
# wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev12.tar.gz
# tar xzf haproxy-1.5-dev12.tar.gz
# cd haproxy-1.5-dev12/
# make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1
# make PREFIX=/opt/haproxy-ssl install

8.Create ssl cert file.
# openssl req -new -x509 -days 365 -nodes -out haproxy-cert.crt -keyout haproxy-cert.key
# cat haproxy-cert.crt haproxy-cert.key > /root/haproxy.pem

9.Create the following haproxy.cfg
# cat haproxy.cfg
frontend ft_test
  mode http
  bind 0.0.0.0:443 ssl crt /root/haproxy.pem
  default_backend bk_test

backend bk_test
  mode http
  server srv1 10.66.78.201:80 # broker1's IP
  server srv2 10.66.78.199:80 # broker2's IP

10. Start haproxy
# /opt/haproxy-ssl/sbin/haproxy -f haproxy.cfg  -d

11. On the node, modify /etc/openshift/env/OPENSHIFT_BROKER_HOST, point it to haproxy machine's IP.
# cat /etc/openshift/env/OPENSHIFT_BROKER_HOST
10.66.78.205

12. Create an app with jenkins-client enabled

13. Do some change in app'repo, git push to trigger jenkins build.

14. On the two brokers, monitor broker's log.
# tailf /var/log/openshift/broker/production.log

Actual results:
Jenkins build failed, and "Broker key authentication failed" is seen in log file of one of 2 brokers.

# tailf /var/log/openshift/broker/production.log
Started POST "/broker/rest/domains/jialiu/applications/scaphpappbldr/events" for 127.0.0.1 at 2013-09-05 04:14:19 -0400
Processing by AppEventsController#create as JSON
  Parameters: {"event"=>"stop", "domain_id"=>"jialiu", "application_id"=>"scaphpappbldr"}
Broker key authentication failed. bad decrypt
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/auth/broker_key.rb:57:in `final'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/auth/broker_key.rb:57:in `validate_broker_key'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/auth/broker_key.rb:22:in `authenticate_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:160:in `authenticate_broker_key'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:152:in `block (2 levels) in authentication_types'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:148:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:148:in `block in authentication_types'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:52:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:52:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:52:in `find'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:52:in `authenticate_user!'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:452:in `block in _run__2897473546292892725__process_action__357735668347465333__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_345'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/action_log.rb:34:in `set_logged_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_345'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__2897473546292892725__process_action__357735668347465333__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__466272219800521290__call__4529186595921756845__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/request_handler.rb:97:in `process_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:521:in `accept_and_process_next_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/utils.rb:470:in `safe_fork'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:180:in `start'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:129:in `start'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
  <internal:prelude>:10:in `synchronize'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>'
Filter chain halted as :authenticate_user! rendered or redirected
Completed 401 Unauthorized in 8ms
Filter chain halted as :authenticate_user! rendered or redirected
Completed 401 Unauthorized in 8ms

Expected results:
Jenkins build should pass.

Additional info:
Comment 5 Luke Meyer 2014-01-20 12:11:46 EST
Broker keys and AUTH_SALT need to be the same across all the brokers, c.f.
https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/2/html-single/Deployment_Guide/index.html#Configuring_the_Required_Services
"Note: The AUTH_PRIV_KEY_FILE, AUTH_PRIV_KEY_PASS and AUTH_PUB_KEY_FILE settings must specify the same private key on all associated brokers for the Jenkins authentication to work." If this is not the case, you will see exactly these symptoms (or a slightly different one, "padding error" but with the same stack trace in production.log).

Can you comment on whether this note was followed? Three things will actually break if not:
1. Jenkins
2. Auto-scaling
3. Registering deployments to the broker
... all for the same reason. #3 wasn't a concern when this bug was filed, and #2 can easily go unnoticed.
The docs need updating to call this out better.

One other thing that I suspect could go wrong is that since Jenkins is using the Java client, it could stop connecting if the SSL cert it's getting from the load balancer is different from the URL of the load balancer. I expect that would look different, though.
Comment 6 Johnny Liu 2014-05-06 01:44:09 EDT
Indeed follow the instruction,
"Note: The AUTH_PRIV_KEY_FILE, AUTH_PRIV_KEY_PASS and AUTH_PUB_KEY_FILE settings must specify the same private key on all associated brokers for the Jenkins authentication to work."

Jenkins build and auto-scaling is working well, seem like it is a configuration issue from me. 

Thanks for your comments.
Comment 7 Brenton Leanhardt 2014-05-12 10:04:40 EDT
I'm closing this based on Comment #6.  Thanks for all the help.

Note You need to log in before you can comment on or make changes to this bug.