Description of problem: Jenkins build will fail when using haproxy as load balancer fronting multiple OpenShift brokers Version-Release number of selected component (if applicable): 1.2/2013-08-22.2 puddle How reproducible: Always Steps to Reproduce: 1. Install 2 brokers, and make srue they are using the same named,activemq,datastore. 2. Install 1 node. 3. Against the 2 brokers, create app individually. Make sure the 2 brokers are working well. 4. According to https://access.redhat.com/site/solutions/455673, modify /etc/httpd/conf.d/000002_openshift_origin_broker_proxy.conf on the two brokers. # cat /etc/httpd/conf.d/000002_openshift_origin_broker_proxy.conf <VirtualHost *:80> # ServerName we will inherit from other config; # ServerAlias is to make sure "localhost" traffic goes here regardless. ServerAlias localhost ServerAdmin root@localhost DocumentRoot /var/www/html RequestHeader set X_FORWARDED_PROTO 'https' RequestHeader set Front-End-Https "On" ProxyTimeout 300 ProxyPass /console http://127.0.0.1:8118/console ProxyPassReverse /console http://127.0.0.1:8118/console ProxyPass /broker http://127.0.0.1:8080/broker ProxyPassReverse /broker http://127.0.0.1:8080/broker </VirtualHost> 5. On 2 brokers, restart httpd service. 6. On 2 brokers, access rest api via http. # curl http://127.0.0.1/broker/rest/api 7. Install a new machine, access http://haproxy.1wt.eu/ to downoad haproxy 1.5 or 1.5+ version. # wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev12.tar.gz # tar xzf haproxy-1.5-dev12.tar.gz # cd haproxy-1.5-dev12/ # make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 # make PREFIX=/opt/haproxy-ssl install 8.Create ssl cert file. # openssl req -new -x509 -days 365 -nodes -out haproxy-cert.crt -keyout haproxy-cert.key # cat haproxy-cert.crt haproxy-cert.key > /root/haproxy.pem 9.Create the following haproxy.cfg # cat haproxy.cfg frontend ft_test mode http bind 0.0.0.0:443 ssl crt /root/haproxy.pem default_backend bk_test backend bk_test mode http server srv1 10.66.78.201:80 # broker1's IP server srv2 10.66.78.199:80 # broker2's IP 10. Start haproxy # /opt/haproxy-ssl/sbin/haproxy -f haproxy.cfg -d 11. On the node, modify /etc/openshift/env/OPENSHIFT_BROKER_HOST, point it to haproxy machine's IP. # cat /etc/openshift/env/OPENSHIFT_BROKER_HOST 10.66.78.205 12. Create an app with jenkins-client enabled 13. Do some change in app'repo, git push to trigger jenkins build. 14. On the two brokers, monitor broker's log. # tailf /var/log/openshift/broker/production.log Actual results: Jenkins build failed, and "Broker key authentication failed" is seen in log file of one of 2 brokers. # tailf /var/log/openshift/broker/production.log Started POST "/broker/rest/domains/jialiu/applications/scaphpappbldr/events" for 127.0.0.1 at 2013-09-05 04:14:19 -0400 Processing by AppEventsController#create as JSON Parameters: {"event"=>"stop", "domain_id"=>"jialiu", "application_id"=>"scaphpappbldr"} Broker key authentication failed. bad decrypt /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/auth/broker_key.rb:57:in `final' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/auth/broker_key.rb:57:in `validate_broker_key' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/auth/broker_key.rb:22:in `authenticate_request' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:160:in `authenticate_broker_key' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:152:in `block (2 levels) in authentication_types' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:148:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:148:in `block in authentication_types' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:52:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:52:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:52:in `find' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/authentication.rb:52:in `authenticate_user!' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:452:in `block in _run__2897473546292892725__process_action__357735668347465333__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_345' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/lib/openshift/controller/action_log.rb:34:in `set_logged_request' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_345' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__2897473546292892725__process_action__357735668347465333__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call' /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__466272219800521290__call__4529186595921756845__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/request_handler.rb:97:in `process_request' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:521:in `accept_and_process_next_request' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/utils.rb:470:in `safe_fork' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:180:in `start' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:129:in `start' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize' <internal:prelude>:10:in `synchronize' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>' Filter chain halted as :authenticate_user! rendered or redirected Completed 401 Unauthorized in 8ms Filter chain halted as :authenticate_user! rendered or redirected Completed 401 Unauthorized in 8ms Expected results: Jenkins build should pass. Additional info:
Broker keys and AUTH_SALT need to be the same across all the brokers, c.f. https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/2/html-single/Deployment_Guide/index.html#Configuring_the_Required_Services "Note: The AUTH_PRIV_KEY_FILE, AUTH_PRIV_KEY_PASS and AUTH_PUB_KEY_FILE settings must specify the same private key on all associated brokers for the Jenkins authentication to work." If this is not the case, you will see exactly these symptoms (or a slightly different one, "padding error" but with the same stack trace in production.log). Can you comment on whether this note was followed? Three things will actually break if not: 1. Jenkins 2. Auto-scaling 3. Registering deployments to the broker ... all for the same reason. #3 wasn't a concern when this bug was filed, and #2 can easily go unnoticed. The docs need updating to call this out better. One other thing that I suspect could go wrong is that since Jenkins is using the Java client, it could stop connecting if the SSL cert it's getting from the load balancer is different from the URL of the load balancer. I expect that would look different, though.
Indeed follow the instruction, "Note: The AUTH_PRIV_KEY_FILE, AUTH_PRIV_KEY_PASS and AUTH_PUB_KEY_FILE settings must specify the same private key on all associated brokers for the Jenkins authentication to work." Jenkins build and auto-scaling is working well, seem like it is a configuration issue from me. Thanks for your comments.
I'm closing this based on Comment #6. Thanks for all the help.