Created attachment 703872 [details] jenkins server log Description of problem: After upgrade and migration, can not build an application with jenkins. The jenkins server app logged " Your credentials are not authorized to access "https://10.151.110.187/broker/rest/user"(10.151.110.187 is the ip of the broker and the jenkins server app resides on a different node from broker). The slave app can't be launched. Version-Release number of selected component (if applicable): After upgrading devenv-stage_311 to devenv_2872 How reproducible: Always Steps to Reproduce: 1. Launch devenv-stage_311 ami 2. Create jenkins server app, create other apps with jenkins-client-1.4 added 3. Upgrade from devenv-stage_311 to devenv_2872 4. Migrate rhc-admin-migrate --version 2.0.24 5. Git push an app which has jenkins-client added Actual results: After step 5: Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 272 bytes, done. Total 3 (delta 2), reused 0 (delta 0) remote: restart_on_add=false remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkins1-311t1.dev.rhcloud.com/job/ruby19-build remote: remote: ERROR - Job not available: http://jenkins1-311t1.dev.rhcloud.com/job/ruby19-build remote: !!!!!!!! remote: Deployment Halted! remote: If the build failed before the deploy step, your previous remote: build is still running. Otherwise, your application may be remote: partially deployed or inaccessible. remote: Fix the build and try again. remote: !!!!!!!! To ssh://512ee87414db5147f0000113.rhcloud.com/~/git/ruby19.git/ f1c81eb..97a9416 master -> master The logs in jenkins server app has such warning: WARNING: Caught com.openshift.client.InvalidCredentialsOpenShiftException: Your credentials are not authorized to access "https://10.151.110.187/broker/rest/user". Will retry 4 more times before canceling build. The full log is attached Expected results: Jenkins slave app should be launched to build Additional info: Newly created jenkins server app works fine.
Able to recreate. Jenkins is not restarted on migrate.
Post migration there is some strange behavior restarting. Looks like the migrate thinks jenkins is restarted when it in fact isn't. [root@ip-10-118-66-136 ~]# rhc-admin-migrate --version 2.0.24 Getting all active gears... Getting all logins... .. logins.length: 3 Writing 2 gears for node ip-10-118-66-136_active to file /tmp/rhc/migration/migrate_ip-10-118-66-136_active ##################################################### ##################################################### Migrating gears on node ip-10-118-66-136_active Migrating app 'jenkins' gear 'jenkins' with uuid '512f7b57786e087880000005' on node 'ip-10-118-66-136' for user: bdecoste77e Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bdecoste77e' --migrate-gear '512f7b57786e087880000005' --app-name 'jenkins' --version '2.0.24' Migrating on node... ***time_migrate_on_node_measured_from_broker=5249*** Migrate on node output: DONE: 512f7b57786e087880000005_bdecoste77e_jenkins.migrated 512f7b57786e087880000005_bdecoste77e_jenkins.migrated***time_migrate_on_node_measured_from_node=4227*** ***time_redeploy_httpd_proxy=0*** ***time_restart=0*** ***time_total_migrate_gear_measured_from_broker=5302*** Migrating app 'as7' gear 'as7' with uuid '512f7bc0786e087880000023' on node 'ip-10-118-66-136' for user: bdecoste77e Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bdecoste77e' --migrate-gear '512f7bc0786e087880000023' --app-name 'as7' --version '2.0.24' Migrating on node... ***time_migrate_on_node_measured_from_broker=4572*** Migrate on node output: DONE: 512f7bc0786e087880000023_bdecoste77e_as7.migrated 512f7bc0786e087880000023_bdecoste77e_as7.migrated***time_migrate_on_node_measured_from_node=4293*** ***time_redeploy_httpd_proxy=0*** ***time_restart=0*** ***time_total_migrate_gear_measured_from_broker=4642*** ##################################################### Summary: # of users: 3 # of gears: 2 # of failures: 0 Gear counts per thread: [2] Additional timings: migrate_on_node_measured_from_broker=9.821s migrate_on_node_measured_from_node=8.52s redeploy_httpd_proxy=0.0s restart=0.0s total_migrate_gear_measured_from_broker=9.944s Time gathering users: 0.114s Time gathering active gears: 20.761s Total execution time: 43.175s ##################################################### [root@ip-10-118-66-136 as7]# rhc app restart -p 123456 -a jenkins -d DEBUG: Connecting to https://localhost/broker/rest/api DEBUG: Getting all domains DEBUG: SSL Verification failed -- Using self signed cert DEBUG: SSL Verification failed -- Using self signed cert DEBUG: SSL Verification failed -- Using self signed cert DEBUG: Getting all domains DEBUG: Restarting application jenkins Reference ID: 1473154b2fc7e7314f0b8f76a46c96c7
2013-02-28 12:59:01.572 [ERROR] Reference ID: 412dd4092b3897fe76a17edd414ca3a3 - Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.7/lib/openshift/mcollective_application_container_proxy.rb:2549:in `parse_result' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.7/lib/openshift/mcollective_application_container_proxy.rb:2697:in `run_cartridge_command' /var/www/openshift/broker/lib/online/broker/mcollective_ext.rb:13:in `run_cartridge_command' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.7/lib/openshift/mcollective_application_container_proxy.rb:2660:in `run_cartridge_command_ignore_components' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.7/lib/openshift/mcollective_application_container_proxy.rb:1023:in `restart' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/gear.rb:158:in `method_missing' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/pending_app_op_group.rb:204:in `block in execute' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/pending_app_op_group.rb:144:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/pending_app_op_group.rb:144:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/application.rb:1038:in `run_jobs' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/application.rb:576:in `block in restart' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/application.rb:1080:in `run_in_application_lock' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/application.rb:567:in `restart' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/controllers/app_events_controller.rb:67:in `create' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:459:in `block in _run__2921671840262873673__process_action__3065637204444691853__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_311' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/lib/openshift/controller/action_log.rb:34:in `set_logged_request' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_311' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__2921671840262873673__process_action__3065637204444691853__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call' /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call' /opt/rh/ruby193/root/usr/local/share/gems/gems/mongoid-3.0.19/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call' /opt/rh/ruby193/root/usr/local/share/gems/gems/mongoid-3.0.19/lib/mongoid/unit_of_work.rb:39:in `unit_of_work' /opt/rh/ruby193/root/usr/local/share/gems/gems/mongoid-3.0.19/lib/rack/mongoid/middleware/identity_map.rb:34:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__3714216341062189985__call__945686273669824955__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:517:in `accept_and_process_next_request' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/utils.rb:470:in `safe_fork' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/passenger-spawn-server:102:in `<main>' (pid:3729) 2013-02-28 12:59:01.573 [DEBUG] FAILURE ACTION=RESTART_APPLICATION USER_ID=512f7b37786e087880000001 LOGIN=bdecoste77e APP=jenkins DOMAIN=bdecoste77e APP_UUID=512f7b57786e087880000005 Reference ID: 412dd4092b3897fe76a17edd414ca3a3 (pid:3729) 2013-02-28 12:59:01.575 [INFO ] Completed 500 Internal Server Error in 151ms (Views: 0.5ms) (pid:3729)
stop_component results in a method_missing. Looks like the upgrade is broken.
This fixes rhc-admin-migrate so it restart jenkins like it should: https://github.com/openshift/li/pull/948
Adam, I did the upgrade process and I got: rubygem-openshift-origin-controller.noarch 1.5.8-1.el6oso when I see: rubygem-openshift-origin-controller-1.5.10-1.el6oso.noarch.rpm As the latest version published. Thoughts?
Rtest migrate from devenv-stage_311 to devenv_2878 and manually update the rhc-admin-migrate with https://github.com/openshift/li/pull/948 When run rhc-admin-migrate, show error: ***time_redeploy_httpd_proxy=0*** ***time_restart=0*** ***time_total_migrate_gear_measured_from_broker=235*** Migrating app 'jenkins1' gear 'jenkins1' with uuid '513009b9c6b6c7c6b6000034' on node 'ip-10-144-76-35' for user: jhou Failed to migrate with cmd: '/usr/bin/rhc-admin-migrate --login 'jhou' --migrate-gear '513009b9c6b6c7c6b6000034' --app-name 'jenkins1' --version '2.0.24'' after 2 tries with exception: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.8/lib/openshift/mcollective_application_container_proxy.rb:2539:in `parse_result'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.8/lib/openshift/mcollective_application_container_proxy.rb:2687:in `run_cartridge_command'", "/var/www/openshift/broker/lib/online/broker/mcollective_ext.rb:13:in `run_cartridge_command'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.8/lib/openshift/mcollective_application_container_proxy.rb:2650:in `run_cartridge_command_ignore_components'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.8/lib/openshift/mcollective_application_container_proxy.rb:1059:in `status'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.10/app/models/gear.rb:108:in `status'", "/usr/bin/rhc-admin-migrate:142:in `restart_component'", "/usr/bin/rhc-admin-migrate:112:in `block (2 levels) in migrate_gear'", "/usr/bin/rhc-admin-migrate:102:in `each'", "/usr/bin/rhc-admin-migrate:102:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:68:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:560:in `block in migrate_from_file'", "/usr/bin/rhc-admin-migrate:558:in `each'", "/usr/bin/rhc-admin-migrate:558:in `migrate_from_file'", "/usr/bin/rhc-admin-migrate:701:in `<main>'"] Output: Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'jhou' --migrate-gear '513009b9c6b6c7c6b6000034' --app-name 'jenkins1' --version '2.0.24' Migrating on node... ***time_migrate_on_node_measured_from_broker=127*** Migrate on node output: ***time_migrate_on_node_measured_from_node=0*** ============================ [root@ip-10-152-165-26 ~]# rpm -qa|grep rubygem-openshift-origin-controlle rubygem-openshift-origin-controller-1.5.10-1.git.15.99a7e52.el6.noarch ======================== And can't do any operation to applicatin, same error as comment3 [qgong@dhcp-10-92 dev]$ rhc app stop -a jbossews1s -d DEBUG: Connecting to https://ec2-54-234-75-154.compute-1.amazonaws.com/broker/rest/api DEBUG: Getting all domains DEBUG: SSL Verification failed -- Using self signed cert DEBUG: SSL Verification failed -- Using self signed cert DEBUG: Getting all domains DEBUG: Stopping application jbossews1s force-false Reference ID: 491dcf82a98882ebc0f81667369e979f
I tried again with the correct packages published. If you add a step 3a) service mcollective restart And that should always be done as part of upgrades. The restart works fine. The remaining issue was a change made with auth tokens that changed the key from login to user_id. This is on line 34 and 71 in broker_key.rb.
Retest on the evn that migrate from devenv_stage_312 to devenv_2888 1.rhc-admin-migrate: ***time_redeploy_httpd_proxy=0*** ***time_restart=0*** ***time_total_migrate_gear_measured_from_broker=456*** Migrating app 'jenkins1' gear 'jenkins1' with uuid '5133ff579165aca9600000ae' on node 'ip-10-12-75-150' for user: qgong Failed to migrate with cmd: '/usr/bin/rhc-admin-migrate --login 'qgong' --migrate-gear '5133ff579165aca9600000ae' --app-name 'jenkins1' --version '2.0.24'' after 2 tries with exception: undefined local variable or method `result' for main:Object ["/usr/bin/rhc-admin-migrate:143:in `restart_component'", "/usr/bin/rhc-admin-migrate:112:in `block (2 levels) in migrate_gear'", "/usr/bin/rhc-admin-migrate:102:in `each'", "/usr/bin/rhc-admin-migrate:102:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:68:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:560:in `block in migrate_from_file'", "/usr/bin/rhc-admin-migrate:558:in `each'", "/usr/bin/rhc-admin-migrate:558:in `migrate_from_file'", "/usr/bin/rhc-admin-migrate:701:in `<main>'"] Output: Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'qgong' --migrate-gear '5133ff579165aca9600000ae' --app-name 'jenkins1' --version '2.0.24' Migrating on node... ***time_migrate_on_node_measured_from_broker=168*** Migrate on node output: ***time_migrate_on_node_measured_from_node=4*** 2.Jenkins build application, get the error in jenkins.log WARNING: Caught com.openshift.client.InvalidCredentialsOpenShiftException: Your credentials are not authorized to access "https://10.114.11.209/broker/rest/user". Will retry 4 more times before canceling build. com.openshift.client.InvalidCredentialsOpenShiftException: Your credentials are not authorized to access "https://10.114.11.209/broker/rest/user" at com.openshift.internal.client.RestService.request(RestService.java:99) at com.openshift.internal.client.RestService.request(RestService.java:91) at com.openshift.internal.client.RestService.request(RestService.java:76) at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:137) at com.openshift.internal.client.APIResource$GetUserRequest.execute(APIResource.java:234) at com.openshift.internal.client.APIResource.getUser(APIResource.java:93) at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:840) at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:433) at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:404) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:227) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:310) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: com.openshift.internal.client.httpclient.UnauthorizedException: HTTP Basic: Access denied.
Cannot work on latest INT, output as below: Counting objects: 3, done. Compressing objects: 100% (2/2), done. Writing objects: 100% (2/2), 247 bytes, done. Total 2 (delta 1), reused 0 (delta 0) remote: restart_on_add=false remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkins1-jhou.int.rhcloud.com/job/python1-build remote: remote: ERROR - Job not available: http://jenkins1-jhou.int.rhcloud.com/job/python1-build remote: !!!!!!!! remote: Deployment Halted! remote: If the build failed before the deploy step, your previous remote: build is still running. Otherwise, your application may be remote: partially deployed or inaccessible. remote: Fix the build and try again. remote: !!!!!!!! To ssh://511f0fe26cec0ed7d2000940.rhcloud.com/~/git/python1.git/ a2e0505..00b84e3 master -> master
Sorry about 1) from comment 9... I shouldn't have trusted the pull request from comment 5 fixed the whole problem. Restart has been fixed in: 6c77d014914c14cab5e30e910e7478bf9ef54a3f
Fixed 2) in https://github.com/openshift/origin-server/pull/1536
Commits pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/1622ff9daa625eb96c3f3061d2593cf9ca34cb12 Bug 916559 - Existing broker keys broken after stage upgrade Broker keys used to store login. Code in sprint24 changed it to use user UUID. This broke existing broker keys. Reverting back to original behavior and will update in the future (for identity support where login is != unique/present) https://github.com/openshift/origin-server/commit/53d45c97b2b14fae0e12254a26cb3edd4394af83 Merge pull request #1536 from smarterclayton/bug_916559_existing_broker_keys_broken Bug 916559 - Existing broker keys broken after stage upgrade
Verified after upgrading from devenv-stage_312 to devenv_2895 Jenkins build was OK after migration 1 file changed, 1 insertion(+) Warning: Permanently added 'python1-312t1.dev.rhcloud.com,54.224.31.196' (RSA) to the list of known hosts. Counting objects: 5, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 272 bytes, done. Total 3 (delta 2), reused 0 (delta 0) remote: restart_on_add=false remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkins1-312t1.dev.rhcloud.com/job/python1-build remote: remote: Waiting for build to schedule.........................................Done remote: Waiting for job to complete.......................................Done remote: SUCCESS remote: New build has been deployed. To ssh://51355d8bf6d5500cfb00005c.rhcloud.com/~/git/python1.git/ f135711..a6baea1 master -> master