Bug 916559

Summary:

Existing jenkins server app lost access to broker after upgrade and migration

Product:

OKD

Reporter:

Jianwei Hou <jhou>

Component:

Pod

Assignee:

Clayton Coleman <ccoleman>

Status:

CLOSED CURRENTRELEASE

QA Contact:

libra bugs <libra-bugs>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

2.x

CC:

bmeng, dmcphers, qgong, xtian

Target Milestone:

---

Keywords:

TestBlocker

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2013-03-15 14:22:27 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
jenkins server log	none

Description Jianwei Hou 2013-02-28 10:16:10 UTC

Created attachment 703872 [details]
jenkins server log

Description of problem:
After upgrade and migration, can not build an application with jenkins. The jenkins server app logged " Your credentials are not authorized to access "https://10.151.110.187/broker/rest/user"(10.151.110.187 is the ip of the broker and the jenkins server app resides on a different node from broker). The slave app can't be launched.

Version-Release number of selected component (if applicable):
After upgrading devenv-stage_311 to devenv_2872

How reproducible:
Always

Steps to Reproduce:
1. Launch devenv-stage_311 ami
2. Create jenkins server app, create other apps with jenkins-client-1.4 added
3. Upgrade from devenv-stage_311 to devenv_2872
4. Migrate
rhc-admin-migrate --version 2.0.24
5. Git push an app which has jenkins-client added
  
Actual results:
After step 5:
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 272 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: restart_on_add=false
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins1-311t1.dev.rhcloud.com/job/ruby19-build
remote: 
remote: ERROR - Job not available: http://jenkins1-311t1.dev.rhcloud.com/job/ruby19-build
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
To ssh://512ee87414db5147f0000113.rhcloud.com/~/git/ruby19.git/
   f1c81eb..97a9416  master -> master

The logs in jenkins server app has such warning:
WARNING: Caught com.openshift.client.InvalidCredentialsOpenShiftException: Your credentials are not authorized to access "https://10.151.110.187/broker/rest/user". Will retry 4 more times before canceling build.

The full log is attached

Expected results:
Jenkins slave app should be launched to build

Additional info:
Newly created jenkins server app works fine.

Comment 1 Bill DeCoste 2013-02-28 16:39:57 UTC

Able to recreate. Jenkins is not restarted on migrate.

Comment 2 Bill DeCoste 2013-02-28 16:46:54 UTC

Post migration there is some strange behavior restarting. Looks like the migrate thinks jenkins is restarted when it in fact isn't.

[root@ip-10-118-66-136 ~]# rhc-admin-migrate --version 2.0.24
Getting all active gears...
Getting all logins...
..
logins.length: 3
Writing 2 gears for node ip-10-118-66-136_active to file /tmp/rhc/migration/migrate_ip-10-118-66-136_active
#####################################################
#####################################################
Migrating gears on node ip-10-118-66-136_active

Migrating app 'jenkins' gear 'jenkins' with uuid '512f7b57786e087880000005' on node 'ip-10-118-66-136' for user: bdecoste77e
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bdecoste77e' --migrate-gear '512f7b57786e087880000005' --app-name 'jenkins' --version '2.0.24'
Migrating on node...
***time_migrate_on_node_measured_from_broker=5249***
Migrate on node output:
 DONE: 512f7b57786e087880000005_bdecoste77e_jenkins.migrated 512f7b57786e087880000005_bdecoste77e_jenkins.migrated***time_migrate_on_node_measured_from_node=4227***

***time_redeploy_httpd_proxy=0***
***time_restart=0***
***time_total_migrate_gear_measured_from_broker=5302***
Migrating app 'as7' gear 'as7' with uuid '512f7bc0786e087880000023' on node 'ip-10-118-66-136' for user: bdecoste77e
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'bdecoste77e' --migrate-gear '512f7bc0786e087880000023' --app-name 'as7' --version '2.0.24'
Migrating on node...
***time_migrate_on_node_measured_from_broker=4572***
Migrate on node output:
 DONE: 512f7bc0786e087880000023_bdecoste77e_as7.migrated 512f7bc0786e087880000023_bdecoste77e_as7.migrated***time_migrate_on_node_measured_from_node=4293***

***time_redeploy_httpd_proxy=0***
***time_restart=0***
***time_total_migrate_gear_measured_from_broker=4642***
#####################################################
Summary:
# of users: 3
# of gears: 2
# of failures: 0
Gear counts per thread: [2]
Additional timings:
    migrate_on_node_measured_from_broker=9.821s
    migrate_on_node_measured_from_node=8.52s
    redeploy_httpd_proxy=0.0s
    restart=0.0s
    total_migrate_gear_measured_from_broker=9.944s
Time gathering users: 0.114s
Time gathering active gears: 20.761s
Total execution time: 43.175s
#####################################################




[root@ip-10-118-66-136 as7]# rhc app restart -p 123456 -a jenkins -d
DEBUG: Connecting to https://localhost/broker/rest/api
DEBUG: Getting all domains
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG: Getting all domains
DEBUG: Restarting application jenkins

Reference ID: 1473154b2fc7e7314f0b8f76a46c96c7

Comment 3 Bill DeCoste 2013-02-28 18:00:25 UTC

2013-02-28 12:59:01.572 [ERROR] Reference ID: 412dd4092b3897fe76a17edd414ca3a3 - Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.7/lib/openshift/mcollective_application_container_proxy.rb:2549:in `parse_result'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.7/lib/openshift/mcollective_application_container_proxy.rb:2697:in `run_cartridge_command'
  /var/www/openshift/broker/lib/online/broker/mcollective_ext.rb:13:in `run_cartridge_command'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.7/lib/openshift/mcollective_application_container_proxy.rb:2660:in `run_cartridge_command_ignore_components'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.7/lib/openshift/mcollective_application_container_proxy.rb:1023:in `restart'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/gear.rb:158:in `method_missing'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/pending_app_op_group.rb:204:in `block in execute'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/pending_app_op_group.rb:144:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/pending_app_op_group.rb:144:in `execute'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/application.rb:1038:in `run_jobs'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/application.rb:576:in `block in restart'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/application.rb:1080:in `run_in_application_lock'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/models/application.rb:567:in `restart'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/app/controllers/app_events_controller.rb:67:in `create'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:459:in `block in _run__2921671840262873673__process_action__3065637204444691853__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_311'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.9/lib/openshift/controller/action_log.rb:34:in `set_logged_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_311'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__2921671840262873673__process_action__3065637204444691853__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call'
  /opt/rh/ruby193/root/usr/local/share/gems/gems/mongoid-3.0.19/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'
  /opt/rh/ruby193/root/usr/local/share/gems/gems/mongoid-3.0.19/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'
  /opt/rh/ruby193/root/usr/local/share/gems/gems/mongoid-3.0.19/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__3714216341062189985__call__945686273669824955__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:517:in `accept_and_process_next_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/utils.rb:470:in `safe_fork'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/passenger-spawn-server:102:in `<main>' (pid:3729)
2013-02-28 12:59:01.573 [DEBUG] FAILURE ACTION=RESTART_APPLICATION USER_ID=512f7b37786e087880000001 LOGIN=bdecoste77e APP=jenkins DOMAIN=bdecoste77e APP_UUID=512f7b57786e087880000005 Reference ID: 412dd4092b3897fe76a17edd414ca3a3 (pid:3729)
2013-02-28 12:59:01.575 [INFO ] Completed 500 Internal Server Error in 151ms (Views: 0.5ms) (pid:3729)

Comment 4 Bill DeCoste 2013-02-28 19:44:28 UTC

stop_component results in a method_missing. Looks like the upgrade is broken.

Comment 5 Dan McPherson 2013-03-01 02:01:13 UTC

This fixes rhc-admin-migrate so it restart jenkins like it should:

https://github.com/openshift/li/pull/948

Comment 6 Dan McPherson 2013-03-01 02:37:27 UTC

Adam,

   I did the upgrade process and I got:

rubygem-openshift-origin-controller.noarch                        1.5.8-1.el6oso

when I see:

rubygem-openshift-origin-controller-1.5.10-1.el6oso.noarch.rpm

As the latest version published.

Thoughts?

Comment 7 Rony Gong 🔥 2013-03-01 06:09:09 UTC

Rtest migrate from devenv-stage_311 to devenv_2878
and manually update the rhc-admin-migrate
with https://github.com/openshift/li/pull/948

When run rhc-admin-migrate, show error:

***time_redeploy_httpd_proxy=0***
***time_restart=0***
***time_total_migrate_gear_measured_from_broker=235***
Migrating app 'jenkins1' gear 'jenkins1' with uuid '513009b9c6b6c7c6b6000034' on node 'ip-10-144-76-35' for user: jhou
Failed to migrate with cmd: '/usr/bin/rhc-admin-migrate --login 'jhou' --migrate-gear '513009b9c6b6c7c6b6000034' --app-name 'jenkins1' --version '2.0.24'' after 2 tries with exception: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.8/lib/openshift/mcollective_application_container_proxy.rb:2539:in `parse_result'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.8/lib/openshift/mcollective_application_container_proxy.rb:2687:in `run_cartridge_command'", "/var/www/openshift/broker/lib/online/broker/mcollective_ext.rb:13:in `run_cartridge_command'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.8/lib/openshift/mcollective_application_container_proxy.rb:2650:in `run_cartridge_command_ignore_components'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.5.8/lib/openshift/mcollective_application_container_proxy.rb:1059:in `status'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.5.10/app/models/gear.rb:108:in `status'", "/usr/bin/rhc-admin-migrate:142:in `restart_component'", "/usr/bin/rhc-admin-migrate:112:in `block (2 levels) in migrate_gear'", "/usr/bin/rhc-admin-migrate:102:in `each'", "/usr/bin/rhc-admin-migrate:102:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:68:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:560:in `block in migrate_from_file'", "/usr/bin/rhc-admin-migrate:558:in `each'", "/usr/bin/rhc-admin-migrate:558:in `migrate_from_file'", "/usr/bin/rhc-admin-migrate:701:in `<main>'"]
Output:
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'jhou' --migrate-gear '513009b9c6b6c7c6b6000034' --app-name 'jenkins1' --version '2.0.24'
Migrating on node...
***time_migrate_on_node_measured_from_broker=127***
Migrate on node output:
 ***time_migrate_on_node_measured_from_node=0***

============================

[root@ip-10-152-165-26 ~]# rpm -qa|grep rubygem-openshift-origin-controlle
rubygem-openshift-origin-controller-1.5.10-1.git.15.99a7e52.el6.noarch


========================
And can't do any operation to applicatin,  same error as comment3
[qgong@dhcp-10-92 dev]$ rhc app stop -a jbossews1s -d
DEBUG: Connecting to https://ec2-54-234-75-154.compute-1.amazonaws.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG: Getting all domains
DEBUG: Stopping application jbossews1s force-false

Reference ID: 491dcf82a98882ebc0f81667369e979f

Comment 8 Dan McPherson 2013-03-03 21:10:52 UTC

I tried again with the correct packages published.  If you add a step

3a) service mcollective restart

And that should always be done as part of upgrades.  The restart works fine.  

The remaining issue was a change made with auth tokens that changed the key from login to user_id.  This is on line 34 and 71 in broker_key.rb.

Comment 9 Rony Gong 🔥 2013-03-04 05:06:26 UTC

Retest on the evn that migrate from devenv_stage_312 to devenv_2888
1.rhc-admin-migrate:
***time_redeploy_httpd_proxy=0***
***time_restart=0***
***time_total_migrate_gear_measured_from_broker=456***
Migrating app 'jenkins1' gear 'jenkins1' with uuid '5133ff579165aca9600000ae' on node 'ip-10-12-75-150' for user: qgong
Failed to migrate with cmd: '/usr/bin/rhc-admin-migrate --login 'qgong' --migrate-gear '5133ff579165aca9600000ae' --app-name 'jenkins1' --version '2.0.24'' after 2 tries with exception: undefined local variable or method `result' for main:Object
["/usr/bin/rhc-admin-migrate:143:in `restart_component'", "/usr/bin/rhc-admin-migrate:112:in `block (2 levels) in migrate_gear'", "/usr/bin/rhc-admin-migrate:102:in `each'", "/usr/bin/rhc-admin-migrate:102:in `block in migrate_gear'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:68:in `timeout'", "/usr/bin/rhc-admin-migrate:51:in `migrate_gear'", "/usr/bin/rhc-admin-migrate:560:in `block in migrate_from_file'", "/usr/bin/rhc-admin-migrate:558:in `each'", "/usr/bin/rhc-admin-migrate:558:in `migrate_from_file'", "/usr/bin/rhc-admin-migrate:701:in `<main>'"]
Output:
Migrating gear on node with: /usr/bin/rhc-admin-migrate --login 'qgong' --migrate-gear '5133ff579165aca9600000ae' --app-name 'jenkins1' --version '2.0.24'
Migrating on node...
***time_migrate_on_node_measured_from_broker=168***
Migrate on node output:
 ***time_migrate_on_node_measured_from_node=4***



2.Jenkins build application, get the error in jenkins.log
WARNING: Caught com.openshift.client.InvalidCredentialsOpenShiftException: Your credentials are not authorized to access "https://10.114.11.209/broker/rest/user". Will retry 4 more times before canceling build.
com.openshift.client.InvalidCredentialsOpenShiftException: Your credentials are not authorized to access "https://10.114.11.209/broker/rest/user"
	at com.openshift.internal.client.RestService.request(RestService.java:99)
	at com.openshift.internal.client.RestService.request(RestService.java:91)
	at com.openshift.internal.client.RestService.request(RestService.java:76)
	at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:137)
	at com.openshift.internal.client.APIResource$GetUserRequest.execute(APIResource.java:234)
	at com.openshift.internal.client.APIResource.getUser(APIResource.java:93)
	at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:840)
	at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:433)
	at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:404)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:227)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:310)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: com.openshift.internal.client.httpclient.UnauthorizedException: HTTP Basic: Access denied.

Comment 10 Meng Bo 2013-03-04 10:32:00 UTC

Cannot work on latest INT, output as below:

Counting objects: 3, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 247 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: restart_on_add=false
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins1-jhou.int.rhcloud.com/job/python1-build
remote: 
remote: ERROR - Job not available: http://jenkins1-jhou.int.rhcloud.com/job/python1-build
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
To ssh://511f0fe26cec0ed7d2000940.rhcloud.com/~/git/python1.git/
   a2e0505..00b84e3  master -> master

Comment 11 Dan McPherson 2013-03-04 15:42:14 UTC

Sorry about 1) from comment 9...  I shouldn't have trusted the pull request from comment 5 fixed the whole problem.  Restart has been fixed in:  6c77d014914c14cab5e30e910e7478bf9ef54a3f

Comment 12 Clayton Coleman 2013-03-04 20:25:56 UTC

Fixed 2) in https://github.com/openshift/origin-server/pull/1536

Comment 13 openshift-github-bot 2013-03-04 22:39:36 UTC

Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/1622ff9daa625eb96c3f3061d2593cf9ca34cb12
Bug 916559 - Existing broker keys broken after stage upgrade

Broker keys used to store login.  Code in sprint24 changed it to use user UUID.  This broke existing broker keys.  Reverting back to original behavior and will update in the future (for identity support where login is != unique/present)

https://github.com/openshift/origin-server/commit/53d45c97b2b14fae0e12254a26cb3edd4394af83
Merge pull request #1536 from smarterclayton/bug_916559_existing_broker_keys_broken

Bug 916559 - Existing broker keys broken after stage upgrade

Comment 14 Jianwei Hou 2013-03-05 07:39:05 UTC

Verified after upgrading from devenv-stage_312 to devenv_2895

Jenkins build was OK after migration

 1 file changed, 1 insertion(+)
Warning: Permanently added 'python1-312t1.dev.rhcloud.com,54.224.31.196' (RSA) to the list of known hosts.
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 272 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: restart_on_add=false
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins1-312t1.dev.rhcloud.com/job/python1-build
remote: 
remote: Waiting for build to schedule.........................................Done
remote: Waiting for job to complete.......................................Done
remote: SUCCESS
remote: New build has been deployed.
To ssh://51355d8bf6d5500cfb00005c.rhcloud.com/~/git/python1.git/
   f135711..a6baea1  master -> master