Created attachment 817739 [details] mongodump Description of problem: The error below is seen during datastore compatiable migration in version 2.0.35 /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3511:in `ensure in execute_parallel_jobs_impl': undefined method `disconnect' for nil:NilClass (NoMethodError) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3511:in `execute_parallel_jobs_impl' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/lib/openshift/application_container_proxy.rb:62:in `execute_parallel_jobs' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/app/models/remote_job.rb:66:in `execute_parallel_jobs' from /usr/bin/rhc-admin-migrate-datastore:63:in `expose_haproxy_port' from /usr/bin/rhc-admin-migrate-datastore:22:in `migrate' from /usr/bin/rhc-admin-migrate-datastore:222:in `<main>' Version-Release number of selected component (if applicable): On devenv_3973 How reproducible: Always Steps to Reproduce: 1. Prepare applications in devenv-stage_528 instance 2. Upgrade the devent-stage instance to latest as devenv_3973 3. After upgrade, clear broker cache, restart broker and mcollective services 4. Do datastore migration rhc-admin-migrate-datastore --compatible --version 2.0.35 Actual results: After step 4: [root@ip-10-196-171-192 ~]# /usr/bin/rhc-admin-migrate-datastore --compatible --version 2.0.35 Starting migration: compatible /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3511:in `ensure in execute_parallel_jobs_impl': undefined method `disconnect' for nil:NilClass (NoMethodError) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3511:in `execute_parallel_jobs_impl' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/lib/openshift/application_container_proxy.rb:62:in `execute_parallel_jobs' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/app/models/remote_job.rb:66:in `execute_parallel_jobs' from /usr/bin/rhc-admin-migrate-datastore:63:in `expose_haproxy_port' from /usr/bin/rhc-admin-migrate-datastore:22:in `migrate' from /usr/bin/rhc-admin-migrate-datastore:222:in `<main>' Expected results: Should be successful. Additional info: Attached mongodump
Some databases for scalable app cannot be connected after migration, not sure if it is related to this issue.
The first error: ESC[0;37m2013-10-31 10:26:21.137ESC[0m [ESC[0;37mDEBUGESC[0m] can't convert nil into Hash (pid:648) ESC[0;37m2013-10-31 10:26:21.157ESC[0m [ESC[0;37mDEBUGESC[0m] ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3497:in `merge'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3497:in `execute_parallel_jobs_impl'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/lib/openshift/application_container_proxy.rb:62:in `execute_parallel_jobs'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/app/models/remote_job.rb:66:in `execute_parallel_jobs'", "/usr/bin/rhc-admin-migrate-datastore:63:in `expose_haproxy_port'", "/usr/bin/rhc-admin-migrate-datastore:22:in `migrate'", "/usr/bin/rhc-admin-migrate-datastore:222:in `<main>'"] (pid:648)
Fixed in https://github.com/openshift/origin-server/pull/4063
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/3738e6d993291417b3cb0b087b2196fbd6e7d2b4 Bug 1025179 - Defend against code that calls execute_parallel with nil args Some code paths are calling into remote_job.rb without calling create_parallel_job first. Defend against that.
Checked after upgrade from devenv-stage_528 to latest stage branch. There is no such issue when run mongo migrate. # rhc-admin-migrate-datastore --version 2.0.35 --compatible Starting migration: compatible Migration complete Done! And the databases for scalable app can be connected after migration. Move bug to verified.