Bug 1025179

Summary: Met undefined method `disconnect' error during datastore compatiable migration
Product: OpenShift Online Reporter: Jianwei Hou <jhou>
Component: PodAssignee: Clayton Coleman <ccoleman>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bmeng, ccoleman, dmcphers
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-24 03:28:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
mongodump none

Description Jianwei Hou 2013-10-31 07:45:40 UTC
Created attachment 817739 [details]
mongodump

Description of problem:
The error below is seen during datastore compatiable migration in version 2.0.35
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3511:in `ensure in execute_parallel_jobs_impl': undefined method `disconnect' for nil:NilClass (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3511:in `execute_parallel_jobs_impl'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/lib/openshift/application_container_proxy.rb:62:in `execute_parallel_jobs'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/app/models/remote_job.rb:66:in `execute_parallel_jobs'
	from /usr/bin/rhc-admin-migrate-datastore:63:in `expose_haproxy_port'
	from /usr/bin/rhc-admin-migrate-datastore:22:in `migrate'
	from /usr/bin/rhc-admin-migrate-datastore:222:in `<main>'


Version-Release number of selected component (if applicable):
On devenv_3973

How reproducible:
Always

Steps to Reproduce:
1. Prepare applications in devenv-stage_528 instance
2. Upgrade the devent-stage instance to latest as devenv_3973
3. After upgrade, clear broker cache, restart broker and mcollective services
4. Do datastore migration
rhc-admin-migrate-datastore --compatible --version 2.0.35

Actual results:
After step 4:
[root@ip-10-196-171-192 ~]# /usr/bin/rhc-admin-migrate-datastore --compatible --version  2.0.35
Starting migration: compatible
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3511:in `ensure in execute_parallel_jobs_impl': undefined method `disconnect' for nil:NilClass (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3511:in `execute_parallel_jobs_impl'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/lib/openshift/application_container_proxy.rb:62:in `execute_parallel_jobs'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/app/models/remote_job.rb:66:in `execute_parallel_jobs'
	from /usr/bin/rhc-admin-migrate-datastore:63:in `expose_haproxy_port'
	from /usr/bin/rhc-admin-migrate-datastore:22:in `migrate'
	from /usr/bin/rhc-admin-migrate-datastore:222:in `<main>'


Expected results:
Should be successful.

Additional info:
Attached mongodump

Comment 1 Meng Bo 2013-10-31 10:46:24 UTC
Some databases for scalable app cannot be connected after migration, not sure if it is related to this issue.

Comment 2 Dan McPherson 2013-10-31 14:27:07 UTC
The first error:

ESC[0;37m2013-10-31 10:26:21.137ESC[0m [ESC[0;37mDEBUGESC[0m] can't convert nil into Hash (pid:648)
ESC[0;37m2013-10-31 10:26:21.157ESC[0m [ESC[0;37mDEBUGESC[0m] ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3497:in `merge'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.16.4/lib/openshift/mcollective_application_container_proxy.rb:3497:in `execute_parallel_jobs_impl'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/lib/openshift/application_container_proxy.rb:62:in `execute_parallel_jobs'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.9/app/models/remote_job.rb:66:in `execute_parallel_jobs'", "/usr/bin/rhc-admin-migrate-datastore:63:in `expose_haproxy_port'", "/usr/bin/rhc-admin-migrate-datastore:22:in `migrate'", "/usr/bin/rhc-admin-migrate-datastore:222:in `<main>'"] (pid:648)

Comment 3 Clayton Coleman 2013-10-31 15:19:50 UTC
Fixed in https://github.com/openshift/origin-server/pull/4063

Comment 4 openshift-github-bot 2013-10-31 16:46:31 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/3738e6d993291417b3cb0b087b2196fbd6e7d2b4
Bug 1025179 - Defend against code that calls execute_parallel with nil args

Some code paths are calling into remote_job.rb without calling create_parallel_job first.  Defend against that.

Comment 5 Meng Bo 2013-11-01 06:47:43 UTC
Checked after upgrade from devenv-stage_528 to latest stage branch.

There is no such issue when run mongo migrate.

# rhc-admin-migrate-datastore --version 2.0.35 --compatible
Starting migration: compatible

Migration complete
Done!


And the databases for scalable app can be connected after migration.


Move bug to verified.