Bug 982499

Summary: Failed to migrate applications that have postgresql-8.4 added
Product: OpenShift Online Reporter: Jianwei Hou <jhou>
Component: PodAssignee: Krishna Raman <kraman>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bmeng, mfisher, rpenta
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-22 15:22:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
migration log none

Description Jianwei Hou 2013-07-09 07:58:25 UTC
Created attachment 770835 [details]
migration log

Description of problem:
During migration, found there are 7 out of 60 gears failed to migrate. Investigated these applications/gears, and found all the applications failed to migrate have postgresql-8.4 added.


Version-Release number of selected component (if applicable):
On devenv_3463

How reproducible:
Always

Steps to Reproduce:
1. Create applications(both scalable and non-scalable) with postgresql-8.4 added on a devenv-stage_384 instance
2. Upgrade to lasted devenv instance, which is devenv_3463 currently
2.1 scp /etc/yum.repos.d/devenv.repo from devenv_3464 to devenv-stage_384
2.2 scp -r /root/devenv-local from devenv_3464 to devenv-stage_384
2.3 scp /etc/yum.repos.d/local.repo from devenv_3464 to devenv-stage_384
2.4 On devenv-stage_384 yum -y update --enablerepo devenv-local
3. Follow release ticket 2.0.30 to execute required actions
4. Reboot
5. oo-admin-upgrade --version 2.0.30

Actual results:
Applications that have postgresql-8.4 associated failed to migrate, here is one sample:
Migrating gear on node with: /usr/sbin/oo-admin-upgrade --login 'jhou' --upgrade-gear '318537485061877600878592' --app-name 'python27' --version '2.0.30' --ignore-cartridge-version
Upgrading on node...
***time_upgrade_on_node_measured_from_broker=2451***
Upgrade on node output:
 Beginning 2.0.30 upgrade for 318537485061877600878592
Inspecting gear at /var/lib/openshift/318537485061877600878592
Migrating gear at /var/lib/openshift/318537485061877600878592
No upgrade required for cartridge redhat:python:2.7:0.0.2, already at latest version 0.0.2.
Creating migration instruction validate_gear
Stopping gear with uuid '318537485061877600878592' on node 'ip-10-138-31-22'
Caught an exception during upgrade: wrong number of arguments (0 for 1)
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/application_container_ext/cartridge_actions.rb:395:in `stop'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/upgrade.rb:401:in `stop_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/upgrade.rb:232:in `block (3 levels) in upgrade_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `block in each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/v2_cart_model.rb:941:in `block in process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/v2_cart_model.rb:939:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/v2_cart_model.rb:939:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/v2_cart_model.rb:76:in `each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/upgrade.rb:188:in `block (2 levels) in upgrade_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/upgrade.rb:187:in `chdir'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/upgrade.rb:187:in `block in upgrade_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/utils/cgroups.rb:89:in `with_no_cpu_limits'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/upgrade.rb:186:in `upgrade_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.6/lib/openshift-origin-node/model/upgrade.rb:117:in `upgrade'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:221:in `upgrade_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'
Upgrade on node exit code: 1


Expected results:
Should be able to migrate without failure

Additional info:
Attached migration log

Comment 1 Meng Bo 2013-07-10 06:02:03 UTC
Checked on INT(devenv_3470), no sure if the bug blocked the migration of postgresql-8.4.

For the existing non-scale app with postgresql-8.4 added, there is no OPENSHIFT_POSTGRESQL_DB_VERSION env added after migration.

Comment 2 Meng Bo 2013-07-10 06:08:48 UTC
Oh, please ignore the previous comment, 
the env var is there with name OPENSHIFT_POSTGRES_VERSION

Comment 3 Krishna Raman 2013-07-10 18:15:09 UTC
Unable to reproduce this with latest code.

Comment 4 Jianwei Hou 2013-07-11 06:39:07 UTC
This is not reproducible after upgraded to devenv_3482, the apps with postgresql-8.4 can be migrated successfully.