Bug 990948 - Migration is performed duplicate times and target version is recognized as nil
Migration is performed duplicate times and target version is recognized as nil
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Dan McPherson
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-01 05:26 EDT by Jianwei Hou
Modified: 2015-05-14 20:19 EDT (History)
1 user (show)

See Also:
Fixed In Version: devenv-stage_431
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-07 18:58:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
migration output (368.32 KB, text/x-log)
2013-08-01 05:26 EDT, Jianwei Hou
no flags Details

  None (edit)
Description Jianwei Hou 2013-08-01 05:26:16 EDT
Created attachment 781487 [details]
migration output

Description of problem:
Migrate with "oo-admin-upgrade", duplicate upgrade is performed on a same gear several times, the "--version" argument is recognized as nil, and the whole upgrade failed.

Version-Release number of selected component (if applicable):
On devenv-stage_429
openshift-origin-broker-util-1.12.8-1.el6oso.noarch

How reproducible:
Always

Steps to Reproduce:
1. Prepare test data on a stage ami of sprint 30, which was devenv-stage_424
2. Upgrade to lastest devenv-stage instance, devenv-stage_429
3. After upgrade, enable cgroups, restart mcollective, clear broker cache and restart broker
4. Start migration
oo-admin-upgrade --version 2.0.31

Actual results:
Migration failed, according to the log printed to screen, maybe there are duplicate migration running against a same gear, and the '--version' argument is taken as nil. Therefore, all gears failed to migrate. 

Also noticed in the final result, the number of failures is greater than the number of gears:
# of users: 8
# of gears: 62
# of failures: 64
Gear counts per thread: [60, 2]

Please refer to the attachment for detail

Expected results:
Should successfully migrate the gears

Additional info:
This could be directly reproduced on devenv-stage_429, just create one app on it and do the migration.
Comment 2 Jianwei Hou 2013-08-02 01:14:27 EDT
Verified on devenv-stage_435

erforming upgrade with options: {:version=>"2.0.31", :continue=>false, :ignore_cartridge_version=>false, :target_server_identity=>nil, :upgrade_position=>1, :num_upgraders=>1, :rerun=>false}
Getting all active gears...
Getting all logins...
..........................................
logins.length: 8
Writing 60 gears for node ip-10-138-56-220_active to file /tmp/oo-upgrade/upgrade_ip-10-138-56-220_active
Writing 2 gears for node ip-10-138-56-220 to file /tmp/oo-upgrade/upgrade_ip-10-138-56-220
#####################################################
#####################################################
Migrating gears on node ip-10-138-56-220
Migrating gears on node ip-10-138-56-220_active

Migrating app 'stopapp' gear 'stopapp' with uuid '51fb19ad598baa1aad000001' on node 'ip-10-138-56-220' for user: jhou+2@redhat.com
Migrating gear on node with: /usr/sbin/oo-admin-upgrade --login 'jhou+2@redhat.com' --upgrade-gear '51fb19ad598baa1aad000001' --app-name 'stopapp' --version '2.0.31' --ignore-cartridge-version
Upgrading on node...
***time_upgrade_on_node_measured_from_broker=5714***
Upgrade on node output:
 Beginning 2.0.31 upgrade for 51fb19ad598baa1aad000001
Inspecting gear at /var/lib/openshift/51fb19ad598baa1aad000001
Pre-upgrade state: stopped
Marking step inspect_gear_state complete
Creating itinerary entry for compatible upgrade of redhat:php:5.3:0.0.2
Marking step compute_itinerary complete
Migrating gear at /var/lib/openshift/51fb19ad598baa1aad000001
Compatible upgrade of cartridge redhat:php:5.3:0.0.2
Removed ERB templates for php
No upgrade script exists for php; skipping
Marking step php_upgrade complete
Marking step php_rebuild_ident complete
Cleaning up after upgrade
Total upgrade time on node (ms): 1922
gear_upgrade_json={"gear_uuid":"51fb19ad598baa1aad000001","hostname":"ip-10-138-56-220","steps":{"inspect_gear_state":{"status":"complete","errors":[],"context":{"preupgrade_state":"stopped"}},"compute_itinerary":{"status":"complete","errors":[],"context":{}},"php_upgrade":{"status":"complete","errors":[],"context":{"cartridge":"php","compatible":true}},"upgrade_script_php":{"status":"incomplete","errors":[],"context":{}},"php_rebuild_ident":{"status":"complete","errors":[],"context":{"cartridge":"php"}}},"upgrade_complete":true,"errors":[],"itinerary_entries":{"php-5.3":"compatible"},"times":{"upgrade_on_node_measured_from_node":1922,"restart":0}}


....................


#####################################################
Summary:
# of users: 8
# of gears: 62
# of failures: 0
Gear counts per thread: [60, 2]
Additional timings:
    upgrade_on_node_measured_from_broker=765.916s
    upgrade_on_node_measured_from_node=3.349s
    restart=0.0s
    redeploy_httpd_proxy=0.001s
    total_upgrade_gear_measured_from_broker=766.717s
Time gathering users: 0.295s
Time gathering active gears: 22.248s
Total execution time: 815.742s
#####################################################

Note You need to log in before you can comment on or make changes to this bug.