Created attachment 781487 [details] migration output Description of problem: Migrate with "oo-admin-upgrade", duplicate upgrade is performed on a same gear several times, the "--version" argument is recognized as nil, and the whole upgrade failed. Version-Release number of selected component (if applicable): On devenv-stage_429 openshift-origin-broker-util-1.12.8-1.el6oso.noarch How reproducible: Always Steps to Reproduce: 1. Prepare test data on a stage ami of sprint 30, which was devenv-stage_424 2. Upgrade to lastest devenv-stage instance, devenv-stage_429 3. After upgrade, enable cgroups, restart mcollective, clear broker cache and restart broker 4. Start migration oo-admin-upgrade --version 2.0.31 Actual results: Migration failed, according to the log printed to screen, maybe there are duplicate migration running against a same gear, and the '--version' argument is taken as nil. Therefore, all gears failed to migrate. Also noticed in the final result, the number of failures is greater than the number of gears: # of users: 8 # of gears: 62 # of failures: 64 Gear counts per thread: [60, 2] Please refer to the attachment for detail Expected results: Should successfully migrate the gears Additional info: This could be directly reproduced on devenv-stage_429, just create one app on it and do the migration.
Commits pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/51e21e048795f095a6fa3082367b74479afeb0ef Bug 990948 https://github.com/openshift/origin-server/commit/a16400edc04e4626b96a2a83288074afd5aa3598 Merge pull request #3263 from danmcp/master Bug 990948
Verified on devenv-stage_435 erforming upgrade with options: {:version=>"2.0.31", :continue=>false, :ignore_cartridge_version=>false, :target_server_identity=>nil, :upgrade_position=>1, :num_upgraders=>1, :rerun=>false} Getting all active gears... Getting all logins... .......................................... logins.length: 8 Writing 60 gears for node ip-10-138-56-220_active to file /tmp/oo-upgrade/upgrade_ip-10-138-56-220_active Writing 2 gears for node ip-10-138-56-220 to file /tmp/oo-upgrade/upgrade_ip-10-138-56-220 ##################################################### ##################################################### Migrating gears on node ip-10-138-56-220 Migrating gears on node ip-10-138-56-220_active Migrating app 'stopapp' gear 'stopapp' with uuid '51fb19ad598baa1aad000001' on node 'ip-10-138-56-220' for user: jhou+2 Migrating gear on node with: /usr/sbin/oo-admin-upgrade --login 'jhou+2' --upgrade-gear '51fb19ad598baa1aad000001' --app-name 'stopapp' --version '2.0.31' --ignore-cartridge-version Upgrading on node... ***time_upgrade_on_node_measured_from_broker=5714*** Upgrade on node output: Beginning 2.0.31 upgrade for 51fb19ad598baa1aad000001 Inspecting gear at /var/lib/openshift/51fb19ad598baa1aad000001 Pre-upgrade state: stopped Marking step inspect_gear_state complete Creating itinerary entry for compatible upgrade of redhat:php:5.3:0.0.2 Marking step compute_itinerary complete Migrating gear at /var/lib/openshift/51fb19ad598baa1aad000001 Compatible upgrade of cartridge redhat:php:5.3:0.0.2 Removed ERB templates for php No upgrade script exists for php; skipping Marking step php_upgrade complete Marking step php_rebuild_ident complete Cleaning up after upgrade Total upgrade time on node (ms): 1922 gear_upgrade_json={"gear_uuid":"51fb19ad598baa1aad000001","hostname":"ip-10-138-56-220","steps":{"inspect_gear_state":{"status":"complete","errors":[],"context":{"preupgrade_state":"stopped"}},"compute_itinerary":{"status":"complete","errors":[],"context":{}},"php_upgrade":{"status":"complete","errors":[],"context":{"cartridge":"php","compatible":true}},"upgrade_script_php":{"status":"incomplete","errors":[],"context":{}},"php_rebuild_ident":{"status":"complete","errors":[],"context":{"cartridge":"php"}}},"upgrade_complete":true,"errors":[],"itinerary_entries":{"php-5.3":"compatible"},"times":{"upgrade_on_node_measured_from_node":1922,"restart":0}} .................... ##################################################### Summary: # of users: 8 # of gears: 62 # of failures: 0 Gear counts per thread: [60, 2] Additional timings: upgrade_on_node_measured_from_broker=765.916s upgrade_on_node_measured_from_node=3.349s restart=0.0s redeploy_httpd_proxy=0.001s total_upgrade_gear_measured_from_broker=766.717s Time gathering users: 0.295s Time gathering active gears: 22.248s Total execution time: 815.742s #####################################################