Description of problem: After upgrading from devenv-stage_281 to devenv-stage_282, the "applications" collection is not found in Mongo datastore. Version-Release number of selected component (if applicable): Upgrade from devenv-stage_281 to devenv-stage_282 How reproducible: Always Steps to Reproduce: 1. Launch an devenv-stage_281 instance, prepare test data against it 2. Upgrade devenv-stage_281 instance to latest stage ami(devenv-stage_282) yum update 3. Restart rhc-datastore since monogod-server is updated too 4. Connect via mongo shell, the "applications" collection is missing mongo openshift_broker_dev Actual results: [root@ip-10-108-37-17 broker]# mongo openshift_broker_dev MongoDB shell version: 2.2.3 connecting to: openshift_broker_dev libra_rs:PRIMARY> show collections; cloud_users distributed_lock distributed_locks districts domains gears locks system.indexes system.users usages user Expected results: The "applications" collection shouldn't be missing Additional info: This is blocking other migration related scripts from executing
Created attachment 694254 [details] The upgrade log Here are some mongo related update log, from which we can see there is no "applications" collection. Full log is attached. useradd: user 'nagios_monitor' already exists MongoDB shell version: 2.2.3 connecting to: openshift_broker_dev cloud_users distributed_lock distributed_locks districts domains gears system.indexes system.users usages user bye Creating named test user Adding medium gear size to user Adding gear size medium for user user_with_multiple_gear_sizes... Done. User user_with_multiple_gear_sizes: consumed gears: 0 max gears: 3 max storage per gear: 0 plan upgrade enabled: true gear sizes: small, medium sub accounts allowed: false Creating test user: user_with_extra_storage Adding additional storage to user
I'm not sure if the missing of "applications" collection could have caused the failure of migration, or the migrate-mongo-2.0.23 should have created the collection if it's missing. But it can not work at present. This script has been working well previously, does some other packages blocked it? [root@ip-10-108-37-17 bin]# ./migrate-mongo-2.0.23 Migration Started... Processing user user_with_multiple_gear_sizes Processing user user_with_extra_storage Processing user jhou+1 ERROR: Migration failed for User jhou+1: undefined method `component_instances' for nil:NilClass /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.4.7/app/models/group_instance.rb:47:in `all_component_instances' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.4.7/app/models/group_instance.rb:110:in `get_group_override' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.4.7/app/models/group_instance.rb:55:in `gear_size=' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/attributes/processing.rb:102:in `process_attribute' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/attributes/processing.rb:27:in `block in process_attributes' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/attributes/processing.rb:25:in `each_pair' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/attributes/processing.rb:25:in `process_attributes' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/document.rb:147:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/threaded/lifecycle.rb:84:in `_building' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/document.rb:142:in `initialize' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.4.7/app/models/group_instance.rb:34:in `initialize' ./migrate-mongo-2.0.23:169:in `new' ./migrate-mongo-2.0.23:169:in `block in create_group_instances' ./migrate-mongo-2.0.23:144:in `each' ./migrate-mongo-2.0.23:144:in `create_group_instances' ./migrate-mongo-2.0.23:118:in `block in create_applications' ./migrate-mongo-2.0.23:81:in `each' ./migrate-mongo-2.0.23:81:in `create_applications' ./migrate-mongo-2.0.23:243:in `migrate_user' ./migrate-mongo-2.0.23:279:in `block in <main>' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:78:in `block in each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `block in each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:77:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:77:in `each' ./migrate-mongo-2.0.23:276:in `<main>' Processing user jhou ERROR: Migration failed for User jhou: undefined method `component_instances' for nil:NilClass /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.4.7/app/models/group_instance.rb:47:in `all_component_instances' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.4.7/app/models/group_instance.rb:110:in `get_group_override' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.4.7/app/models/group_instance.rb:55:in `gear_size=' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/attributes/processing.rb:102:in `process_attribute' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/attributes/processing.rb:27:in `block in process_attributes' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/attributes/processing.rb:25:in `each_pair' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/attributes/processing.rb:25:in `process_attributes' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/document.rb:147:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/threaded/lifecycle.rb:84:in `_building' /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.5/lib/mongoid/document.rb:142:in `initialize' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.4.7/app/models/group_instance.rb:34:in `initialize' ./migrate-mongo-2.0.23:169:in `new' ./migrate-mongo-2.0.23:169:in `block in create_group_instances' ./migrate-mongo-2.0.23:144:in `each' ./migrate-mongo-2.0.23:144:in `create_group_instances' ./migrate-mongo-2.0.23:118:in `block in create_applications' ./migrate-mongo-2.0.23:81:in `each' ./migrate-mongo-2.0.23:81:in `create_applications' ./migrate-mongo-2.0.23:243:in `migrate_user' ./migrate-mongo-2.0.23:279:in `block in <main>' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:78:in `block in each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `block in each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:77:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:77:in `each' ./migrate-mongo-2.0.23:276:in `<main>' User migration complete. Migrating districts... *************************** Migration Done! - 2 failures Failed user logins: jhou+1 jhou ***************************
The applications collection would be auto created once the migration ran but the exceptions obviously are a problem.
Fixed with pull request#866
Verified upgrading devenv-stage_281 to devenv-stage_283 No errors occured. [root@ip-10-151-20-3 bin]# ./migrate-mongo-2.0.23 Migration Started... Migrating districts... Waiting for phase1 to get over..done Half time : 29.588531542 Waiting for phase2 to get over..done Total time : 61.074608904 *************************** Migration Done! - No failures