Hide Forgot
Description of problem: Make some change to district UIDs (e.g, change one app's UID from 1182 to 3597 ), apps' ssh-key content (e.g, change the content of a user's sshkey from AAAA* to BBBB*), and consumed gears (e.g, change user's consumed gears via oo-admin-ctl-user -l $RHLOGIN --setconsumedgears 20 ), try to fix them via oo-admin-repair with specified options, it show worked, but actually not. Version-Release number of selected component (if applicable): fork_ami_broker_admin_repair_refactor How reproducible: always Steps to Reproduce: 1. Setup multi-node env. 2. Create some apps of any type. 3. Manually update an app's uid to another one. 4. Manually update an user's ssh-key content. 5. Change user's consumed gears via oo-admin-ctl-user. 6. Run oo-admin-repair --district-uids --report-only 7. Run oo-admin-repair --district-uids 8. Run oo-admin-repair --district-uids --report-only 9. Replace district-uid to consumed-gears/ssh-keys in step 6/7/8 and try again. Actual results: Step 6: # oo-admin-repair --district-uids --report-only Started at: 2014-01-06 06:37:18 UTC Total gears found in mongo: 2 UID '1182' is reserved in district 'dis' but not used by any gear Finished at: 2014-01-06 06:37:18 UTC Total time: 0.035s SUCCESS Step 7: # oo-admin-repair --district-uids Started at: 2014-01-06 06:37:51 UTC Total gears found in mongo: 2 UID '1182' is reserved in district 'dis' but not used by any gear Fixed 1 unused UIDs across all districts. Finished at: 2014-01-06 06:37:51 UTC Total time: 0.036s SUCCESS Step 8 # oo-admin-repair --district-uids --report-only Started at: 2014-01-06 06:38:17 UTC Total gears found in mongo: 2 UID '1182' is reserved in district 'dis' but not used by any gear Finished at: 2014-01-06 06:38:17 UTC Total time: 0.034s SUCCESS Expected results: oo-admin-repair fixed unused UIDs across all districts Additional info:
Fixed in fork_ami_broker_admin_repair_refactor_975
On fork_ami_broker_admin_repair_refactor_975, tested following steps, 1.rhc setup 2.change the content of a user's sshkey from AAAA* to BBBB* 3.run oo-admin-repair -r --ssh-keys # oo-admin-repair -r --ssh-keys Started at: 2014-01-07 06:36:17 UTC /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/app/helpers/admin_helper.rb:128:in `merge': can't convert Array into Hash (TypeError) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/app/helpers/admin_helper.rb:128:in `get_user_info' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/app/helpers/admin_helper.rb:143:in `block in populate_user_hash' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/lib/openshift/data_store.rb:27:in `block (2 levels) in find' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:286:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/lib/openshift/data_store.rb:26:in `block in find' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:276:in `find' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/lib/openshift/data_store.rb:25:in `find' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/app/helpers/admin_helper.rb:139:in `populate_user_hash' from /usr/sbin/oo-admin-repair:714:in `<main>'
Not a bug. Updating record through rockmongo interface will convert hash into arrays and can create problems. Either update specific field using rockmongo or use irb or mongo client to update the record for your testing.
On fork_ami_broker_admin_repair_refactor_975, oo-admin-repair --district-uids failed to repair mismatch between available capacity and available UIDs in district. # oo-admin-repair --district-uids -r -v Started at: 2014-01-08 10:00:11 UTC Total gears found in mongo: 1 Checking for unreserved UIDs in the district Checking for unused UIDs in the district Checking for gears with the same UID District 'dis' has (5999) available UIDs but (6001.0) available capacity Finished at: 2014-01-08 10:00:23 UTC Total time: 11.397s SUCCESS # oo-admin-repair --district-uids Started at: 2014-01-08 10:01:42 UTC Total gears found in mongo: 1 District 'dis' has (5999) available UIDs but (6001.0) available capacity Finished at: 2014-01-08 10:01:53 UTC Total time: 11.226s SUCCESS # oo-admin-repair --district-uids -r -v Started at: 2014-01-08 10:02:47 UTC Total gears found in mongo: 1 Checking for unreserved UIDs in the district Checking for unused UIDs in the district Checking for gears with the same UID District 'dis' has (5999) available UIDs but (6001.0) available capacity Finished at: 2014-01-08 10:02:59 UTC Total time: 11.327s SUCCESS Other mismatches, such as consumed gears, ssh-keys, UIDs reserved but not used by any gear, gear using UID1 but reserve UID2, ..., can be detected and fixed.
oo-admin-repair --district-uids will only resolve unreserved/unused district uids. This script will *not* fix if we simulate mismatch by modifying available_uids or available_capacity fields in the district record. Can you confirm if this is not the case and steps to reproduce the issue?
Based on comment #4, comment #5, oo-admin-repair can fix the mismatch consumed gears, ssh-keys, unreserved/unused district UIDs. The issue in comment #4 is due to manually remove 2 UIDs in district record, based on comment #5, this is not a valid steps. So move this bug to verified.