Bug 1048712 - [fork] Options consumed-gears/ssh-keys/district-uids to oo-admin-repair show mismatch fixed but actually not
Summary: [fork] Options consumed-gears/ssh-keys/district-uids to oo-admin-repair show ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Pod
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Ravi Sankar
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-06 08:27 UTC by Liang Xia
Modified: 2015-05-15 00:24 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-01-30 00:54:54 UTC
Target Upstream Version:
lxia: needinfo-


Attachments (Terms of Use)

Description Liang Xia 2014-01-06 08:27:40 UTC
Description of problem:
Make some change to district UIDs (e.g, change one app's UID from 1182 to 3597 ), apps' ssh-key content (e.g, change the content of a user's sshkey from AAAA* to BBBB*), and consumed gears (e.g, change user's consumed gears via oo-admin-ctl-user -l $RHLOGIN --setconsumedgears 20 ), try to fix them via oo-admin-repair with specified options, it show worked, but actually not.

Version-Release number of selected component (if applicable):
fork_ami_broker_admin_repair_refactor

How reproducible:
always

Steps to Reproduce:
1. Setup multi-node env.
2. Create some apps of any type.
3. Manually update an app's uid to another one.
4. Manually update an user's ssh-key content.
5. Change user's consumed gears via oo-admin-ctl-user.
6. Run oo-admin-repair --district-uids --report-only
7. Run oo-admin-repair --district-uids
8. Run oo-admin-repair --district-uids --report-only
9. Replace district-uid to consumed-gears/ssh-keys in step 6/7/8 and try again.

Actual results:
Step 6:
# oo-admin-repair --district-uids --report-only
Started at: 2014-01-06 06:37:18 UTC
Total gears found in mongo: 2
UID '1182' is reserved in district 'dis' but not used by any gear
Finished at: 2014-01-06 06:37:18 UTC
Total time: 0.035s
SUCCESS

Step 7:
# oo-admin-repair --district-uids 
Started at: 2014-01-06 06:37:51 UTC
Total gears found in mongo: 2
UID '1182' is reserved in district 'dis' but not used by any gear
Fixed 1 unused UIDs across all districts.
Finished at: 2014-01-06 06:37:51 UTC
Total time: 0.036s
SUCCESS

Step 8
# oo-admin-repair --district-uids --report-only
Started at: 2014-01-06 06:38:17 UTC
Total gears found in mongo: 2
UID '1182' is reserved in district 'dis' but not used by any gear
Finished at: 2014-01-06 06:38:17 UTC
Total time: 0.034s
SUCCESS

Expected results:
oo-admin-repair fixed unused UIDs across all districts

Additional info:

Comment 1 Ravi Sankar 2014-01-07 05:10:35 UTC
Fixed in fork_ami_broker_admin_repair_refactor_975

Comment 2 Liang Xia 2014-01-07 06:43:49 UTC
On fork_ami_broker_admin_repair_refactor_975, tested following steps,

1.rhc setup
2.change the content of a user's sshkey from AAAA* to BBBB*
3.run oo-admin-repair -r --ssh-keys
# oo-admin-repair -r --ssh-keys
Started at: 2014-01-07 06:36:17 UTC
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/app/helpers/admin_helper.rb:128:in `merge': can't convert Array into Hash (TypeError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/app/helpers/admin_helper.rb:128:in `get_user_info'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/app/helpers/admin_helper.rb:143:in `block in populate_user_hash'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/lib/openshift/data_store.rb:27:in `block (2 levels) in find'
	from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:286:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/lib/openshift/data_store.rb:26:in `block in find'
	from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:276:in `find'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/lib/openshift/data_store.rb:25:in `find'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.5/app/helpers/admin_helper.rb:139:in `populate_user_hash'
	from /usr/sbin/oo-admin-repair:714:in `<main>'

Comment 3 Ravi Sankar 2014-01-08 08:17:12 UTC
Not a bug. Updating record through rockmongo interface will convert hash into arrays and can create problems. Either update specific field using rockmongo or use irb or mongo client to update the record for your testing.

Comment 4 Liang Xia 2014-01-08 10:12:39 UTC
On fork_ami_broker_admin_repair_refactor_975, 

oo-admin-repair --district-uids failed to repair mismatch between available capacity and available UIDs in district.

# oo-admin-repair --district-uids -r -v
Started at: 2014-01-08 10:00:11 UTC
Total gears found in mongo: 1
Checking for unreserved UIDs in the district
Checking for unused UIDs in the district
Checking for gears with the same UID
District 'dis' has (5999) available UIDs but (6001.0) available capacity
Finished at: 2014-01-08 10:00:23 UTC
Total time: 11.397s
SUCCESS

# oo-admin-repair --district-uids
Started at: 2014-01-08 10:01:42 UTC
Total gears found in mongo: 1
District 'dis' has (5999) available UIDs but (6001.0) available capacity
Finished at: 2014-01-08 10:01:53 UTC
Total time: 11.226s
SUCCESS

# oo-admin-repair --district-uids -r -v
Started at: 2014-01-08 10:02:47 UTC
Total gears found in mongo: 1
Checking for unreserved UIDs in the district
Checking for unused UIDs in the district
Checking for gears with the same UID
District 'dis' has (5999) available UIDs but (6001.0) available capacity
Finished at: 2014-01-08 10:02:59 UTC
Total time: 11.327s
SUCCESS


Other mismatches, such as consumed gears, ssh-keys, UIDs reserved but not used by any gear, gear using UID1 but reserve UID2, ..., can be detected and fixed.

Comment 5 Ravi Sankar 2014-01-09 01:56:01 UTC
oo-admin-repair --district-uids will only resolve unreserved/unused district uids.
This script will *not* fix if we simulate mismatch by modifying available_uids or available_capacity fields in the district record. Can you confirm if this is not the case and steps to reproduce the issue?

Comment 6 Liang Xia 2014-01-09 02:10:36 UTC
Based on comment #4, comment #5, 

oo-admin-repair can fix the mismatch consumed gears, ssh-keys, unreserved/unused district UIDs.

The issue in comment #4 is due to manually remove 2 UIDs in district record, based on comment #5, this is not a valid steps.

So move this bug to verified.


Note You need to log in before you can comment on or make changes to this bug.