Bug 969508 - [oo-admin-chk] UID reservation error returns after repair
[oo-admin-chk] UID reservation error returns after repair
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Kubernetes (Show other bugs)
1.x
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-31 11:38 EDT by Kenny Woodson
Modified: 2013-06-11 00:14 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-11 00:14:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Kenny Woodson 2013-05-31 11:38:56 EDT
Description of problem:

In our stage environment we have been experiencing an ongoing issue that is very strange. We have cleaned up inconsistent UID errors in stage for the oo-admin-chk.  Once these UIDs are cleaned up they reappear in our list.

 agupta and I have been attempting to debug and solve this issue but have had little success so I'm creating a bug for this so we can track our progress.

There is only a single node in this district.  
This node does not exist in any other district.

<output from oo-admin-chk -l 2>
UID '4019' is reserved in district 'std2' but not used by any gear
UID '4053' is reserved in district 'std2' but not used by any gear
UID '4369' is reserved in district 'std2' but not used by any gear
UID '4402' is reserved in district 'std2' but not used by any gear
~1100 messages

Version-Release number of selected component (if applicable):

2.0.27.1

How reproducible:

In stage this is happening and seems reproducible.

Steps to Reproduce:
1. Remove uids from a district.
2. Run oo-admin-chk -l 2 to verify the uids are missing.
3. Run oo-admin-repair --district-uids
4. After a period of time in stage the UIDs reappear in the oo-admin-chk output.

Actual results:

UIDs are fixed and then placed again into the oo-admin-chk output

Expected results:

The UIDs should get fixed.

Additional info:

The oo-app-destroy is being called on these applications and the unreserve uid call is made.  For some reason the UIDs are reappearing in the output.  When fetching the District and checking its available_uids the ones mentioned are definitely missing from the district and not associated to any gears in the district.
Comment 3 Rony Gong 2013-06-06 02:43:10 EDT
Verified on devenv-stage_360
Steps1:
1. setup multi node env, 2 districts and 2 nodes, each district have 1 node
2. create 2 application
3. update one app's uid from 5284 to 5283
4. then run oo-admin-chk -l 2
[root@ip-10-85-2-130 ~]# oo-admin-chk -l 2
Started at: 2013-06-06 01:05:38 -0400
Time to fetch mongo data: 0.055s
Total gears found in mongo: 2
Time to get all gears from nodes: 20.479s
Total gears found on the nodes: 2
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.061s
Total gears found on the nodes: 2
Total nodes that responded : 2
Check failed.
Gear 51b018109dd3fa0a6a000003 is using uid: '5284' but has reserved uid: '5283'
UID '5283' is available in district 'dist1' but used by a gear on node 'ip-10-202-59-6'
UID '5284' is reserved in district 'dist1' but not used by any gear
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.
Total time: 40.702s
Finished at: 2013-06-06 01:06:18 -0400

5. report this issue again by:
[root@ip-10-85-2-130 ~]# oo-admin-repair -r --district-uids
Started at: 2013-06-06 01:06:53 -0400
Time to fetch mongo data: 0.022s
Total gears found in mongo: 2
Check failed.
UID '5283' is available in district 'dist1' but used by a gear on node 'ip-10-202-59-6'
UID '5284' is reserved in district 'dist1' but not used by any gear

Total 1 unreserved UIDs across 1 districts.
Total 1 unused UIDs across 1 districts.

Total time: 0.033s
Finished at: 2013-06-06 01:06:53 -0400

6. repair this issue by:
[root@ip-10-85-2-130 ~]# oo-admin-repair  --district-uids
Started at: 2013-06-06 01:07:39 -0400
Time to fetch mongo data: 0.022s
Total gears found in mongo: 2
Check failed.
UID '5283' is available in district 'dist1' but used by a gear on node 'ip-10-202-59-6'
UID '5284' is reserved in district 'dist1' but not used by any gear

Total 1 unreserved UIDs across 1 districts.
Total 1 unused UIDs across 1 districts.

Fixing unreserved district UIDs for all affected districts:
Fixing unused district UIDs for all affected districts:

Fixed 1 unreserved UIDs across all districts.
Fixed 1 unused UIDs across all districts.

Total time: 0.143s
Finished at: 2013-06-06 01:07:39 -0400


7. Remove all applications in this env, then wait some mins and return oo-admin-chk, no error found.
[root@ip-10-85-2-130 ~]# oo-admin-chk -l 2
Started at: 2013-06-06 01:45:47 -0400
Time to fetch mongo data: 0.022s
Total gears found in mongo: 0
Time to get all gears from nodes: 20.512s
Total gears found on the nodes: 0
Total nodes that responded : 0
Time to get all sshkeys for all gears from nodes: 20.125s
Total gears found on the nodes: 0
Total nodes that responded : 2
Success
Total time: 40.694s
Finished at: 2013-06-06 01:46:27 -0400

Note You need to log in before you can comment on or make changes to this bug.