Bug 1025223

Summary: Met 'undefined method' error while running 'oo-admin-chk -l 1 after migration
Product: OpenShift Online Reporter: zhaozhanqi <zzhao>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: jhou, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-24 03:28:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description zhaozhanqi 2013-10-31 09:43:17 UTC 
Description of problem:
after migration, run 'oo-admin-chk -l 1' will meet error:
/usr/sbin/oo-admin-chk:531:in `block in <main>': undefined method `uniq' for #<Hash:0x00000006a64888> (NoMethodError)
	from /usr/sbin/oo-admin-chk:494:in `each'
	from /usr/sbin/oo-admin-chk:494:in `<main>'

Version-Release number of selected component (if applicable):
devenv_3973

How reproducible:
always

Steps to Reproduce:
1.Prepare applications in devenv-stage_528 instance
2. Upgrade the devent-stage instance to latest as devenv_3973
3. After upgrade, clear broker cache, restart broker and mcollective services
4. Do datastore migration
  rhc-admin-migrate-datastore --compatible --version 2.0.35
5. Do gear migration
# oo-admin-upgrade upgrade-node --version 2.0.35 --ignore-cartridge-version

6. run 'oo-admin-chk -l 1'

Actual results:
[root@ip-10-196-171-192 ~]# oo-admin-chk -l 1 -v
Started at: 2013-10-31 04:56:36 -0400
Time to fetch mongo data: 0.047s
Total gears found in mongo: 76
Checking consumed gear count for user user_with_multiple_gear_sizes...	OK
Checking consumed gear count for user zzhao...	OK
Checking consumed gear count for user lxia...	OK
Checking consumed gear count for user wsun+mi1...	OK
Checking consumed gear count for user wsun...	OK
Checking consumed gear count for user user_with_extra_storage...	OK
Checking consumed gear count for user user_with_certificate_capabilities...	OK
Checking consumed gear count for user c9sub...	OK
Checking consumed gear count for user wjiang...	OK
Checking consumed gear count for user zzhao1...	OK
Checking consumed gear count for user jhou...	OK
Checking consumed gear count for user jhouc9...	OK
Time to get all gears from nodes: 20.953s
Total gears found on the nodes: 76
Total nodes that responded : 4
Time to get all sshkeys for all gears from nodes: 20.085s
Total gears found on the nodes: 76
Total nodes that responded : 4
Checking application gears and ssh keys on corresponding nodes:
5271bffb5c116dab12000029 : String...	OK
Checking ssh keys for gear: 5271bffb5c116dab12000029...	OK
5271bf775c116dab1200000d : String...	OK
Checking ssh keys for gear: 5271bf775c116dab1200000d...	OK
5271c3985c116dab120001a3 : String...	OK
Checking ssh keys for gear: 5271c3985c116dab120001a3...	/usr/sbin/oo-admin-chk:531:in `block in <main>': undefined method `uniq' for #<Hash:0x0000000809ffd0> (NoMethodError)
	from /usr/sbin/oo-admin-chk:494:in `each'
	from /usr/sbin/oo-admin-chk:494:in `<main>'



Expected results:

no this error

Additional info: oo-admin-chk -l 0 could pass, no this error.
Comment 1 Abhishek Gupta 2013-10-31 18:39:58 UTC 
Unable to reproduce this issue.

A closer look at the bug description indicates that there are 4 nodes in the test environment and it seems that at least 1 node was not upgraded (or mcollective was not restarted on it). 

Andy Goldstein was able to track down the gear (5271c3985c116dab120001a3) where the ssh key check failed and it seems to be on node "ip-10-244-47-11". This node failed to upgrade and still has packages from Oct 22.
Comment 2 Jianwei Hou 2013-11-01 12:06:55 UTC 
This issue is not seen in today's test, sorry for the inconvenience that was cause