Bug 1008517 - oo-admin-repair says "Failed to fix ssh key mismatches for application '': "
oo-admin-repair says "Failed to fix ssh key mismatches for application '': "
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-16 10:04 EDT by Thomas Wiest
Modified: 2015-05-14 20:20 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-19 12:51:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Thomas Wiest 2013-09-16 10:04:23 EDT
Description of problem:
I ran oo-admin-chk in INT and STG and got error messages like this:
[...snip...]
Gear '522fd80c03ef643aa1000467' has key with name 'cjk' in mongo but not on the node.
Gear '52300f056cec0ed2c5000261' has key with name 'cjk' in mongo but not on the node.
Gear '523013e76cec0eed870000d3' has  key with comment 'OPENSHIFT-523013e76cec0eed870000d3-5227fa0103ef64df75000031-f19' on the node but not in mongo.
Gear '523015c103ef6441e00000a1' has  key with comment 'OPENSHIFT-523015c103ef6441e00000a1-5227fa0103ef64df75000031-f19' on the node but not in mongo.
Gear '523015c103ef6441e00000a1' has key with name 'application-523015c103ef6441e00000a1' in mongo but not on the node.
Gear '52301f5f6cec0e305a000038' has key with name 'cjk' in mongo but not on the node.
Gear '5231283c6cec0eb47300020e' has key with name 'jenkins' in mongo but not on the node.
Gear '522ffea203ef643e0f000001' has key with name 'cjk' in mongo but not on the node.
Gear '523154866cec0ebc210003c6' has key with name 'cjk' in mongo but not on the node.
Gear '523157296cec0e251e000121' has key with name 'cjk' in mongo but not on the node.
Gear '523158fa03ef64acce0000ce' has key with name 'cjk' in mongo but not on the node.
Gear '5231840503ef6421e40002d0' has key with name 'cjk' in mongo but not on the node.
Gear '5231840503ef6421e40002dc' has key with name 'cjk' in mongo but not on the node.
Gear '5231840503ef6421e40002e8' has key with name 'cjk' in mongo but not on the node.
Gear '5231840503ef6421e40002cd' has key with name 'cjk' in mongo but not on the node.
Gear '5231862103ef64d6b300006f' has key with name 'cjk' in mongo but not on the node.
Gear '5231862103ef64d6b300006c' has key with name 'cjk' in mongo but not on the node.
Gear '52318cde03ef64d6b300018e' has key with name 'cjk' in mongo but not on the node.
[...snip...]


So I tried to repair the problems by running this:
oo-admin-repair -v --consumed-gears --ssh-keys --district-uids


and it error-ed saying this:
Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:
Fixing ssh keys for application: ...	FAIL
Failed to fix ssh key mismatches for application '': 
Problem:
  Calling Document.find with nil is invalid.
Summary:
  Document.find expects the parameters to be 1 or more ids, and will return a single document if 1 id is provided, otherwise an array of documents if multiple ids are provided.
Resolution:
  Most likely this is caused by passing parameters directly through to the find, and the parameter either is not present or the key from which it is accessed is incorrect.

Failed to fix ssh key mismatches for 1 applications.




This is happening in both INT and STG.

I tried to find an application that doesn't have a name set using the following mongo queries:

libra:PRIMARY> db.applications.find({'name' : null})
libra:PRIMARY> db.applications.find({'name' : ''})
libra:PRIMARY> 

Since I wasn't able to find anything, I think this is a bug in oo-admin-repair.

I was able to find the referred to application by looking it up by one of it's gears:
libra:PRIMARY> db.applications.findOne({'group_instances.gears.uuid' : gear_uuid}, {'name':true})
{ "_id" : ObjectId("5231840503ef6421e40002cd"), "name" : "cews1s0" }
libra:PRIMARY>


Notice that the application name is set properly.




Version-Release number of selected component (if applicable):
openshift-origin-broker-util-1.14.6-1.el6oso.noarch


How reproducible:
Very in INT and STG.


Steps to Reproduce:
1. Unknown, found in INT / STG


Actual results:
The error above


Expected results:
No error and it should fix the problem.
Comment 1 Abhishek Gupta 2013-09-16 15:08:56 EDT
Fixed with --> https://github.com/openshift/origin-server/pull/3633
Comment 3 zhaozhanqi 2013-09-17 03:55:58 EDT
Tested this issue on devenv_stage_477,it has been fixed.

[root@ip-10-190-170-16 ~]# oo-admin-repair --ssh-keys
Started at: 2013-09-17 03:52:41 -0400
Time to fetch mongo data: 0.016s
Total gears found in mongo: 3
Time to get all sshkeys for all gears from nodes: 21.178s
Total gears found on the nodes: 3
Total nodes that responded : 2
Check failed.
Gear '523803436100e3d5a9000025' has  key with comment 'OPENSHIFT-523803436100e3d5a9000025-application-523803436100e3d5a9000025' on the node but not in mongo.
Gear '523803436100e3d5a9000025' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node.
Gear '70da74a41f6a11e38d381231393f9ce2' has  key with comment 'OPENSHIFT-70da74a41f6a11e38d381231393f9ce2-application-523803436100e3d5a9000025' on the node but not in mongo.
Gear '70da74a41f6a11e38d381231393f9ce2' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node.
Gear '3cf2aa2e1f6c11e38d381231393f9ce2' has  key with comment 'OPENSHIFT-3cf2aa2e1f6c11e38d381231393f9ce2-application-523803436100e3d5a9000025' on the node but not in mongo.
Gear '3cf2aa2e1f6c11e38d381231393f9ce2' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node.

Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:

Fixed ssh key mismatches for 1 applications.

Total time: 26.091s
Finished at: 2013-09-17 03:53:07 -0400
[root@ip-10-190-170-16 ~]# oo-admin-repair --ssh-keys -r
Started at: 2013-09-17 03:53:49 -0400
Time to fetch mongo data: 0.017s
Total gears found in mongo: 3
Time to get all sshkeys for all gears from nodes: 21.124s
Total gears found on the nodes: 3
Total nodes that responded : 2
Success
Total time: 21.142s
Finished at: 2013-09-17 03:54:11 -0400

Note You need to log in before you can comment on or make changes to this bug.