Bug 1008517 - oo-admin-repair says "Failed to fix ssh key mismatches for application '': "
Summary: oo-admin-repair says "Failed to fix ssh key mismatches for application '': "
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Pod
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Abhishek Gupta
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-16 14:04 UTC by Thomas Wiest
Modified: 2015-05-15 00:20 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-09-19 16:51:30 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Thomas Wiest 2013-09-16 14:04:23 UTC
Description of problem:
I ran oo-admin-chk in INT and STG and got error messages like this:
[...snip...]
Gear '522fd80c03ef643aa1000467' has key with name 'cjk' in mongo but not on the node.
Gear '52300f056cec0ed2c5000261' has key with name 'cjk' in mongo but not on the node.
Gear '523013e76cec0eed870000d3' has  key with comment 'OPENSHIFT-523013e76cec0eed870000d3-5227fa0103ef64df75000031-f19' on the node but not in mongo.
Gear '523015c103ef6441e00000a1' has  key with comment 'OPENSHIFT-523015c103ef6441e00000a1-5227fa0103ef64df75000031-f19' on the node but not in mongo.
Gear '523015c103ef6441e00000a1' has key with name 'application-523015c103ef6441e00000a1' in mongo but not on the node.
Gear '52301f5f6cec0e305a000038' has key with name 'cjk' in mongo but not on the node.
Gear '5231283c6cec0eb47300020e' has key with name 'jenkins' in mongo but not on the node.
Gear '522ffea203ef643e0f000001' has key with name 'cjk' in mongo but not on the node.
Gear '523154866cec0ebc210003c6' has key with name 'cjk' in mongo but not on the node.
Gear '523157296cec0e251e000121' has key with name 'cjk' in mongo but not on the node.
Gear '523158fa03ef64acce0000ce' has key with name 'cjk' in mongo but not on the node.
Gear '5231840503ef6421e40002d0' has key with name 'cjk' in mongo but not on the node.
Gear '5231840503ef6421e40002dc' has key with name 'cjk' in mongo but not on the node.
Gear '5231840503ef6421e40002e8' has key with name 'cjk' in mongo but not on the node.
Gear '5231840503ef6421e40002cd' has key with name 'cjk' in mongo but not on the node.
Gear '5231862103ef64d6b300006f' has key with name 'cjk' in mongo but not on the node.
Gear '5231862103ef64d6b300006c' has key with name 'cjk' in mongo but not on the node.
Gear '52318cde03ef64d6b300018e' has key with name 'cjk' in mongo but not on the node.
[...snip...]


So I tried to repair the problems by running this:
oo-admin-repair -v --consumed-gears --ssh-keys --district-uids


and it error-ed saying this:
Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:
Fixing ssh keys for application: ...	FAIL
Failed to fix ssh key mismatches for application '': 
Problem:
  Calling Document.find with nil is invalid.
Summary:
  Document.find expects the parameters to be 1 or more ids, and will return a single document if 1 id is provided, otherwise an array of documents if multiple ids are provided.
Resolution:
  Most likely this is caused by passing parameters directly through to the find, and the parameter either is not present or the key from which it is accessed is incorrect.

Failed to fix ssh key mismatches for 1 applications.




This is happening in both INT and STG.

I tried to find an application that doesn't have a name set using the following mongo queries:

libra:PRIMARY> db.applications.find({'name' : null})
libra:PRIMARY> db.applications.find({'name' : ''})
libra:PRIMARY> 

Since I wasn't able to find anything, I think this is a bug in oo-admin-repair.

I was able to find the referred to application by looking it up by one of it's gears:
libra:PRIMARY> db.applications.findOne({'group_instances.gears.uuid' : gear_uuid}, {'name':true})
{ "_id" : ObjectId("5231840503ef6421e40002cd"), "name" : "cews1s0" }
libra:PRIMARY>


Notice that the application name is set properly.




Version-Release number of selected component (if applicable):
openshift-origin-broker-util-1.14.6-1.el6oso.noarch


How reproducible:
Very in INT and STG.


Steps to Reproduce:
1. Unknown, found in INT / STG


Actual results:
The error above


Expected results:
No error and it should fix the problem.

Comment 1 Abhishek Gupta 2013-09-16 19:08:56 UTC
Fixed with --> https://github.com/openshift/origin-server/pull/3633

Comment 3 zhaozhanqi 2013-09-17 07:55:58 UTC
Tested this issue on devenv_stage_477,it has been fixed.

[root@ip-10-190-170-16 ~]# oo-admin-repair --ssh-keys
Started at: 2013-09-17 03:52:41 -0400
Time to fetch mongo data: 0.016s
Total gears found in mongo: 3
Time to get all sshkeys for all gears from nodes: 21.178s
Total gears found on the nodes: 3
Total nodes that responded : 2
Check failed.
Gear '523803436100e3d5a9000025' has  key with comment 'OPENSHIFT-523803436100e3d5a9000025-application-523803436100e3d5a9000025' on the node but not in mongo.
Gear '523803436100e3d5a9000025' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node.
Gear '70da74a41f6a11e38d381231393f9ce2' has  key with comment 'OPENSHIFT-70da74a41f6a11e38d381231393f9ce2-application-523803436100e3d5a9000025' on the node but not in mongo.
Gear '70da74a41f6a11e38d381231393f9ce2' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node.
Gear '3cf2aa2e1f6c11e38d381231393f9ce2' has  key with comment 'OPENSHIFT-3cf2aa2e1f6c11e38d381231393f9ce2-application-523803436100e3d5a9000025' on the node but not in mongo.
Gear '3cf2aa2e1f6c11e38d381231393f9ce2' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node.

Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:

Fixed ssh key mismatches for 1 applications.

Total time: 26.091s
Finished at: 2013-09-17 03:53:07 -0400
[root@ip-10-190-170-16 ~]# oo-admin-repair --ssh-keys -r
Started at: 2013-09-17 03:53:49 -0400
Time to fetch mongo data: 0.017s
Total gears found in mongo: 3
Time to get all sshkeys for all gears from nodes: 21.124s
Total gears found on the nodes: 3
Total nodes that responded : 2
Success
Total time: 21.142s
Finished at: 2013-09-17 03:54:11 -0400


Note You need to log in before you can comment on or make changes to this bug.