| Summary: | oo-admin-repair says "Failed to fix ssh key mismatches for application '': " | ||
|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Thomas Wiest <twiest> |
| Component: | Pod | Assignee: | Abhishek Gupta <abhgupta> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.x | CC: | zzhao |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-09-19 16:51:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Fixed with --> https://github.com/openshift/origin-server/pull/3633 Tested this issue on devenv_stage_477,it has been fixed. [root@ip-10-190-170-16 ~]# oo-admin-repair --ssh-keys Started at: 2013-09-17 03:52:41 -0400 Time to fetch mongo data: 0.016s Total gears found in mongo: 3 Time to get all sshkeys for all gears from nodes: 21.178s Total gears found on the nodes: 3 Total nodes that responded : 2 Check failed. Gear '523803436100e3d5a9000025' has key with comment 'OPENSHIFT-523803436100e3d5a9000025-application-523803436100e3d5a9000025' on the node but not in mongo. Gear '523803436100e3d5a9000025' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node. Gear '70da74a41f6a11e38d381231393f9ce2' has key with comment 'OPENSHIFT-70da74a41f6a11e38d381231393f9ce2-application-523803436100e3d5a9000025' on the node but not in mongo. Gear '70da74a41f6a11e38d381231393f9ce2' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node. Gear '3cf2aa2e1f6c11e38d381231393f9ce2' has key with comment 'OPENSHIFT-3cf2aa2e1f6c11e38d381231393f9ce2-application-523803436100e3d5a9000025' on the node but not in mongo. Gear '3cf2aa2e1f6c11e38d381231393f9ce2' has key with name 'application-523803436100e3d5a9000025' in mongo but not on the node. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixed ssh key mismatches for 1 applications. Total time: 26.091s Finished at: 2013-09-17 03:53:07 -0400 [root@ip-10-190-170-16 ~]# oo-admin-repair --ssh-keys -r Started at: 2013-09-17 03:53:49 -0400 Time to fetch mongo data: 0.017s Total gears found in mongo: 3 Time to get all sshkeys for all gears from nodes: 21.124s Total gears found on the nodes: 3 Total nodes that responded : 2 Success Total time: 21.142s Finished at: 2013-09-17 03:54:11 -0400 |
Description of problem: I ran oo-admin-chk in INT and STG and got error messages like this: [...snip...] Gear '522fd80c03ef643aa1000467' has key with name 'cjk' in mongo but not on the node. Gear '52300f056cec0ed2c5000261' has key with name 'cjk' in mongo but not on the node. Gear '523013e76cec0eed870000d3' has key with comment 'OPENSHIFT-523013e76cec0eed870000d3-5227fa0103ef64df75000031-f19' on the node but not in mongo. Gear '523015c103ef6441e00000a1' has key with comment 'OPENSHIFT-523015c103ef6441e00000a1-5227fa0103ef64df75000031-f19' on the node but not in mongo. Gear '523015c103ef6441e00000a1' has key with name 'application-523015c103ef6441e00000a1' in mongo but not on the node. Gear '52301f5f6cec0e305a000038' has key with name 'cjk' in mongo but not on the node. Gear '5231283c6cec0eb47300020e' has key with name 'jenkins' in mongo but not on the node. Gear '522ffea203ef643e0f000001' has key with name 'cjk' in mongo but not on the node. Gear '523154866cec0ebc210003c6' has key with name 'cjk' in mongo but not on the node. Gear '523157296cec0e251e000121' has key with name 'cjk' in mongo but not on the node. Gear '523158fa03ef64acce0000ce' has key with name 'cjk' in mongo but not on the node. Gear '5231840503ef6421e40002d0' has key with name 'cjk' in mongo but not on the node. Gear '5231840503ef6421e40002dc' has key with name 'cjk' in mongo but not on the node. Gear '5231840503ef6421e40002e8' has key with name 'cjk' in mongo but not on the node. Gear '5231840503ef6421e40002cd' has key with name 'cjk' in mongo but not on the node. Gear '5231862103ef64d6b300006f' has key with name 'cjk' in mongo but not on the node. Gear '5231862103ef64d6b300006c' has key with name 'cjk' in mongo but not on the node. Gear '52318cde03ef64d6b300018e' has key with name 'cjk' in mongo but not on the node. [...snip...] So I tried to repair the problems by running this: oo-admin-repair -v --consumed-gears --ssh-keys --district-uids and it error-ed saying this: Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: ... FAIL Failed to fix ssh key mismatches for application '': Problem: Calling Document.find with nil is invalid. Summary: Document.find expects the parameters to be 1 or more ids, and will return a single document if 1 id is provided, otherwise an array of documents if multiple ids are provided. Resolution: Most likely this is caused by passing parameters directly through to the find, and the parameter either is not present or the key from which it is accessed is incorrect. Failed to fix ssh key mismatches for 1 applications. This is happening in both INT and STG. I tried to find an application that doesn't have a name set using the following mongo queries: libra:PRIMARY> db.applications.find({'name' : null}) libra:PRIMARY> db.applications.find({'name' : ''}) libra:PRIMARY> Since I wasn't able to find anything, I think this is a bug in oo-admin-repair. I was able to find the referred to application by looking it up by one of it's gears: libra:PRIMARY> db.applications.findOne({'group_instances.gears.uuid' : gear_uuid}, {'name':true}) { "_id" : ObjectId("5231840503ef6421e40002cd"), "name" : "cews1s0" } libra:PRIMARY> Notice that the application name is set properly. Version-Release number of selected component (if applicable): openshift-origin-broker-util-1.14.6-1.el6oso.noarch How reproducible: Very in INT and STG. Steps to Reproduce: 1. Unknown, found in INT / STG Actual results: The error above Expected results: No error and it should fix the problem.