Description of problem: When using "oo-admin-repair --ssh-keys" on the instance, met following errors: /usr/sbin/oo-admin-repair:584:in `<main>': undefined local variable or method `error_ssh_keys_app_ids' for main:Object (NameError) Version-Release number of selected component (if applicable): devenv_3837 How reproducible: always Steps to Reproduce: 1.ssh into instance. 2.run oo-admin-repair --ssh-keys 3. Actual results: # oo-admin-repair --ssh-keys Started at: 2013-09-27 09:12:52 -0400 Time to fetch mongo data: 0.019s Total gears found in mongo: 5 Time to get all sshkeys for all gears from nodes: 2.786s Total gears found on the nodes: 5 Total nodes that responded : 1 Success /usr/sbin/oo-admin-repair:584:in `<main>': undefined local variable or method `error_ssh_keys_app_ids' for main:Object (NameError) Expected results: ssh key repaired successfully without errors. Additional info:
Ravi: Can you please make sure that this bug is addressed as part of your changes to oo-admin-repair. The app IDs are stored in the datastore_hash and the variable name was correct before your change.
Fixed in https://github.com/openshift/origin-server/pull/3731
Checked on devenv_3844. oo-admin-repair --ssh-key failed to fix ssh key mismatch. Steps and result as below: 1. fix ssh key mismatch with oo-admin-repair. 2. check the ssh key content in gear. 3. check the ssh key content in mongodb. # oo-admin-repair --ssh-key -v Started at: 2013-09-28 22:00:37 -0400 Time to fetch mongo data: 0.019s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 20.665s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 524786e6fc27c242d8000007... FAIL Check failed. Gear '524786e6fc27c242d8000007' has key with comment 'OPENSHIFT-524786e6fc27c242d8000007-application-524786e6fc27c242d8000007' on the node but not in mongo. Gear '524786e6fc27c242d8000007' has key with comment 'OPENSHIFT-524786e6fc27c242d8000007-524786c9fc27c242d8000001-default' on the node but not in mongo. Gear '524786e6fc27c242d8000007' has key with name 'default' in mongo but not on the node. Gear '524786e6fc27c242d8000007' has key with name 'application-524786e6fc27c242d8000007' in mongo but not on the node. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 524786e6fc27c242d8000007... FAIL Failed to fix ssh key mismatches for application '524786e6fc27c242d8000007': Problem: Calling Document.find with nil is invalid. Summary: Document.find expects the parameters to be 1 or more ids, and will return a single document if 1 id is provided, otherwise an array of documents if multiple ids are provided. Resolution: Most likely this is caused by passing parameters directly through to the find, and the parameter either is not present or the key from which it is accessed is incorrect. Failed to fix ssh key mismatches for 1 applications. Total time: 20.733s Finished at: 2013-09-28 22:00:57 -0400 # cat /var/lib/openshift/524786e6fc27c242d8000007/.ssh/authorized_keys command="/usr/bin/oo-trap-user",no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6mTc9Oftu6XPNg78qoA745Zy46YkHj/erRTvHzlGVLqcehkuJHR6nQkJm6pCAADWHZ2IWc2Hxo3fQ6ms0HfwaVp5avm0Z/edUD+me3SvmLlr9cReFBmwaOLxQAKQVyKXBrKbvwJ1LTYqu9fVSthodU+rnXuT102u/fB1MaXgdnkdDloD+q3cTG4aXzuB6WyHzRVCzHhas6B74F3VT7+ThcIyO/+27i9HfOPAT/khcEALjy1f5V5RphuxBYvC/cnxfeizbVRP6C9gFprMkcTuQssxzfNaisM1qOx0je/InY5YYuXrgBFr1N/ZX0VQ4uytObwkkjucOAdXI/1GXh+zFQ== OPENSHIFT-524786e6fc27c242d8000007-524786c9fc27c242d8000001-default command="/usr/bin/oo-trap-user",no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuzumDSDPQo++JlbU/FJ3WXnIQOqy+tNKvFubcAd7TTJAwRvxXeurK6D2AtL/XyyS+sdbyhfUK9uRaMzjkPyfrDyMnAR7IdZ6Di8BS+sFEx4qwMQvwvJNQYf6WREjEGubCPpDSyKvleupGVSHfj+E8tLvmRagZcGNrymZSX+qVTntj19jEBgevskEZkwkNoBeTs0w199xM8/DMFo3qYTh+KNPox/f0XvPM7yOdVERqpPAe10L7cXNkhY5kmtD4GwjzEuzxxkFh7UnhL8JOvxKpS4jVLFHLguC+zuZzmUL5tdoapfOuH1yVB/PEm1tVUB4ybD/UfWGYx3lcQO1KokSxQ== OPENSHIFT-524786e6fc27c242d8000007-application-524786e6fc27c242d8000007 libra_rs:PRIMARY>db.applications.findOne({"_id": ObjectId("524786e6fc27c242d8000007")}) "app_ssh_keys" : [ { "_id" : ObjectId("524786fbfc27c242d800001f"), "_type" : "ApplicationSshKey", "component_id" : null, "content" : "B3NzaC1yc2EAAAABIwAAAQEAuzumDSDPQo++JlbU/FJ3WXnIQOqy+tNKvFubcAd7TTJAwRvxXeurK6D2AtL/XyyS+sdbyhfUK9uRaMzjkPyfrDyMnAR7IdZ6Di8BS+sFEx4qwMQvwvJNQYf6WREjEGubCPpDSyKvleupGVSHfj+E8tLvmRagZcGNrymZSX+qVTntj19jEBgevskEZkwkNoBeTs0w199xM8/DMFo3qYTh+KNPox/f0XvPM7yOdVERqpPAe10L7cXNkhY5kmtD4GwjzEuzxxkFh7UnhL8JOvxKpS4jVLFHLguC+zuZzmUL5tdoapfOuH1yVB/PEm1tVUB4ybD/UfWGYx3lcQO1KokSxQ==", "name" : "application-524786e6fc27c242d8000007", "type" : "ssh-rsa" } ],
Fixed in https://github.com/openshift/origin-server/pull/3742
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/8b3caf11a3ee5519acaf86ac95855546d658e09f Bug 1012970 - Fix pending op group: ReplaceAllSshKeysOpGroup.
verified with devenv_3581 [root@ip-10-185-15-146 ~]# oo-admin-repair --ssh-key -v Started at: 2013-10-02 19:17:59 -0400 Time to fetch mongo data: 0.019s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 20.583s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 524ca41fe772ff4e420000f1... OK Success Total time: 20.602s Finished at: 2013-10-02 19:18:20 -0400