Bug 1040360

Summary: [fork]oo-admin-chk does not detect mismatched applicaiton ssh keys
Product: OpenShift Online Reporter: Jianwei Hou <jhou>
Component: PodAssignee: Ravi Sankar <rpenta>
Status: CLOSED NOTABUG QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: rpenta, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-12 00:51:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jianwei Hou 2013-12-11 10:04:41 UTC
Description of problem:
Given an application ssh key content is updated in mongo, which makes the ssh key inconsistent with the gear ssh key on node, when oo-admin-chk -l 1 is executed, the mismatched ssh key is not detected.

Version-Release number of selected component (if applicable):
On fork_ami_origin_broker_admin-chk_960

How reproducible:
Always

Steps to Reproduce:
1. Create a scalable application
2. Update the app_ssh_keys with rockmongo interface, change the content of the app sshkey
3. Verify that the mongo ssh key content is different gear ssh key
4. oo-admin-chk

Actual results:
After step 3: 
Mongo ssh key content, the first 'AAAA' is updated to 'BBBB'
{
      "_id": ObjectId("52a835dc6e8f04987a000216"),
      "_type": "ApplicationSshKey",
      "component_id": ObjectId("52a835bc6e8f04987a0001fd"),
      "content": "BBBBB3NzaC1yc2EAAAABIwAAAQEAs8N4aklwloBByoV/YVZ3OI4SohtkIv1IRDWpXHFk21KiFf3X4tiNgGdCks5bk3LKEczQUd+xLn96zP7IoldYydmxOpz7anr/LicpdJahYrIiPz4wf4VbP8JrhexN1JI+xyTeMz18t+Ct5p6w14tl/lvtZyl+kAt1+gL5HHDNZNm4AXFHW9q4JI+dSA6kfCXkzqao5vSokjq9fMymiT1lZXNhCRS+SHpldjhG2XWtvsllL6KzGnrwASbt3szsJgweg0uP1hHmv0OvmmhSgiTcAucZoDrReH/NIskLmAaqsSaEw8tipa8YK+h5pA8nVLeLE/XeCMIQfMhthUqtq7hCIw==",
      "name": "application-305794084953813554298880",
      "type": "ssh-rsa"
    }

ssh key content in .ssh/ 
command="/usr/bin/oo-trap-user",no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAs8N4aklwloBByoV/YVZ3OI4SohtkIv1IRDWpXHFk21KiFf3X4tiNgGdCks5bk3LKEczQUd+xLn96zP7IoldYydmxOpz7anr/LicpdJahYrIiPz4wf4VbP8JrhexN1JI+xyTeMz18t+Ct5p6w14tl/lvtZyl+kAt1+gL5HHDNZNm4AXFHW9q4JI+dSA6kfCXkzqao5vSokjq9fMymiT1lZXNhCRS+SHpldjhG2XWtvsllL6KzGnrwASbt3szsJgweg0uP1hHmv0OvmmhSgiTcAucZoDrReH/NIskLmAaqsSaEw8tipa8YK+h5pA8nVLeLE/XeCMIQfMhthUqtq7hCIw== OPENSHIFT-52a835906e8f04987a0001da-application-305794084953813554298880

After step 4:
[root@domU-12-31-39-0E-8C-89 ~]# oo-admin-chk -l 1
Started at: 2013-12-11 04:59:19 -0500
Time to fetch mongo data: 0.027s
Total gears found in mongo: 2
Time to get all gears from nodes: 20.69s
Total gears found on the nodes: 2
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.055s
Total gears found on the nodes: 2
Total nodes that responded : 1
Success
Total time: 42.0s
Finished at: 2013-12-11 05:00:01 -0500


Expected results:
Step 4 should report fail instead of success since the ssh keys are mismatching

Additional info:

Comment 1 Jianwei Hou 2013-12-11 10:20:08 UTC
Sorry, after a short while, the mismatch was detected. Looks like there is some wait time for mcollective to collect the ssh key content in order to discover the inconsistency of mongo ssh key and node ssh key.
Please close it if this does not need fixing. Thanks

Comment 2 Ravi Sankar 2013-12-12 00:51:59 UTC
No problem with mcollective, oo-admin-chk finds ssh key inconsistencies for apps that are at least 10 mins old and the rationale for that is to avoid the case where ssh keys might be persisted in mongo but might not have propagated to the gear.
Marking the bug as closed.