Bug 1022370 - Could not remove SSH key from .ssh/authorized_keys when it's comment is modified
Could not remove SSH key from .ssh/authorized_keys when it's comment is modified
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-23 03:24 EDT by Wei Sun
Modified: 2015-05-14 20:22 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-23 22:26:18 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Wei Sun 2013-10-23 03:24:24 EDT
Description of problem:
After modify the sshkey's comment in .ssh/authorized_keys,try to remove sshkey via rhc ,could remove sshkey via rhc,but the sshkey is not removed from .ssh/authorized_keys.

But could not ssh to app after removing sshkey via `rhc sshkey remove `

Version-Release number of selected component (if applicable):
devenv_3933

How reproducible:
Always

Steps to Reproduce:
1.Create an application
2.Create ssh key
ssh-keygen -t rsa -N '' -f testkey
3.Add new sshkey
rhc sshkey add testkey testkey.pub
4.Edit /var/lib/openshift/UUID/.ssh/authorized_keys , change the comment
5. Remove the sshkey
6. check if the ssh key is deleted from authorized_keys

Actual results:
3.[openshift@dhcp-10-143 test]$ rhc sshkey add testkey testkey.pub -l wsun+1@redhat.com -p x
RESULT:
SSH key testkey.pub has been added as 'testkey'

4.[root@ip-10-73-134-110 5267727671250cbe47000061]# cat .ssh/authorized_keys 
command="/usr/bin/oo-trap-user",no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3qYaSq7US46Z3JbmJdEx5ms9cXgW8c7uSQyQwaLkMDxdNpHpJcBM/sZyH5Fp7ACg9cbUQTmDFFsA820+Y5s+mkU2OwwgLaJWN0rrAmmqVJnvqB0pRrsOnxXTX0D94yKKQQQvZc4bIJ/9FQma+1UeHaOirX4iTmFirSQVeR0HBupzKkuJ/bg+K53Ip4gwJoMgeByC4VvSgsdTUcVoDLGipaWfNVghu4pL5DH0I/5IuNFipuhJik1h103PX6yqLHcK9eApR+HZRjvEtxKPozQfs1DO3gmu9IfxlHQ43RFn2QOsLrISt5zGln+4hcipw81A16lyFIjeFqe2IpfXqp6a0Q== OPENSHIFT-5267727671250cbe47000061-application-5267727671250cbe47000061
command="/usr/bin/oo-trap-user",no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvlOlQPx8FPOyJbquLWmflJKMY2YJuhqnmgukYhnpwhf+mtuZkd9It6NHU+/zb2ZoVyjsjGtJdEj4Qq6RvHXGkOC0nXv2803BidJHVvdYPo5TFbqkNYj3m3R9aZ6iddCudo+fa5RGePaNgnZPVUzJw1jsZHWQRVzg2PlywLzZhaDM8UalTwts9yr8MNX5WzjzNvoR6g+2O5HG5pR0otACFsVhdiRprB+Xz9XPcfbcZmh6Kcqn+86fWtpIkx87m+Thkq2xQJYafHLYbn05Iq8w6kctpD7SRI8+Q/78v7JuXnKts+zk/fQn1oJoxjRiuWrudGu4hVwq1bZe1LX7jlsQd OPENSHIFT-5267727671250cbe47000061-5267725171250cbe4700004f-yektset

5.[openshift@dhcp-10-143 test]$ rhc sshkey remove testkey -l wsun+1@redhat.com -p x
Removing the key 'testkey ... removed

6.[root@ip-10-73-134-110 5267727671250cbe47000061]# cat /var/lib/openshift/5267727671250cbe47000061/.ssh/authorized_keys 
command="/usr/bin/oo-trap-user",no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3qYaSq7US46Z3JbmJdEx5ms9cXgW8c7uSQyQwaLkMDxdNpHpJcBM/sZyH5Fp7ACg9cbUQTmDFFsA820+Y5s+mkU2OwwgLaJWN0rrAmmqVJnvqB0pRrsOnxXTX0D94yKKQQQvZc4bIJ/9FQma+1UeHaOirX4iTmFirSQVeR0HBupzKkuJ/bg+K53Ip4gwJoMgeByC4VvSgsdTUcVoDLGipaWfNVghu4pL5DH0I/5IuNFipuhJik1h103PX6yqLHcK9eApR+HZRjvEtxKPozQfs1DO3gmu9IfxlHQ43RFn2QOsLrISt5zGln+4hcipw81A16lyFIjeFqe2IpfXqp6a0Q== OPENSHIFT-5267727671250cbe47000061-application-5267727671250cbe47000061
command="/usr/bin/oo-trap-user",no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvlOlQPx8FPOyJbquLWmflJKMY2YJuhqnmgukYhnpwhf+mtuZkd9It6NHU+/zb2ZoVyjsjGtJdEj4Qq6RvHXGkOC0nXv2803BidJHVvdYPo5TFbqkNYj3m3R9aZ6iddCudo+fa5RGePaNgnZPVUzJw1jsZHWQRVzg2PlywLzZhaDM8UalTwts9yr8MNX5WzjzNvoR6g+2O5HG5pR0otACFsVhdiRprB+Xz9XPcfbcZmh6Kcqn+86fWtpIkx87m+Thkq2xQJYafHLYbn05Iq8w6kctpD7SRI8+Q/78v7JuXnKts+zk/fQn1oJoxjRiuWrudGu4hVwq1bZe1LX7jlsQd OPENSHIFT-5267727671250cbe47000061-5267725171250cbe4700004f-yektset


Expected results:
The sshkey is removed from authorized_keys file

Additional info:
Comment 1 Clayton Coleman 2013-10-23 14:28:00 EDT
This is an invalid scenario - you cannot edit the SSH comment in the authorized_keys file because the node and broker use it to associate the key with a physical user account.
Comment 2 Abhishek Gupta 2013-10-23 19:45:54 EDT
This regression happened based on the change in this commit --> 2fe1f834c10f3946b10db899317184a0cb2d4dc9 

Clayton: Any idea why the logic to remove ssh keys was altered?


The scenario that the test case aims to check is that older ssh keys created by the broker might have had a different comment and the logic for generating the comment on the broker side was changed earlier this year. To prevent older keys with older format comments from not being deleted, we had made this fix on the runtime side.
Comment 3 Clayton Coleman 2013-10-23 19:59:17 EDT
It's an intentional change.  The comment is now required to match the account key.  Multiple users can have the same key - removing the key from all accounts would leave the gear in an incorrect state.

If we need to do a migration it would be to scan all gears, find any authorized keys file with an incorrect comment, then mark that gear for key resync.  If that's the case let me know and I'll whip it up.
Comment 4 Abhishek Gupta 2013-10-23 20:11:29 EDT
Multiple users should not be using the same key. If they are, removing a particular user's key would have no effect as that user would still have ssh access to the gear. 

We were aware of the potential of removing ALL key entries (with the same key content) from the authorized_keys file if one of the key was being removed.

Will discuss with the team and get back to you on how we need to proceed on this.
Comment 5 openshift-github-bot 2013-10-23 23:26:19 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/d859a5662ac329150862049f0e5cd4fc00fa5484
Bug 1022370 - Add forcestopgear and forcestopall options to completely stop all processes owned by a gear.
Comment 6 Clayton Coleman 2013-10-24 09:22:15 EDT
Multiple users can and do share keys - that is a hard customer requirement and cannot be changed.

The design of the access control sharing mechanism is correct - reverting it to work around a legacy bug is not the correct approach.
Comment 7 Jordan Liggitt 2013-10-25 08:43:44 EDT
Closing, working as intended
Comment 8 Abhishek Gupta 2013-10-25 15:05:16 EDT
Please stop marking this bug as invalid and closing it.

The fix to ignore the ssh key comment was placed as a workaround to handle an issue with the ssh key comment/name format being changed on the broker side. While the current functionality is intended, we still need a fix to handle the cases where the ssh key comment/name format is older.

I have a fix (coming shortly) to oo-admin-chk and oo-admin-repair in place to take care of this. If that fix is acceptable, I would like to get that in and let QE verify this scenario. Otherwise, will go ahead and close this bug.
Comment 9 Abhishek Gupta 2013-10-25 15:35:19 EDT
Fixed with --> https://github.com/openshift/origin-server/pull/4001

oo-admin-chk will now report mismatches with key comments in addition to the key content and oo-admin-repair will fix the mismatches.
Comment 12 Wei Sun 2013-10-27 23:41:14 EDT
Verified on devenv_3953

Step:
1.Create an application
2.Create ssh key
ssh-keygen -t rsa -N '' -f testkey
3.Add new sshkey
rhc sshkey add testkey testkey.pub
4.Edit /var/lib/openshift/UUID/.ssh/authorized_keys , change the comment
5. Remove the sshkey
6.Run `oo-admin-chk -l 1`
7.Run `oo-admin-repair --ssh-keys`
8.Run `oo-admin-chk -l 1`
9.Check $appuuid/.ssh/authorized_keys

Result:
6.[root@ip-10-239-2-225 app2-wsundev3953]# oo-admin-chk -l 1
Started at: 2013-10-27 23:34:47 -0400
Time to fetch mongo data: 0.036s
Total gears found in mongo: 2
Time to get all gears from nodes: 20.705s
Total gears found on the nodes: 2
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.059s
Total gears found on the nodes: 2
Total nodes that responded : 1
Check failed.
Gear '526dd7b25ebecfd459000001' has key with hash '501ef287206d00df12641eb443ac1843' and comment 'OPENSHIFT-526dd7b25ebecfd459000001-526dd7635ebecfe69a000004-yektset' on the node but not in mongo.
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.
Total time: 40.836s
Finished at: 2013-10-27 23:35:28 -0400

7.[root@ip-10-239-2-225 app2-wsundev3953]# oo-admin-repair --ssh-keys
Started at: 2013-10-27 23:36:19 -0400
Time to fetch mongo data: 0.018s
Total gears found in mongo: 2
Time to get all sshkeys for all gears from nodes: 20.565s
Total gears found on the nodes: 2
Total nodes that responded : 1
Check failed.
Gear '526dd7b25ebecfd459000001' has key with hash '501ef287206d00df12641eb443ac1843' and comment 'OPENSHIFT-526dd7b25ebecfd459000001-526dd7635ebecfe69a000004-yektset' on the node but not in mongo.


Fixing ssh key inconsistencies for all affected applications:

Fixed ssh key mismatches for 1 applications.

Total time: 21.368s
Finished at: 2013-10-27 23:36:41 -0400

8.[root@ip-10-239-2-225 app2-wsundev3953]# oo-admin-chk -l 1
Started at: 2013-10-27 23:37:17 -0400
Time to fetch mongo data: 0.037s
Total gears found in mongo: 2
Time to get all gears from nodes: 20.81s
Total gears found on the nodes: 3
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.066s
Total gears found on the nodes: 3
Total nodes that responded : 1
Success
Total time: 40.959s
Finished at: 2013-10-27 23:37:58 -0400

9.The sshkey is removed from authorized_keys file

Note You need to log in before you can comment on or make changes to this bug.