| Summary: | Instructions do not enable passwordless ssh between all nodes | ||
|---|---|---|---|
| Product: | Red Hat Gluster Storage | Reporter: | Dustin Black <dblack> |
| Component: | doc-Administration_Guide | Assignee: | Bhavana <bmohanra> |
| doc-Administration_Guide sub component: | Default | QA Contact: | Shashank Raj <sraj> |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | asriram, asrivast, bmohanra, dblack, mhideo, nlevinki, rcyriac, rhinduja, rhs-bugs, rwheeler, sashinde, storage-doc |
| Version: | rhgs-3.1 | Keywords: | ZStream |
| Target Milestone: | --- | ||
| Target Release: | RHGS 3.1.2 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-05-19 08:27:58 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | |||
| Bug Blocks: | 1329186 | ||
|
Description
Dustin Black
2016-04-07 16:25:44 UTC
We should actually pass the generated private key to the scp command to make it easier. scp -i /var/lib/glusterd/nfs/secret.pem \ /var/lib/glusterd/nfs/secret.* root@n2:/var/lib/glusterd/nfs/ Also, we need to self-copy the public key to the authorized_keys file on the node where the keypair is generated. Either by including the local node's hostname in the list of nodes for which we run the `ssh-copy-id` command, or with: cat /var/lib/glusterd/nfs/secret.pem.pub >> ~/.ssh/authorized_keys Hi Dustin, An additional step is added under "7.2.3.3.1. Prerequisites to run NFS-Ganesha" http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-3.1.2-Stage_Administration_Guide%20%28html-single%29/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#sect-NFS_Ganesha ------------------------------------------------------------------------- Passwordless ssh needs to be enabled on all the HA nodes. Follow these steps, On one of the nodes (node1) in the cluster, run: # ssh-keygen -f /var/lib/glusterd/nfs/secret.pem -t rsa -N '' Deploy the generated public key from node1 to all the nodes by executing the following command for every node: # ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root@<node-ip/hostname> Copy the ssh keypair from node1 to all the nodes in the Ganesha-HA cluster by executing the following command for every node: # scp /var/lib/glusterd/nfs/secret.* \ root@:<node-ip/hostname>/var/lib/glusterd/nfs/ ------------------------------------------------------------------------- Does this address the issue (including comment 3) ? Also, the command "# scp /var/lib/glusterd/nfs/secret.* <node-ip/hostname>:/var/lib/glusterd/nfs/" was removed earlier based on a bug raised by the GSS: https://bugzilla.redhat.com/show_bug.cgi?id=1270276#c1 Just wanted to bring this to your notice, and understand your perspective on the same. I think you need one adjustment and one addition. As noted in my comment 2 above, you should pass the the new private key to the scp command to ease the copy of the keypair to the other nodes. Since you've already done the ssh-copy-id to all nodes, the public key is in the authorized_keys files on those nodes, so this will allow us to do the scp command passwordless. scp -i /var/lib/glusterd/nfs/secret.pem /var/lib/glusterd/nfs/secret.* <node-ip/hostname>:/var/lib/glusterd/nfs/ As noted in my comment 3 above, the node where the keypair is generated also needs to have the public key copied to its own authorized_keys file for the root user. Without this, passwordless ssh from the other nodes will fail even after performing the scp above. You can either include the local node in the list of nodes when performing the ssh-copy-id command, or you can directly add the public key to the local authorized_keys file with the command below. cat /var/lib/glusterd/nfs/secret.pem.pub >> /root/.ssh/authorized_keys NOTE: I believe the request in BZ 1270276 to remove this scp command was misguided. The ssh-copy-id command only copies the public key to the authorized_keys file, meaning that the recipient node can become a destination for passwordless ssh only. In order for that node to be the source of the passwordless ssh, it needs the copy of the private key file, and that is accomplished with the scp command above. QA testing here is straightforward -- Simply ensure that each node can perform passwordless ssh to all other nodes when passing the generated private key file '/var/lib/glusterd/nfs/secret.pem' to the ssh command with the '-i' flag. Site notes (not blockers for this BZ).... I still don't have a straight answer on _why_ passwordless ssh is even needed, but according to upstream references it is. So all of this BZ is proceeding with the assumption that passwordless ssh is indeed required. I'm sure after this there will be security-related arguments for us to move away from this configuration in the future. Also, even if we do keep root (or unprivileged user) passwordless ssh between the nodes, it's probably a better idea from a security perspective to generate unique ssh keypairs for each node instead of copying the same keypair to all nodes as in this process. The manual process for this would include significantly more steps, as the user would need to run separate ssh-keygen and ssh-copy-id commands on every node in the NFS-Ganesha HA cluster. (In reply to Dustin Black from comment #5) > I still don't have a straight answer on _why_ passwordless ssh is even > needed, but according to upstream references it is. So all of this BZ is > proceeding with the assumption that passwordless ssh is indeed required. I'm > sure after this there will be security-related arguments for us to move away > from this configuration in the future. Following up on the point above... The /usr/libexec/ganesha/ganesha-ha.sh script utilizes the generated /var/lib/glusterd/nfs/secret.pem private key for passwordless ssh to the other nodes. Thus, if universal passwordless ssh is not enabled specifically using this key as described in the directions above, a user will not be able to execute commands that use this script from any node in the cluster. Thanks for the detailed description Dustin. I have edited the command and mentioned that the remaining commands have to be executed on all the nodes including node1 http://jenkinscat.gsslab.pnq.redhat.com:8080/job/doc-Red_Hat_Gluster_Storage-3.1-3.1.2-Stage_Administration_Guide%20%28html-single%29/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#sect-NFS_Ganesha Let me know if this addresses the issue. For the last instruction: "Copy the ssh keypair from node1 to all the nodes (including node1) in the Ganesha-HA cluster by executing the following command for every node" The keypair actually doesn't need to be copied to the "node1" example node, as this is where it already exists and is being copied from. Also, for consistency in the command instructions, you should probably add the 'root@' user designation to the scp command. And that actually makes me think.... We should probably be explicit at the head of this set of instructions that passwordless ssh for the root user is required. We imply it with the # command prompts, but we don't actually say it. Thanks Dustin. I have made the necessary changes as suggested: http://10.34.3.139:8080/view/Gluster/job/doc-Red_Hat_Gluster_Storage-3.1-Administration_Guide%20%28html-single%29/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#sect-NFS_Ganesha Verified this bug according to the newly provided doc link: http://10.34.3.139:8080/view/Gluster/job/doc-Red_Hat_Gluster_Storage-3.1.2-Administration_Guide%20%28html-single%29/lastSuccessfulBuild/artifact/tmp/en-US/html-single/index.html#sect-NFS_Ganesha Below are the steps followed for the verification: On a cluster of 4 nodes: dhcp43-188.lab.eng.blr.redhat.com dhcp42-83.lab.eng.blr.redhat.com dhcp42-115.lab.eng.blr.redhat.com dhcp43-133.lab.eng.blr.redhat.com >>>> On one of the nodes (node1) in the cluster, run: ssh-keygen -f /var/lib/glusterd/nfs/secret.pem -t rsa -N '' [root@dhcp43-188 ~]# ssh-keygen -f /var/lib/glusterd/nfs/secret.pem -t rsa -N '' Generating public/private rsa key pair. Your identification has been saved in /var/lib/glusterd/nfs/secret.pem. Your public key has been saved in /var/lib/glusterd/nfs/secret.pem.pub. The key fingerprint is: 1c:a5:cf:c6:3d:c0:24:bb:aa:33:7b:79:8a:9b:cf:1a root.eng.blr.redhat.com The key's randomart image is: +--[ RSA 2048]----+ | . o | | B | | + o | | . * o | | S = o | | . . . | | E .. | | o*o . | | BO+o | +-----------------+ [root@dhcp43-188 ~]# >>>>> Deploy the generated public key from node1 to all the nodes (including node1) by executing the following command for every node: [root@dhcp43-188 ~]# ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root.eng.blr.redhat.com The authenticity of host 'dhcp42-83.lab.eng.blr.redhat.com (10.70.42.83)' can't be established. ECDSA key fingerprint is a8:26:6b:da:bc:54:e0:b8:92:75:04:05:66:19:cd:e8. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root.eng.blr.redhat.com's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root.eng.blr.redhat.com'" and check to make sure that only the key(s) you wanted were added. [root@dhcp43-188 ~]# ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root.eng.blr.redhat.com The authenticity of host 'dhcp42-115.lab.eng.blr.redhat.com (10.70.42.115)' can't be established. ECDSA key fingerprint is bd:7d:8a:eb:96:fb:61:7b:b0:9c:5e:99:8e:6a:a9:02. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root.eng.blr.redhat.com's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root.eng.blr.redhat.com'" and check to make sure that only the key(s) you wanted were added. [root@dhcp43-188 ~]# ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root.eng.blr.redhat.com The authenticity of host 'dhcp43-133.lab.eng.blr.redhat.com (10.70.43.133)' can't be established. ECDSA key fingerprint is d2:f2:e9:c2:94:73:f3:c5:aa:c5:73:94:8c:5c:68:4e. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root.eng.blr.redhat.com's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root.eng.blr.redhat.com'" and check to make sure that only the key(s) you wanted were added. [root@dhcp43-188 ~]# ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root.eng.blr.redhat.com The authenticity of host 'dhcp43-188.lab.eng.blr.redhat.com (10.70.43.188)' can't be established. ECDSA key fingerprint is 7c:17:6e:34:0a:13:d4:9a:61:62:8b:eb:1e:dd:32:ef. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root.eng.blr.redhat.com's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root.eng.blr.redhat.com'" and check to make sure that only the key(s) you wanted were added. >>>>>> Copy the ssh keypair from node1 to all the nodes in the Ganesha-HA cluster by executing the following command for every node: [root@dhcp43-188 ~]# scp -i /var/lib/glusterd/nfs/secret.pem /var/lib/glusterd/nfs/secret.* root.eng.blr.redhat.com:/var/lib/glusterd/nfs/ secret.pem 100% 1679 1.6KB/s 00:00 secret.pem.pub 100% 420 0.4KB/s 00:00 [root@dhcp43-188 ~]# scp -i /var/lib/glusterd/nfs/secret.pem /var/lib/glusterd/nfs/secret.* root.eng.blr.redhat.com:/var/lib/glusterd/nfs/ secret.pem 100% 1679 1.6KB/s 00:00 secret.pem.pub 100% 420 0.4KB/s 00:00 [root@dhcp43-188 ~]# scp -i /var/lib/glusterd/nfs/secret.pem /var/lib/glusterd/nfs/secret.* root.eng.blr.redhat.com:/var/lib/glusterd/nfs/ secret.pem 100% 1679 1.6KB/s 00:00 secret.pem.pub 100% 420 0.4KB/s 00:00 >>>>> Verifying the passwordless ssh between the nodes of cluster >>>> Doing an passwordless ssh from first node to other nodes: [root@dhcp43-188 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com Last login: Wed Apr 27 20:19:09 2016 from dhcp43-188.lab.eng.blr.redhat.com [root@dhcp42-83 ~]# exit logout Connection to dhcp42-83.lab.eng.blr.redhat.com closed. [root@dhcp43-188 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com Last login: Wed Apr 27 20:18:58 2016 from dhcp43-188.lab.eng.blr.redhat.com [root@dhcp42-115 ~]# exit logout Connection to dhcp42-115.lab.eng.blr.redhat.com closed. [root@dhcp43-188 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com Last login: Wed Apr 27 20:18:49 2016 from dhcp43-188.lab.eng.blr.redhat.com [root@dhcp43-133 ~]# exit logout Connection to dhcp43-133.lab.eng.blr.redhat.com closed. >>>> Doing an passwordless ssh from second node to other nodes: [root@dhcp42-83 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp43-188.lab.eng.blr.redhat.com (10.70.43.188)' can't be established. ECDSA key fingerprint is 7c:17:6e:34:0a:13:d4:9a:61:62:8b:eb:1e:dd:32:ef. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp43-188.lab.eng.blr.redhat.com,10.70.43.188' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 18:39:37 2016 from dhcp-0-194.blr.redhat.com [root@dhcp43-188 ~]# exit logout Connection to dhcp43-188.lab.eng.blr.redhat.com closed. [root@dhcp42-83 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp42-115.lab.eng.blr.redhat.com (10.70.42.115)' can't be established. ECDSA key fingerprint is bd:7d:8a:eb:96:fb:61:7b:b0:9c:5e:99:8e:6a:a9:02. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp42-115.lab.eng.blr.redhat.com,10.70.42.115' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 22:31:30 2016 from dhcp43-188.lab.eng.blr.redhat.com [root@dhcp42-115 ~]# exit logout Connection to dhcp42-115.lab.eng.blr.redhat.com closed. [root@dhcp42-83 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp43-133.lab.eng.blr.redhat.com (10.70.43.133)' can't be established. ECDSA key fingerprint is d2:f2:e9:c2:94:73:f3:c5:aa:c5:73:94:8c:5c:68:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp43-133.lab.eng.blr.redhat.com,10.70.43.133' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 22:31:42 2016 from dhcp43-188.lab.eng.blr.redhat.com [root@dhcp43-133 ~]# exit logout Connection to dhcp43-133.lab.eng.blr.redhat.com closed. >>>> Doing an passwordless ssh from third node to other nodes: [root@dhcp42-115 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp43-188.lab.eng.blr.redhat.com (10.70.43.188)' can't be established. ECDSA key fingerprint is 7c:17:6e:34:0a:13:d4:9a:61:62:8b:eb:1e:dd:32:ef. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp43-188.lab.eng.blr.redhat.com,10.70.43.188' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 22:33:50 2016 from dhcp42-83.lab.eng.blr.redhat.com [root@dhcp43-188 ~]# exit logout Connection to dhcp43-188.lab.eng.blr.redhat.com closed. [root@dhcp42-115 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp42-83.lab.eng.blr.redhat.com (10.70.42.83)' can't be established. ECDSA key fingerprint is a8:26:6b:da:bc:54:e0:b8:92:75:04:05:66:19:cd:e8. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp42-83.lab.eng.blr.redhat.com,10.70.42.83' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 22:36:56 2016 from dhcp42-115.lab.eng.blr.redhat.com [root@dhcp42-83 ~]# exit logout Connection to dhcp42-83.lab.eng.blr.redhat.com closed. [root@dhcp42-115 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp43-133.lab.eng.blr.redhat.com (10.70.43.133)' can't be established. ECDSA key fingerprint is d2:f2:e9:c2:94:73:f3:c5:aa:c5:73:94:8c:5c:68:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp43-133.lab.eng.blr.redhat.com,10.70.43.133' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 22:35:49 2016 from dhcp42-83.lab.eng.blr.redhat.com [root@dhcp43-133 ~]# exit logout Connection to dhcp43-133.lab.eng.blr.redhat.com closed. >>>> Doing an passwordless ssh from fourth node to other nodes: [root@dhcp43-133 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp43-188.lab.eng.blr.redhat.com (10.70.43.188)' can't be established. ECDSA key fingerprint is 7c:17:6e:34:0a:13:d4:9a:61:62:8b:eb:1e:dd:32:ef. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp43-188.lab.eng.blr.redhat.com,10.70.43.188' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 22:36:20 2016 from dhcp42-115.lab.eng.blr.redhat.com [root@dhcp43-188 ~]# exit logout Connection to dhcp43-188.lab.eng.blr.redhat.com closed. [root@dhcp43-133 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp42-83.lab.eng.blr.redhat.com (10.70.42.83)' can't be established. ECDSA key fingerprint is a8:26:6b:da:bc:54:e0:b8:92:75:04:05:66:19:cd:e8. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp42-83.lab.eng.blr.redhat.com,10.70.42.83' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 22:37:50 2016 from dhcp42-115.lab.eng.blr.redhat.com [root@dhcp42-83 ~]# exit logout Connection to dhcp42-83.lab.eng.blr.redhat.com closed. [root@dhcp43-133 ~]# ssh -i /var/lib/glusterd/nfs/secret.pem root.eng.blr.redhat.com The authenticity of host 'dhcp42-115.lab.eng.blr.redhat.com (10.70.42.115)' can't be established. ECDSA key fingerprint is bd:7d:8a:eb:96:fb:61:7b:b0:9c:5e:99:8e:6a:a9:02. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'dhcp42-115.lab.eng.blr.redhat.com,10.70.42.115' (ECDSA) to the list of known hosts. Last login: Wed Apr 27 22:35:07 2016 from dhcp42-83.lab.eng.blr.redhat.com [root@dhcp42-115 ~]# exit logout Connection to dhcp42-115.lab.eng.blr.redhat.com closed. As can be seen above, by following the given document, we are able to do a passowrdless ssh between the nodes in a cluster. Hence, based on the content of the document and the verification results, marking this bug as Verified. Admin Guide is published live on the customer portal: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html-single/Administration_Guide/index.html#sect-NFS_Ganesha |