Bug 1260918

+++ This bug was initially created as a clone of Bug #1260119 +++

Description of problem:
======================

If more than 1 node from cluster do not have entry in the known_host of a node which is creating glusterfind session, the create hungs forever.

[root@georep1 scripts]# glusterfind create s1 master
The authenticity of host '10.70.46.97 (10.70.46.97)' can't be established.
ECDSA key fingerprint is 76:e4:6d:07:1e:82:26:1c:0a:95:b2:4c:a3:3f:f1:e2.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.154 (10.70.46.154)' can't be established.
ECDSA key fingerprint is b4:a8:00:41:ec:f8:12:a9:89:88:cb:7a:20:a8:83:3c.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.97 (10.70.46.97)' can't be established.
ECDSA key fingerprint is 76:e4:6d:07:1e:82:26:1c:0a:95:b2:4c:a3:3f:f1:e2.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.154 (10.70.46.154)' can't be established.
ECDSA key fingerprint is b4:a8:00:41:ec:f8:12:a9:89:88:cb:7a:20:a8:83:3c.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.97 (10.70.46.97)' can't be established.
ECDSA key fingerprint is 76:e4:6d:07:1e:82:26:1c:0a:95:b2:4c:a3:3f:f1:e2.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.93 (10.70.46.93)' can't be established.
ECDSA key fingerprint is 0d:bc:e3:70:e0:86:65:5e:3e:d2:ea:9c:fb:a9:53:66.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.93 (10.70.46.93)' can't be established.
ECDSA key fingerprint is 0d:bc:e3:70:e0:86:65:5e:3e:d2:ea:9c:fb:a9:53:66.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.93 (10.70.46.93)' can't be established.
ECDSA key fingerprint is 0d:bc:e3:70:e0:86:65:5e:3e:d2:ea:9c:fb:a9:53:66.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.154 (10.70.46.154)' can't be established.
ECDSA key fingerprint is b4:a8:00:41:ec:f8:12:a9:89:88:cb:7a:20:a8:83:3c.
Are you sure you want to continue connecting (yes/no)? yes


[root@georep1 scripts]# cat /root/.ssh/known_hosts


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.1-14.el7rhgs.x86_64

How reproducible:
=================

Always

Steps to Reproduce:
===================
1. Flush the known_hosts from node or remove the cluster host entries
2. Create glusterfind session


Actual results:
===============
glusterfind session creation hungs

Expected results:
================

Should create the session


Workaround:
===========

SSH to all the nodes in cluster to have known_hosts updated.

[root@georep1 scripts]# cat /root/.ssh/known_hosts
[root@georep1 scripts]# for i in {97,93,154}; do ssh root.46.$i; doneThe authenticity of host '10.70.46.97 (10.70.46.97)' can't be established.
ECDSA key fingerprint is 76:e4:6d:07:1e:82:26:1c:0a:95:b2:4c:a3:3f:f1:e2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.70.46.97' (ECDSA) to the list of known hosts.
root.46.97's password: 
Last login: Fri Sep  4 12:38:50 2015 from 10.70.6.115
[root@georep2 ~]# exit
logout
Connection to 10.70.46.97 closed.
The authenticity of host '10.70.46.93 (10.70.46.93)' can't be established.
ECDSA key fingerprint is 0d:bc:e3:70:e0:86:65:5e:3e:d2:ea:9c:fb:a9:53:66.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.70.46.93' (ECDSA) to the list of known hosts.
root.46.93's password: 
Last login: Fri Sep  4 12:38:50 2015 from 10.70.6.115
[root@georep3 ~]# exit
logout
Connection to 10.70.46.93 closed.
The authenticity of host '10.70.46.154 (10.70.46.154)' can't be established.
ECDSA key fingerprint is b4:a8:00:41:ec:f8:12:a9:89:88:cb:7a:20:a8:83:3c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.70.46.154' (ECDSA) to the list of known hosts.
root.46.154's password: 
Last login: Fri Sep  4 12:38:50 2015 from 10.70.6.115
[root@georep4 ~]# exit
logout
Connection to 10.70.46.154 closed.
[root@georep1 scripts]# cat /root/.ssh/known_hosts
10.70.46.97 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDWD/hxpscM20kGEWOTsiIzgmnBd78d2uyQRI7AGIX2JRRr0hIoZPOGCrW/ytRpluPEnJVr7s+vAYglVYLZlOo=
10.70.46.93 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGCE5KvbJtmgXmQXfVVjUVjG0bjkP7fb0v7owFJnzAxy5FKjtTDQSF+qVAHA17MBh9Br7KP+SZQOxSmHyY9Tq8s=
10.70.46.154 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNMULJn47vZ1Azq/SCi4i5VBSrLQAqs6sMZTSamzpwkhedtHrNhKe5QW7W5l+mirLJTIrLuqy8HQYSp5jDYyfrk=
[root@georep1 scripts]# glusterfind create s1 master
Session s1 created with volume master
[root@georep1 scripts]#

--- Additional comment from Rahul Hinduja on 2015-09-07 02:28:54 EDT ---

glusterfind pre also hungs for local host checking as: 


[root@georep1 scripts]# glusterfind pre --output-prefix '/mnt/glusterfs/' s1 master /root/log2
10.70.46.97 - pre failed: /rhs/brick2/b6 Historical Changelogs not available: [Errno 2] No such file or directory

10.70.46.97 - pre failed: /rhs/brick3/b10 Historical Changelogs not available: [Errno 2] No such file or directory

10.70.46.97 - pre failed: /rhs/brick1/b2 Historical Changelogs not available: [Errno 2] No such file or directory

10.70.46.93 - pre failed: /rhs/brick1/b3 Historical Changelogs not available: [Errno 2] No such file or directory

10.70.46.93 - pre failed: /rhs/brick2/b7 Historical Changelogs not available: [Errno 2] No such file or directory

10.70.46.93 - pre failed: /rhs/brick3/b11 Historical Changelogs not available: [Errno 2] No such file or directory

10.70.46.154 - pre failed: /rhs/brick2/b8 Historical Changelogs not available: [Errno 2] No such file or directory

10.70.46.154 - pre failed: /rhs/brick3/b12 Historical Changelogs not available: [Errno 2] No such file or directory

10.70.46.154 - pre failed: /rhs/brick1/b4 Historical Changelogs not available: [Errno 2] No such file or directory

The authenticity of host '10.70.46.96 (10.70.46.96)' can't be established.
ECDSA key fingerprint is 44:23:1a:4b:3c:78:63:a6:66:3d:18:01:8d:dd:17:74.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.96 (10.70.46.96)' can't be established.
ECDSA key fingerprint is 44:23:1a:4b:3c:78:63:a6:66:3d:18:01:8d:dd:17:74.
Are you sure you want to continue connecting (yes/no)? The authenticity of host '10.70.46.96 (10.70.46.96)' can't be established.
ECDSA key fingerprint is 44:23:1a:4b:3c:78:63:a6:66:3d:18:01:8d:dd:17:74.
Are you sure you want to continue connecting (yes/no)? yes

--- Additional comment from Aravinda VK on 2015-09-07 06:52:20 EDT ---

RCA:

While connecting to other nodes programatically, Geo-rep uses an additional option with ssh(-oStrictHostKeyChecking=no). We need to use the option with Glusterfind too.

Other issue is about asking yes/no prompt for localhost, which is during scp command. We need to use the same option as used in ssh. Other fix is required in not running scp command if local node.

Workaround:
Add all the hosts in peer including local node to known_hosts.