Bug 1234213

Summary: [Backup]: Password of the peer nodes prompted whenever a glusterfind session is deleted.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sweta Anandpara <sanandpa>
Component: glusterfindAssignee: Milind Changire <mchangir>
Status: CLOSED ERRATA QA Contact: Rahul Hinduja <rhinduja>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: annair, asrivast, avishwan, divya, khiremat, mchangir, rhinduja, rhs-bugs, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.1-14 Doc Type: Bug Fix
Doc Text:
Previously, on executing the 'glusterfind delete' command, the user was presented with password prompts for peer nodes. This was due to the peer node SSH key, set up for password-less SSH, getting deleted on local node before the keys got deleted on peer nodes. As a consequence, password prompts got displayed for all peer nodes all at once. The user had to enter passwords for all peer nodes as many times as there are peer nodes in the cluster. With this fix, checks have been added to avoid deleting SSH keys on local node before deleting them on peer node. The SSH keys on local node eventually get deleted as part of session cleanup. Now, the password prompts are no longer presented on executing a 'glusterfind delete' command.
 Story Points: ---
Clone Of:
: 1243785 1250410 (view as bug list) Environment:
Last Closed: 2015-10-05 07:14:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1223636, 1243785, 1250410, 1251815    

Description Sweta Anandpara 2015-06-22 04:31:31 UTC 
Description of problem:

When we delete a session using glusterfind delete, password of the peers is prompted. This happens even after the workaround suggested in bug 1229664 is executed:
restorecon -R -v /root

This does seem to be a different issue as I have seen this happen intermittently even in the previous build.

Version-Release number of selected component (if applicable):
glusterfs-3.7.1-4.el6rhs.x86_64

How reproducible: 80% of the times


Steps to Reproduce:
1. Have a >2 node cluster, create a volume and a glusterfind session corresponding to the volume.
2. Delete the session using glusterfind delete, and that is when a password is prompted for the peer node(s)


Additional info:

[root@dhcp43-191 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
fds                       ozone                     2015-06-22 14:58:44      
sesso3                    ozone                     2015-06-18 16:27:30      
sesso1                    ozone                     2015-06-19 23:20:19      
^%#@*                     ozone                     Session Corrupted        
sesso5                    ozone                     2015-06-20 00:18:03      
fdjksl                    ozone                     2015-06-22 14:58:09      
sesso2                    ozone                     2015-06-19 22:44:40      
5543                      ozone                     2015-06-22 14:59:10      
sesso4                    ozone                     2015-06-18 16:27:38      
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind delete 5543 ozone
root.42.147's password: root.42.30's password: root.42.202's password: root.42.147's password: 


root.42.30's password: 

root.42.147's password: 


root.42.147's password: 
Session 5543 with volume ozone deleted
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
fds                       ozone                     2015-06-22 14:58:44      
sesso3                    ozone                     2015-06-18 16:27:30      
sesso1                    ozone                     2015-06-19 23:20:19      
^%#@*                     ozone                     Session Corrupted        
sesso5                    ozone                     2015-06-20 00:18:03      
fdjksl                    ozone                     2015-06-22 14:58:09      
sesso2                    ozone                     2015-06-19 22:44:40      
sesso4                    ozone                     2015-06-18 16:27:38      
[root@dhcp43-191 ~]# glusterfind delete 5543 ozone
Invalid session 5543
[root@dhcp43-191 ~]# glusterfind delete fgdsdfd  ozone
Invalid session fgdsdfd
[root@dhcp43-191 ~]# glusterfind delete fdjksl ozone
root.42.147's password: root.42.30's password: root.42.147's password: 


root.42.30's password: 

root.42.30's password: 
Session fdjksl with volume ozone deleted
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# 
[root@dhcp43-191 ~]# glusterfind delete fds  ozone
root.42.147's password: root.42.147's password: 


root.42.147's password: 
Session fds with volume ozone deleted
[root@dhcp43-191 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
sesso3                    ozone                     2015-06-18 16:27:30      
sesso1                    ozone                     2015-06-19 23:20:19        
sesso5                    ozone                     2015-06-20 00:18:03      
sesso2                    ozone                     2015-06-19 22:44:40      
sesso4                    ozone                     2015-06-18 16:27:38      
[root@dhcp43-191 ~]#
Comment 3 Sweta Anandpara 2015-06-23 04:37:26 UTC 
Sosreports updated at: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1234213/
Comment 5 Aravinda VK 2015-07-02 07:02:07 UTC 
Delete command gets nodes list from gluster volume info, and cleans up the session directory one by one. If the first node in the list is main node then, it deletes the session directory. Ssh pem key present in session directory also gets deleted. After this all the cleanup commands will fail since ssh key not present for passwordless ssh login.

Fix:
Execute delete in the main node only when cleanup command completed in all the nodes.
Comment 8 Aravinda VK 2015-08-18 09:38:35 UTC 
Upstream Patch posted. 
Master: http://review.gluster.org/#/c/11693/
Release-3.7: http://review.gluster.org/#/c/11832/
Comment 9 Aravinda VK 2015-08-27 10:46:57 UTC 
Downstream patch https://code.engineering.redhat.com/gerrit/#/c/54471/
Comment 10 Rahul Hinduja 2015-09-04 12:28:14 UTC 
Verified with build: glusterfs-3.7.1-14.el7rhgs.x86_64

If a session is deleted from the node which is used to create the session, the passwords are not asked as:

[root@georep1 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
gfs                       master                    2015-09-04 11:59:30      
g                         master                    2015-09-04 12:14:10      
[root@georep1 ~]# glusterfind delete g master
Session g with volume master deleted
[root@georep1 ~]# 
[root@georep1 ~]# time glusterfind delete gfs master
Session gfs with volume master deleted

real	0m0.832s
user	0m1.230s
sys	0m0.432s
[root@georep1 ~]# glusterfind list
No sessions found
[root@georep1 ~]# 


But if a session is deleted from any other node in cluster, the passwords are asked as: 

[root@georep2 scripts]# glusterfind delete g master
root.46.93's password: root.46.154's password: root.46.96's password: root.46.96's password: root.46.93's password: root.46.96's password: root.46.93's password: root.46.154's password: root.46.154's password: 

According to the comment in bz: https://bugzilla.redhat.com/show_bug.cgi?id=1232569#c4
glusterfind commands will not work in peer nodes other than the initiated node

ADMIN Guide doc bug is raised to have a note:
https://bugzilla.redhat.com/show_bug.cgi?id=1260106

Moving this bug to verified as it doesn't ask for password from the node involved in creation of session.
Comment 11 Divya 2015-09-29 06:14:35 UTC 
Please review and sign-off the edited doc text.
Comment 12 Milind Changire 2015-09-29 08:15:33 UTC 
Doc Text looks good to me.
Comment 14 errata-xmlrpc 2015-10-05 07:14:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html