Bug 1243838

Summary:	[Backup]: Glusterfind list shows the session as corrupted on the peer node
Product:	[Community] GlusterFS	Reporter:	Aravinda VK <avishwan>
Component:	glusterfind	Assignee:	Aravinda VK <avishwan>
Status:	CLOSED CURRENTRELEASE	QA Contact:	bugs <bugs>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	avishwan, khiremat, mchangir, sanandpa, smohan
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.8rc2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1232569
Clones:	1257546 (view as bug list)		Environment:
Last Closed:	2016-06-16 13:24:44 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1232569
Bug Blocks:	1223636, 1257546

Description Aravinda VK 2015-07-16 12:41:46 UTC

+++ This bug was initially created as a clone of Bug #1232569 +++

Description of problem:

When a glusterfind session is created on one of the nodes, glusterfind list displays the healthy state of the session. However, on the peer node, it tries to open the status file at the $GLUSTERD_WORKDIR, not finding which, updates the state of the session as 'corrupted'.

The error displayed in the logs (at the peer node where it shows as failed):

[2015-06-17 15:15:24,124] ERROR [utils - 152:fail] - Error Opening Session file /var/lib/glusterd/glusterfind/sessn4/nash/status: [Errno 2] No such file or directory: '/var/lib/glusterd/glusterfind/sessn4/nash/status'


Version-Release number of selected component (if applicable):
glusterfs-3.7.1-3.el6rhs.x86_64

How reproducible: Always


Steps to Reproduce:
1. Have a >=2 node cluster, with a volume (of any type), say 'vol1'
2. Create a glusterfind session from node1 and verify the output of glusterfind list
3. Type the same command 'glusterfind list' on the peer node(s) and verify the output

Actual results:
On the peer node, step3 displays the session as 'corrupted'

Expected results:
The state of the glusterfind session should be displayed as healthy when viewed from any of the peer node

Additional info:

[root@dhcp43-93 ~]# 
[root@dhcp43-93 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
sessn2                    nash                      2015-06-16 20:17:24      
sessn3                    nash                      2015-06-16 17:47:02      
sessp1                    pluto                     2015-06-16 21:15:06      
sesso1                    ozone                     2015-06-15 23:48:42      
sessn1                    nash                      2015-06-16 18:02:11      
sessp2                    pluto                     2015-06-16 21:12:53      
[root@dhcp43-93 ~]# gluster v list
nash
[root@dhcp43-93 ~]# glusterfind create sessn4 nash
Failed to set volume option build-pgfid on: volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again

[root@dhcp43-93 ~]# glusterfind create sessn4 nash
Session sessn4 created with volume nash
[root@dhcp43-93 ~]# 
[root@dhcp43-93 ~]# 
[root@dhcp43-93 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
sessn2                    nash                      2015-06-16 20:17:24      
sessn3                    nash                      2015-06-16 17:47:02      
sessp1                    pluto                     2015-06-16 21:15:06      
sessn4                    nash                      2015-06-17 15:14:18      
sesso1                    ozone                     2015-06-15 23:48:42      
sessn1                    nash                      2015-06-16 18:02:11      
sessp2                    pluto                     2015-06-16 21:12:53      
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/
.keys/  sessn1/ sessn2/ sessn3/ sessn4/ sesso1/ sesso2/ sesso3/ sessp1/ sessp2/ sessv1/ 
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/
%2Frhs%2Fthinbrick1%2Fnash.status  %2Frhs%2Fthinbrick2%2Fnash.status  sessn4_nash_secret.pem             sessn4_nash_secret.pem.pub         status
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/^C
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/
.keys/  sessn1/ sessn2/ sessn3/ sessn4/ sesso1/ sesso2/ sesso3/ sessp1/ sessp2/ sessv1/ 
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/
%2Frhs%2Fthinbrick1%2Fnash.status  %2Frhs%2Fthinbrick2%2Fnash.status  sessn4_nash_secret.pem             sessn4_nash_secret.pem.pub         status
[root@dhcp43-93 ~]# ls /var/log/glusterfs/glusterfind/
cli.log  nash/    sess21/  sessn1/  sessn2/  sessn3/  sessn4/  sesso1/  sesso2/  sesso3/  sessp1/  sessp2/  sessv1/  
[root@dhcp43-93 ~]# vi /var/log/glusterfs/glusterfind/sessn4/nash/cli.log



###############  peer node   ########################3

[root@dhcp43-155 ~]# 
[root@dhcp43-155 ~]# # after creating a new session 'sessn4'
[root@dhcp43-155 ~]# 
[root@dhcp43-155 ~]# 
[root@dhcp43-155 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
sessp1                    pluto                     Session Corrupted        
sessn3                    nash                      Session Corrupted        
sessp2                    pluto                     Session Corrupted        
sessn4                    nash                      Session Corrupted        
sessn2                    nash                      Session Corrupted        
sessn1                    nash                      Session Corrupted        
sesso1                    ozone                     Session Corrupted        
[root@dhcp43-155 ~]# vi /var/log/glusterfs/glusterfind/
cli.log  sessn1/  sessn2/  sesso1/  sessp1/  sessp2/  
[root@dhcp43-155 ~]# glusterfind pre sessn4 nash /tmp/outn.txt
Error Opening Session file /var/lib/glusterd/glusterfind/sessn4/nash/status: [Errno 2] No such file or directory: '/var/lib/glusterd/glusterfind/sessn4/nash/status'
[root@dhcp43-155 ~]# ls /var/lib/glusterd/glusterfind/
.keys/  sessn1/ sessn2/ sessn3/ sessn4/ sesso1/ sesso2/ sesso3/ sessp1/ sessp2/ sessv1/ 
[root@dhcp43-155 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/%2Frhs%2Fthinbrick
%2Frhs%2Fthinbrick1%2Fnash.status  %2Frhs%2Fthinbrick2%2Fnash.status  
[root@dhcp43-155 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/%2Frhs%2Fthinbrick^C
[root@dhcp43-155 ~]# ls /var/log/glusterfs/glusterfind/sessn4/nash/
cli.log
[root@dhcp43-155 ~]# vi /var/log/glusterfs/glusterfind/sessn4/nash/cli.log 
[root@dhcp43-155 ~]# rpm -qa | grep glustefs
[root@dhcp43-155 ~]# rpm -qa | grep glusterfs
glusterfs-api-3.7.1-3.el6rhs.x86_64
glusterfs-libs-3.7.1-3.el6rhs.x86_64
glusterfs-3.7.1-3.el6rhs.x86_64
glusterfs-fuse-3.7.1-3.el6rhs.x86_64
glusterfs-server-3.7.1-3.el6rhs.x86_64
glusterfs-client-xlators-3.7.1-3.el6rhs.x86_64
glusterfs-cli-3.7.1-3.el6rhs.x86_64
[root@dhcp43-155 ~]#

Comment 1 Aravinda VK 2015-07-16 12:50:50 UTC

glusterfind commands will not work in peer nodes other than the initiated node. We need to enhance glusterfind to use Meta volume to save status files and collect pem keys from all nodes and distribute to all Nodes of cluster. With this enhancement, we can run glusterfind command from any peer node

Comment 2 Anand Avati 2015-07-16 13:11:19 UTC

REVIEW: http://review.gluster.org/11699 (tools/glusterfind: Do not show session corrupted if no status file) posted (#1) for review on master by Aravinda VK (avishwan)

Comment 3 Anand Avati 2015-08-27 06:28:26 UTC

REVIEW: http://review.gluster.org/11699 (tools/glusterfind: Do not show session corrupted if no status file) posted (#4) for review on master by Aravinda VK (avishwan)

Comment 4 Anand Avati 2015-08-27 10:08:31 UTC

COMMIT: http://review.gluster.org/11699 committed in master by Venky Shankar (vshankar) 
------
commit cf3d6f14ae031ba2f5269cea6dbf80e60d00cce5
Author: Aravinda VK <avishwan>
Date:   Thu Jul 16 18:23:36 2015 +0530

    tools/glusterfind: Do not show session corrupted if no status file
    
    When a glusterfind session is created it creates session directories
    in all the nodes which are part of the Volume. But session status
    file only present in initiated node.
    
    Show Session corrupted only if status file exists and invalid content.
    
    Change-Id: I8443c0335b872645b54c0aa77d9893dbe2589c92
    Signed-off-by: Aravinda VK <avishwan>
    BUG: 1243838
    Reviewed-on: http://review.gluster.org/11699
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Milind Changire <mchangir>
    Reviewed-by: Kotresh HR <khiremat>

Comment 5 Niels de Vos 2016-06-16 13:24:44 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user