Bug 1257546 - [Backup]: Glusterfind list shows the session as corrupted on the peer node
[Backup]: Glusterfind list shows the session as corrupted on the peer node
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: glusterfind (Show other bugs)
3.7.3
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: bugs@gluster.org
bugs@gluster.org
:
Depends On: 1232569 1243838
Blocks: 1223636
  Show dependency treegraph
 
Reported: 2015-08-27 06:10 EDT by Aravinda VK
Modified: 2016-04-19 03:46 EDT (History)
6 users (show)

See Also:
Fixed In Version: glusterfs-3.7.7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1243838
Environment:
Last Closed: 2016-03-22 04:19:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Aravinda VK 2015-08-27 06:10:03 EDT
+++ This bug was initially created as a clone of Bug #1243838 +++

+++ This bug was initially created as a clone of Bug #1232569 +++

Description of problem:

When a glusterfind session is created on one of the nodes, glusterfind list displays the healthy state of the session. However, on the peer node, it tries to open the status file at the $GLUSTERD_WORKDIR, not finding which, updates the state of the session as 'corrupted'.

The error displayed in the logs (at the peer node where it shows as failed):

[2015-06-17 15:15:24,124] ERROR [utils - 152:fail] - Error Opening Session file /var/lib/glusterd/glusterfind/sessn4/nash/status: [Errno 2] No such file or directory: '/var/lib/glusterd/glusterfind/sessn4/nash/status'


Version-Release number of selected component (if applicable):
glusterfs-3.7.1-3.el6rhs.x86_64

How reproducible: Always


Steps to Reproduce:
1. Have a >=2 node cluster, with a volume (of any type), say 'vol1'
2. Create a glusterfind session from node1 and verify the output of glusterfind list
3. Type the same command 'glusterfind list' on the peer node(s) and verify the output

Actual results:
On the peer node, step3 displays the session as 'corrupted'

Expected results:
The state of the glusterfind session should be displayed as healthy when viewed from any of the peer node

Additional info:

[root@dhcp43-93 ~]# 
[root@dhcp43-93 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
sessn2                    nash                      2015-06-16 20:17:24      
sessn3                    nash                      2015-06-16 17:47:02      
sessp1                    pluto                     2015-06-16 21:15:06      
sesso1                    ozone                     2015-06-15 23:48:42      
sessn1                    nash                      2015-06-16 18:02:11      
sessp2                    pluto                     2015-06-16 21:12:53      
[root@dhcp43-93 ~]# gluster v list
nash
[root@dhcp43-93 ~]# glusterfind create sessn4 nash
Failed to set volume option build-pgfid on: volume set: failed: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again

[root@dhcp43-93 ~]# glusterfind create sessn4 nash
Session sessn4 created with volume nash
[root@dhcp43-93 ~]# 
[root@dhcp43-93 ~]# 
[root@dhcp43-93 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
sessn2                    nash                      2015-06-16 20:17:24      
sessn3                    nash                      2015-06-16 17:47:02      
sessp1                    pluto                     2015-06-16 21:15:06      
sessn4                    nash                      2015-06-17 15:14:18      
sesso1                    ozone                     2015-06-15 23:48:42      
sessn1                    nash                      2015-06-16 18:02:11      
sessp2                    pluto                     2015-06-16 21:12:53      
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/
.keys/  sessn1/ sessn2/ sessn3/ sessn4/ sesso1/ sesso2/ sesso3/ sessp1/ sessp2/ sessv1/ 
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/
%2Frhs%2Fthinbrick1%2Fnash.status  %2Frhs%2Fthinbrick2%2Fnash.status  sessn4_nash_secret.pem             sessn4_nash_secret.pem.pub         status
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/^C
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/
.keys/  sessn1/ sessn2/ sessn3/ sessn4/ sesso1/ sesso2/ sesso3/ sessp1/ sessp2/ sessv1/ 
[root@dhcp43-93 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/
%2Frhs%2Fthinbrick1%2Fnash.status  %2Frhs%2Fthinbrick2%2Fnash.status  sessn4_nash_secret.pem             sessn4_nash_secret.pem.pub         status
[root@dhcp43-93 ~]# ls /var/log/glusterfs/glusterfind/
cli.log  nash/    sess21/  sessn1/  sessn2/  sessn3/  sessn4/  sesso1/  sesso2/  sesso3/  sessp1/  sessp2/  sessv1/  
[root@dhcp43-93 ~]# vi /var/log/glusterfs/glusterfind/sessn4/nash/cli.log



###############  peer node   ########################3

[root@dhcp43-155 ~]# 
[root@dhcp43-155 ~]# # after creating a new session 'sessn4'
[root@dhcp43-155 ~]# 
[root@dhcp43-155 ~]# 
[root@dhcp43-155 ~]# glusterfind list
SESSION                   VOLUME                    SESSION TIME             
---------------------------------------------------------------------------
sessp1                    pluto                     Session Corrupted        
sessn3                    nash                      Session Corrupted        
sessp2                    pluto                     Session Corrupted        
sessn4                    nash                      Session Corrupted        
sessn2                    nash                      Session Corrupted        
sessn1                    nash                      Session Corrupted        
sesso1                    ozone                     Session Corrupted        
[root@dhcp43-155 ~]# vi /var/log/glusterfs/glusterfind/
cli.log  sessn1/  sessn2/  sesso1/  sessp1/  sessp2/  
[root@dhcp43-155 ~]# glusterfind pre sessn4 nash /tmp/outn.txt
Error Opening Session file /var/lib/glusterd/glusterfind/sessn4/nash/status: [Errno 2] No such file or directory: '/var/lib/glusterd/glusterfind/sessn4/nash/status'
[root@dhcp43-155 ~]# ls /var/lib/glusterd/glusterfind/
.keys/  sessn1/ sessn2/ sessn3/ sessn4/ sesso1/ sesso2/ sesso3/ sessp1/ sessp2/ sessv1/ 
[root@dhcp43-155 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/%2Frhs%2Fthinbrick
%2Frhs%2Fthinbrick1%2Fnash.status  %2Frhs%2Fthinbrick2%2Fnash.status  
[root@dhcp43-155 ~]# ls /var/lib/glusterd/glusterfind/sessn4/nash/%2Frhs%2Fthinbrick^C
[root@dhcp43-155 ~]# ls /var/log/glusterfs/glusterfind/sessn4/nash/
cli.log
[root@dhcp43-155 ~]# vi /var/log/glusterfs/glusterfind/sessn4/nash/cli.log 
[root@dhcp43-155 ~]# rpm -qa | grep glustefs
[root@dhcp43-155 ~]# rpm -qa | grep glusterfs
glusterfs-api-3.7.1-3.el6rhs.x86_64
glusterfs-libs-3.7.1-3.el6rhs.x86_64
glusterfs-3.7.1-3.el6rhs.x86_64
glusterfs-fuse-3.7.1-3.el6rhs.x86_64
glusterfs-server-3.7.1-3.el6rhs.x86_64
glusterfs-client-xlators-3.7.1-3.el6rhs.x86_64
glusterfs-cli-3.7.1-3.el6rhs.x86_64
[root@dhcp43-155 ~]#

--- Additional comment from Aravinda VK on 2015-07-16 08:50:50 EDT ---

glusterfind commands will not work in peer nodes other than the initiated node. We need to enhance glusterfind to use Meta volume to save status files and collect pem keys from all nodes and distribute to all Nodes of cluster. With this enhancement, we can run glusterfind command from any peer node

--- Additional comment from Anand Avati on 2015-07-16 09:11:19 EDT ---

REVIEW: http://review.gluster.org/11699 (tools/glusterfind: Do not show session corrupted if no status file) posted (#1) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Anand Avati on 2015-08-27 02:28:26 EDT ---

REVIEW: http://review.gluster.org/11699 (tools/glusterfind: Do not show session corrupted if no status file) posted (#4) for review on master by Aravinda VK (avishwan@redhat.com)

--- Additional comment from Anand Avati on 2015-08-27 06:08:31 EDT ---

COMMIT: http://review.gluster.org/11699 committed in master by Venky Shankar (vshankar@redhat.com) 
------
commit cf3d6f14ae031ba2f5269cea6dbf80e60d00cce5
Author: Aravinda VK <avishwan@redhat.com>
Date:   Thu Jul 16 18:23:36 2015 +0530

    tools/glusterfind: Do not show session corrupted if no status file
    
    When a glusterfind session is created it creates session directories
    in all the nodes which are part of the Volume. But session status
    file only present in initiated node.
    
    Show Session corrupted only if status file exists and invalid content.
    
    Change-Id: I8443c0335b872645b54c0aa77d9893dbe2589c92
    Signed-off-by: Aravinda VK <avishwan@redhat.com>
    BUG: 1243838
    Reviewed-on: http://review.gluster.org/11699
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Milind Changire <mchangir@redhat.com>
    Reviewed-by: Kotresh HR <khiremat@redhat.com>
Comment 1 Aravinda VK 2015-08-27 06:13:41 EDT
Patch sent to release-3.7 branch
http://review.gluster.org/#/c/12028/
Comment 2 Vijay Bellur 2015-09-08 13:17:02 EDT
REVIEW: http://review.gluster.org/12028 (tools/glusterfind: Do not show session corrupted if no status file) posted (#2) for review on release-3.7 by Aravinda VK (avishwan@redhat.com)
Comment 3 Vijay Bellur 2015-11-18 23:02:33 EST
COMMIT: http://review.gluster.org/12028 committed in release-3.7 by Venky Shankar (vshankar@redhat.com) 
------
commit 6046c73eb86d2409b93ae59a4a59124eacb7c711
Author: Aravinda VK <avishwan@redhat.com>
Date:   Thu Jul 16 18:23:36 2015 +0530

    tools/glusterfind: Do not show session corrupted if no status file
    
    When a glusterfind session is created it creates session directories
    in all the nodes which are part of the Volume. But session status
    file only present in initiated node.
    
    Show Session corrupted only if status file exists and invalid content.
    
    Change-Id: I8443c0335b872645b54c0aa77d9893dbe2589c92
    Signed-off-by: Aravinda VK <avishwan@redhat.com>
    BUG: 1257546
    Reviewed-on: http://review.gluster.org/11699
    Reviewed-on: http://review.gluster.org/12028
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Venky Shankar <vshankar@redhat.com>
    Reviewed-by: Milind Changire <mchangir@redhat.com>
Comment 4 Kaushal 2016-04-19 03:46:47 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.7, please open a new bug report.

glusterfs-3.7.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-February/025292.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.