Bug 1087960 - Client hangs when accessing a file, nothing logged
Summary: Client hangs when accessing a file, nothing logged
Keywords:
Status: CLOSED EOL
Alias: None
Product: GlusterFS
Classification: Community
Component: fuse
Version: 3.4.2
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-04-15 17:42 UTC by Justin Dossey
Modified: 2015-10-07 13:50 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-10-07 13:49:43 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:

Attachments (Terms of Use)

Description Justin Dossey 2014-04-15 17:42:37 UTC 
This is very similar to 799856.

I have a file which, when accessed on the client, causes the caller to hang.  Nothing is logged on the client or servers when this happens.

Details:

The client is running 3.4.2 and the servers are running 3.3.2.

The file I'm trying to access is from the client: 
/home/om/UDS8/7a/18/0f/rtouchtone66815/mail/entries/inbox/welcome.dump

The mountpoint is /home/om/UDS8. 

On the client:
strace ls -l /home/om/UDS8/7a/18/0f/rtouchtone66815/mail/entries/inbox/welcome.dump 
hangs at 
lstat64("/home/om/UDS8/7a/18/0f/rtouchtone66815/mail/entries/inbox/welcome.dump", 

On the servers:
uds-06
# file: export/brick2/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox/welcome.dump
trusted.gfid=0x2ebbb6e7256e46a5b6ca76cccfe37eec

getfattr: Removing leading '/' from absolute path names
uds-07
# file: export/brick2/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox/welcome.dump
trusted.gfid=0x2ebbb6e7256e46a5b6ca76cccfe37eec

The parent directory:

root@uds-08:/var/log/glusterfs# for i in uds-0{3,4,5,6,7,8} ; do echo $i ; ssh $i getfattr -m gfid -d -e hex /export/brick\?/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox   ; done 
uds-03
# file: export/brick1/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

getfattr: Removing leading '/' from absolute path names
uds-04
# file: export/brick1/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

# file: export/brick2/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

getfattr: Removing leading '/' from absolute path names
uds-05
# file: export/brick1/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

# file: export/brick2/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

getfattr: Removing leading '/' from absolute path names
uds-06
# file: export/brick1/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

# file: export/brick2/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

getfattr: Removing leading '/' from absolute path names
uds-07
# file: export/brick1/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

# file: export/brick2/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

getfattr: Removing leading '/' from absolute path names
uds-08
# file: export/brick1/vol1/7a/18/0f/rtouchtone66815/mail/entries/inbox
trusted.gfid=0x844bbed45b704ee7ba4ed742d5bee478

getfattr: Removing leading '/' from absolute path names

Looking at the GFID files:
root@uds-08:/var/log/glusterfs# for i in uds-0{3,4,5,6,7,8} ; do echo $i ; ssh $i ls -l /export/brick\?/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec; ssh $i getfattr -m gfid -d -e hex /export/brick\?/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec   ; done 

uds-06
-rw-r--r-- 2 apache apache 340 Feb  2 14:46 /export/brick2/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec
# file: export/brick2/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec
trusted.gfid=0x2ebbb6e7256e46a5b6ca76cccfe37eec

getfattr: Removing leading '/' from absolute path names
uds-07
-rw-r--r-- 1 apache apache   0 Apr  1 22:11 /export/brick1/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec
-rw-r--r-- 2 apache apache 340 Feb  2 14:46 /export/brick2/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec
# file: export/brick1/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec
trusted.gfid=0x2ebbb6e7256e46a5b6ca76cccfe37eec

# file: export/brick2/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec
trusted.gfid=0x2ebbb6e7256e46a5b6ca76cccfe37eec

getfattr: Removing leading '/' from absolute path names
uds-08
-rw-r--r-- 1 apache apache 0 Apr  1 22:11 /export/brick1/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec
getfattr: Removing leading '/' from absolute path names
# file: export/brick1/vol1/.glusterfs/2e/bb/2ebbb6e7-256e-46a5-b6ca-76cccfe37eec
trusted.gfid=0x2ebbb6e7256e46a5b6ca76cccfe37eec
Comment 1 Justin Dossey 2014-04-18 21:02:24 UTC 
More reseach here:
the GFID file for /export/brick1/vol1//7a/18/0f/rtouchtone66815/mail is /export/brick1/vol1/.glusterfs/87/5c/875c392c-4a4b-4a68-a14c-dcba49ba9840 .

root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/.glusterfs/87/5c/875c392c-4a4b-4a68-a14c-dcba49ba9840
lrwxrwxrwx 1 root root 53 Feb  2 14:46 /export/brick1/vol1/.glusterfs/87/5c/875c392c-4a4b-4a68-a14c-dcba49ba9840 -> ../../a3
/53/a3532ea7-c3eb-44dd-8a7a-62e6f94d902c/mail
root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/.glusterfs/a3/53/a3532ea7-c3eb-44dd-8a7a-62e6f94d902c
lrwxrwxrwx 1 root root 64 Jan 18 09:03 /export/brick1/vol1/.glusterfs/a3/53/a3532ea7-c3eb-44dd-8a7a-62e6f94d902c -> ../../5a
/ed/5aede284-429c-474f-9781-cd4a6b093a6e/rtouchtone66815
root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/.glusterfs/5a/ed/5aede284-429c-474f-9781-cd4a6b093a6e
lrwxrwxrwx 1 root root 51 Jan 18 09:03 /export/brick1/vol1/.glusterfs/5a/ed/5aede284-429c-474f-9781-cd4a6b093a6e -> ../../10/03/1003a10c-4a44-411f-8f05-9837603ee1b2/0f
root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/.glusterfs/10/03/1003a1
1003a10c-4a44-411f-8f05-9837603ee1b2/ 1003a131-e79a-431d-bd45-35edc777be79/ 1003a1f7-0834-43ba-9d51-d85bf6d77b51/
root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/.glusterfs/10/03/1003a10c-4a44-411f-8f05-9837603ee1b2
lrwxrwxrwx 1 root root 51 Aug 15  2013 /export/brick1/vol1/.glusterfs/10/03/1003a10c-4a44-411f-8f05-9837603ee1b2 -> ../../61/e3/61e3116a-6d83-4c51-8186-1159748e96db/18
root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/.glusterfs/61/e3/61e3116a-6d83-4c51-8186-1159748e96db
lrwxrwxrwx 1 root root 51 Jul  2  2013 /export/brick1/vol1/.glusterfs/61/e3/61e3116a-6d83-4c51-8186-1159748e96db -> ../../00/00/00000000-0000-0000-0000-000000000001/7a
root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/00/00/00000000-0000-0000-0000-000000000001/7a
ls: cannot access /export/brick1/vol1/00/00/00000000-0000-0000-0000-000000000001/7a: No such file or directory
root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/00/00/00000000-0000-0000-0000-000000000001
ls: cannot access /export/brick1/vol1/00/00/00000000-0000-0000-0000-000000000001: No such file or directory
root@uds-08:/export/brick1/vol1# ls -l /export/brick1/vol1/00/00/
total 0
drwxr-sr-x 3 apache apache 21 Jul 15  2013 07
drwxr-sr-x 3 apache apache 30 Nov 11 18:31 16
drwxr-xr-x 3 apache apache 22 Mar 23 17:25 24
drwxr-sr-x 3 apache apache 23 Aug 22  2013 2b
drwxr-sr-x 3 apache apache 27 Feb  7 04:19 3a
drwxr-sr-x 3 apache apache 26 Aug 29  2013 7f
drwxr-sr-x 3 apache apache 17 Apr  8 10:46 cf
Comment 2 Niels de Vos 2015-05-17 21:57:59 UTC 
GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5.

This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "bugs".

If there is no response by the end of the month, this bug will get automatically closed.
Comment 3 Kaleb KEITHLEY 2015-10-07 13:49:43 UTC 
GlusterFS 3.4.x has reached end-of-life.

If this bug still exists in a later release please reopen this and change the version or open a new bug.
Comment 4 Kaleb KEITHLEY 2015-10-07 13:50:53 UTC 
GlusterFS 3.4.x has reached end-of-life.\                                                   \                                                                               If this bug still exists in a later release please reopen this and change the version or open a new bug.
Note You need to log in before you can comment on or make changes to this bug.