Hide Forgot
Server and client on a same machine. $ uname -a Linux repository 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux $ cat /etc/redhat-release CentOS release 5.5 (Final) Given volfile: (comments removed) +------------------------------------------------------------------------------+ 1: volume smb01-part01-posix 2: type storage/posix 3: option directory /mnt/gluster01/smb01.part01 5: end-volume 6: 7: volume smb01-part01-acl 8: type features/access-control 9: subvolumes smb01-part01-posix 10: end-volume 11: 12: volume smb01-part01-locks 13: type features/locks 14: option mandatory-locks on 15: subvolumes smb01-part01-acl 16: end-volume 17: 23: volume smb01-part02-posix 24: type storage/posix 25: option directory /mnt/gluster02/smb01.part02 27: end-volume 28: 29: volume smb01-part02-acl 30: type features/access-control 31: subvolumes smb01-part02-posix 32: end-volume 33: 34: volume smb01-part02-locks 35: type features/locks 36: option mandatory-locks on 37: subvolumes smb01-part02-acl 38: end-volume 39: 45: volume smb01-part03-posix 46: type storage/posix 47: option directory /mnt/gluster03/smb01.part03 49: end-volume 50: 51: volume smb01-part03-acl 52: type features/access-control 53: subvolumes smb01-part03-posix 54: end-volume 55: 56: volume smb01-part03-locks 57: type features/locks 58: option mandatory-locks on 59: subvolumes smb01-part03-acl 60: end-volume 61: 67: volume smb01-client 68: type cluster/distribute 70: subvolumes smb01-part01-locks smb01-part02-locks smb01-part03-locks 71: end-volume 72: 73: volume smb01-write-behind 74: type performance/write-behind 75: subvolumes smb01-client 76: end-volume 77: 83: volume smb01-io-stats 84: type debug/io-stats 85: option latency-measurement on 86: subvolumes smb01-write-behind 87: end-volume 88: 89: volume smb01-stat-prefetch 90: type performance/stat-prefetch 91: subvolumes smb01-io-stats 92: end-volume 93: 94: volume smb01-io-threads 95: type performance/io-threads 96: subvolumes smb01-stat-prefetch 97: end-volume 98: 99: volume smb01 100: type performance/io-cache 101: subvolumes smb01-io-threads 102: end-volume +------------------------------------------------------------------------------+ GlusterFS over ext3 used as backend for CIFS with SAMBA 3.0.33-3.29.2 All I/O over regular CIFS shares works perfectly. SAMBA configured with recycle VFS: [sharename] ... vfs objects = recycle recycle:repository = /mnt/smb01/recycled/sharename recycle:maxsize = 100000000 recycle:versions = yes recycle:keeptree = yes But when I try to delete file from .../recycled/ tree, I see a strange situation (all dirnames/filenames replaced as unimportant): [root@host bar]# ls -la total 28 drwxr-xr-x 2 Username Everyone 12288 May 11 16:21 . drwxr-xr-x 4 Username Everyone 12288 May 11 14:23 .. -rw-r--r-- 1 Username Everyone 162 May 11 15:19 filename.doc [root@host bar]# rm filename.doc rm: remove regular file `filename'? y rm: cannot remove `filename': No such file or directory [root@host bar]# echo test >> filename [root@host bar]# ls -la total 28 drwxr-xr-x 2 Username Everyone 12288 May 11 16:21 . drwxr-xr-x 4 Username Everyone 12288 May 11 14:23 .. -rw-r--r-- 1 Username Everyone 167 May 12 09:45 filename.doc [root@host bar]# I.e. I can't delete file, but can read and write it. OK. Get an attributes of file: [root@host bar]# getfattr -dm - -e hex /mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc # file: /mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc trusted.gfid=0x6e904cdada954818a91f5981ee058938 Note: file physically placed on ...part02... volume of DHT Get an attributes of directories: [root@host bar]# getfattr -dm - -e hex /mnt/gluster0*/smb01.part0*/recycled/..../bar # file: /mnt/gluster01/smb01.part01/recycled/..../bar trusted.gfid=0x23d1b62930a345c1af281aae4f55123b trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9 # file: /mnt/gluster02/smb01.part02/recycled/..../bar trusted.gfid=0x23d1b62930a345c1af281aae4f55123b trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff # file: /mnt/gluster03/smb01.part03/recycled/..../bar trusted.gfid=0x23d1b62930a345c1af281aae4f55123b trusted.glusterfs.dht=0x00000001000000000000000055555554 Do a trace of glusterfs process: [root@host bar]# ps ax|grep gluster|grep smb01 30280 ? Ssl 50:20 /opt/glusterfs/3.1.3/sbin/glusterfs --log-level=NORMAL --direct-io-mode=off --volume-name=smb01 --volfile=/etc/glusterfs/smb01.vol /mnt/smb01 [root@host bar]# strace -f -p 30280 2>&1|grep recycled [root@host bar]# rm -f filename.doc lstat("/mnt/gluster01/smb01.part01/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster01/smb01.part01/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 lstat("/mnt/gluster02/smb01.part02/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 lstat("/mnt/gluster03/smb01.part03/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster03/smb01.part03/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 [ skipping lstat of bar/tls, bar/x86_64, bar/libc.so.6 and lgetxattr of bar - total 64 (!!!) lines in logfile are skipped - it's a result of /bin/rm execution? ] lstat("/mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc", {st_mode=S_IFREG|0644, st_size=167, ...}) = 0 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc", "trusted.gfid", "n\x90L\xda\xda\x95H\x18\xa9\x1fY\x81\xee\x05\x898", 16) = 16 lstat("/mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc", {st_mode=S_IFREG|0644, st_size=167, ...}) = 0 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc", "trusted.gfid", "n\x90L\xda\xda\x95H\x18\xa9\x1fY\x81\xee\x05\x898", 16) = 16 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc", "trusted.glusterfs.dht", 0x0, 0) = -1 ENODATA (No data available) lstat("/mnt/gluster02/smb01.part02/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 lstat("/mnt/gluster01/smb01.part01/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster01/smb01.part01/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 unlink("/mnt/gluster01/smb01.part01/recycled/..../bar/filename.doc") = -1 ENOENT (No such file or directory) Ooops. Why if Gluster know via lstat that file placed on part02 mount point do try unlink it on part01? [root@host bar]# cat /var/log/glusterfs/mnt-smb01.log /var/log/glusterfs/mnt-smb01.log.1 |grep UNLINK|sed -e 's|^.*UNLINK() ||'|grep '^/'|cut -d / -f 2|sort|uniq -c 1350 recycled Summary on SAMBA shares placed more then 2M files and as I can see in logs, this situation only occurs with files moved with "recycled" VFS of SAMBA to "recycled" directory. Some workaround: if I send SIGHUP to gluster process, then I can delete this "trouble" files. But some time later another new files (which users are delete via CIFS share and which are placed in "recycled" directory) become undeletable. [root@host bar]# kill -HUP 30280 [root@host bar]# strace -f -p 30280 2>&1|grep recycled [root@host bar]# rm -f filename.doc ... lstat("/mnt/gluster03/smb01.part03/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster03/smb01.part03/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc", "trusted.gfid", "n\x90L\xda\xda\x95H\x18\xa9\x1fY\x81\xee\x05\x898", 16) = 16 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc", "trusted.glusterfs.dht", 0x0, 0) = -1 ENODATA (No data available) lstat("/mnt/gluster02/smb01.part02/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 lstat("/mnt/gluster01/smb01.part01/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster01/smb01.part01/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 lstat("/mnt/gluster02/smb01.part02/recycled/..../bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lgetxattr("/mnt/gluster02/smb01.part02/recycled/..../bar", "trusted.gfid", "#\xd1\xb6)0\xa3E\xc1\xaf(\x1a\xaeOU\x12;", 16) = 16 unlink("/mnt/gluster02/smb01.part02/recycled/..../bar/filename.doc") = 0 File deleted.
Can you check two things - 1. If files which are not getting deleted have 'linkfiles' (on other subvolumes) 2. If they do, whether the GFID of the file and the linkfile are the same or not? You can get the gfid of the file by doing a 'getfattr -e hex -m trusted /backend/path/filename' on the respective bricks. Thanks, Avati
(In reply to comment #1) > Can you check two things - > 1. If files which are not getting deleted have 'linkfiles' (on other > subvolumes) There are no any linkfiles. As I can see (for 3 weeks): a) only unlink() fails: # cat mnt-smb01.log mnt-smb01.log.1 mnt-smb01.log.2|grep '()'|sed -e 's/^.*\s\(.*()\)/\1/' -e 's/().*$//'|sort|uniq -c 2264 UNLINK # b) fails occured only with VFS recycled files: # cat mnt-smb01.log mnt-smb01.log.1 mnt-smb01.log.2|grep '()'|sed -e 's/^.*()\s*//'|cut -d / -f 2|sort|uniq -c 2264 recycled # c) all failed files are readable and writable, but undeletable and I can't rename it d) when fail occurred, I try to look /proc/*/fd/* links - no one of failed files has been opened e) in out of '/usr/sbin/lsof ...' failed files are absent f) all failed files has been successfully deleted after sending SIGHUP to gluster process g) I'm reduce .vol config and at last moment it contains: 3*(Posix+ACL+Locks)+DHT+WB+Threads. All parameters are default. Tomorrow I want try to remove io-threads from config
Please check for the link(with permission -----T) files on the backend for the files on which delete op fails. If they exist can you please provide the gfid for both the files?
This seems to be a duplicate of bug 764196. Unlink fails on subvol part01 as it might be a linkfile with a mismatching gfid, and actual file is residing on part02. *** This bug has been marked as a duplicate of bug 2464 ***