Hide Forgot
Comments from He Xiaobin <allreol> I have tracked the file renaming problem in my configuration ( I have mentioned in http://xiaoyou.qq.com/index.php?mod=home ). It seems that the problem coming because the file has the wrong position. For example, In my configuration(distribute->client->server->lock->posix), I have two servers, on client. The problem behaved as following: In CLIENT side: [root@client home] ls /mnt/glusterfs/src total 0 In Servre side: [root@server1 home] ls /tank/export/src 2 [root@server2 home] ls /tank/export/src 1 I once thought there should be two files (1 and 2), but as you have seen, it shown nothing. If you create a file named 1, it sould be placed on sever1, and server2 for file 2. As the gluster placed file on severs by the file's name(Though I not quite sure), I think there might too reasons invoking this problem: 1) the hash function works in a wrong way sometimes; 2) the sever disconneted when client was creating files. I can provide more information if it helps. Thanks.
reported by Brent A Nelson <brent.edu> on gluster-devel. http://lists.gnu.org/archive/html/gluster-devel/2009-06/msg00052.html At least with 2.0.2, my GlusterFS filesystems (io-threads=>distribute=>replicate=>ha=>client=>server=>io-threads=>posix-locks=>posix) seem to have a serious problem with file renaming: root@node1:/data# mkdir test root@node1:/data# cd test root@node1:/data/test# cp /bin/ls . root@node1:/data/test# mv ls ls.bak mv: cannot move `ls' to a subdirectory of itself, `ls.bak' root@node1:/data/test# ls -al total 240 drwxr-xr-x 2 root root 32768 2009-06-24 10:45 . drwxr-xr-x 4 root root 32768 2009-06-24 10:45 .. -rwxr-xr-x 1 root root 106176 2009-06-24 10:45 ls root@node1:/data/test# cp ls ls.bak2 root@node1:/data/test# ls -al total 352 drwxr-xr-x 2 root root 32768 2009-06-24 10:46 . drwxr-xr-x 4 root root 32768 2009-06-24 10:45 .. -rwxr-xr-x 1 root root 106176 2009-06-24 10:45 ls -rwxr-xr-x 1 root root 106176 2009-06-24 10:46 ls.bak2 This leaves residue on the underlying shares (eonly odd nodes are listed, as the even ones are replicas and show the same contents): root@node1:/data/test# ls -la /disk?/glusterfs/test /disk3/glusterfs/test: total 128 drwxr-xr-x 2 root root 4096 2009-06-24 10:46 . drwxr-xr-x 4 root root 4096 2009-06-24 10:45 .. -rwxr-xr-x 1 root root 106176 2009-06-24 10:46 ls.bak2 /disk4/glusterfs/test: total 16 drwxr-xr-x 2 root root 4096 2009-06-24 10:45 . drwxr-xr-x 4 root root 4096 2009-06-24 10:45 .. root@node3:~# ls -la /disk?/glusterfs/test /disk3/glusterfs/test: total 20 drwxr-xr-x 2 root root 4096 2009-06-24 10:45 . drwxr-xr-x 4 root root 4096 2009-06-24 10:45 .. ---------T 1 root root 0 2009-06-24 10:45 ls /disk4/glusterfs/test: total 16 drwxr-xr-x 2 root root 4096 2009-06-24 10:45 . drwxr-xr-x 4 root root 4096 2009-06-24 10:45 .. root@node5:~# ls -la /disk?/glusterfs/test /disk3/glusterfs/test: total 16 drwxr-xr-x 2 root root 4096 2009-06-24 10:45 . drwxr-xr-x 4 root root 4096 2009-06-24 10:45 .. /disk4/glusterfs/test: total 128 drwxr-xr-x 2 root root 4096 2009-06-24 10:45 . drwxr-xr-x 4 root root 4096 2009-06-24 10:45 .. -rwxr-xr-x 1 root root 106176 2009-06-24 10:45 ls root@node7:~# ls -la /disk?/glusterfs/test /disk3/glusterfs/test: total 16 drwxr-xr-x 2 root root 4096 2009-06-24 10:45 . drwxr-xr-x 4 root root 4096 2009-06-24 10:45 .. /disk4/glusterfs/test: total 16 drwxr-xr-x 2 root root 4096 2009-06-24 10:45 . drwxr-xr-x 4 root root 4096 2009-06-24 10:45 .. The file I tried to rename left an extra, empty version of the file with a+T permissions, in addition to the file itself. When I end up with these a+T files, GlusterFS does not show them, but operations can indicate their presence (e.g., you can't delete a directory containing these files from GlusterFS, as GlusterFS won't delete the file, and then the directory is not empty and can't be unlinked). I know I have seen these a+T files in 2.0.1 and earlier, as well (I believe I mentioned them when I was doing NFS testing).
The extra copy of the file with the strange mode is a distribute-translator link file, sort of like a symbolic link, used to facilitate renaming without actually moving the data. It's created on the file's new home subvolume (according to a hash of the new filename) with an extended attribute identifying its original home subvolume where it actually still lives. You can examine this attribute with the following command to verify. getfattr -d -e text -m trusted.glusterfs.dht.linkto filename Link files should only get created for renames, and should be invisible to users, but (as you pointed out) sometimes they're not quite invisible enough. The real issue seems to be that the rename attempt left a link file even though it failed, which probably shouldn't be the case.
This bug was fixed long back (before migration to git. hence cannot provide commmit id url) Avati