Hide Forgot
We have 30 node mirror distribute glusterfs of 30TB capacity. The hosts are GNU/Debian 5.0 2.6.26 x86_64 running gluster 2.0.7. Clients are identical specs. Yeasterday I added a GNU/Debian 5.0 2.6.26 x86 (32-bit!) client running 2.0.7. It can fuse mount glusterfs and everything works as long as given inode can be reprecented by a 32-bit int. It fails when inodes are bigger than 32-bit: When I stat a directory called 'UNTITLED' I get: `stat UNTITLED` mascot:/glusterfs# stat UNTITLED File: `UNTITLED' Size: 61440 Blocks: 128 IO Block: 4096 directory Device: 12h/18d Inode: 249246592 Links: 6 Access: (0777/drwxrwxrwx) Uid: ( 1000/diradmin) Gid: ( 0/ root) Access: 2010-01-11 14:57:27.000000000 -0600 Modify: 2009-10-22 12:31:25.000000000 -0500 Change: 2010-01-09 06:24:46.000000000 -0600 but debug/trace modules outputs the following: [2010-01-12 10:09:02] N [trace.c:1245:trace_lookup] trace: 586: (loc {path=/, ino=1}) [2010-01-12 10:09:02] N [trace.c:513:trace_lookup_cbk] trace: 586: (op_ret=0, ino=1, *buf {st_dev=64769, st_ino=1, st_mode=42777, st_nlink=18, st_uid=1000, st_gid=1000, st_rdev=0, st_size=61440, st_blksize=4096, st_blocks=240}) [2010-01-12 10:09:02] N [trace.c:1245:trace_lookup] trace: 587: (loc {path=/UNTITLED, ino=68968723328}) [2010-01-12 10:09:02] N [trace.c:513:trace_lookup_cbk] trace: 587: (op_ret=0, ino=68968723328, *buf {st_dev=64769, st_ino=68968723328, st_mode=40777, st_nlink=6, st_uid=1000, st_gid=0, st_rdev=0, st_size=61440, st_blksize=4096, st_blocks=128}) So via fuse inode is 249246592 while glusterfs sees 68968723328... which are in hex are: edb3380 and 100edb3380. Is there any solution/workaround for our problem? I did try updating glusterfs-client on 32-bit node to 2.0.9 but it behaved the same. Please advise, Sincerely, Michal "Saahbs" Sabala (sys-admin)
Once I enter UNTITLED directory, following errors are reported: mascot:/glusterfs# cd UNTITLED/ mascot:/glusterfs/UNTITLED# ls ls: cannot open directory .: No such file or directory mascot:/glusterfs/UNTITLED# stat . stat: cannot stat `.': No such file or directory I do want to state that I'm very happy with GlusterFS overall and it is working very well on our 64-bit clients. Thank You! m.
(In reply to comment #1) > mascot:/glusterfs# stat UNTITLED The stat(1) utility is not completely reliable. See eg.: http://thread.gmane.org/gmane.comp.file-systems.fuse.devel/8042/focus=8051 I know that was a different thing, and most likely stat is not lying here, but just to make sure, could you see what strace -v -e stat,lstat,stat64,lstat64 ls -ld UNTITLED gives? Csaba
(In reply to comment #3) > (In reply to comment #1) > > mascot:/glusterfs# stat UNTITLED > > The stat(1) utility is not completely reliable. See eg.: > > http://thread.gmane.org/gmane.comp.file-systems.fuse.devel/8042/focus=8051 > > I know that was a different thing, and most likely stat is not lying here, but > just to make sure, could you see what > > strace -v -e stat,lstat,stat64,lstat64 ls -ld UNTITLED Thanks for the suggestion Csaba, following is the strace output: lstat64("UNTITLED", {st_dev=makedev(0, 18), st_ino=249246592, st_mode=S_IFDIR|0777, st_nlink=6, st_uid=1000, st_gid=0, st_blksize=4096, st_blocks=128, st_size=61440, st_atime=2010/01/11-14:57:27, st_mtime=2009/10/22-12:31:25, st_ctime=2010/01/09-06:24:46}) = 0 :/
FYI, 32-bit glusterfs was compiled on a system with libfuse-dev 2.7.4-1.1. 64-bit glusterfs was also compiled against libfuse-dev 2.7.4-1.1. Bricks are exporting an xfs filesystem btw. thanks, M.
I take this, as it seems to be at least partially a kernel issue.
Michal, (In reply to comment #1) > So via fuse inode is 249246592 while glusterfs sees 68968723328... which are in > hex are: edb3380 and 100edb3380. > > Is there any solution/workaround for our problem? > > I did try updating glusterfs-client on 32-bit node to 2.0.9 but it behaved the > same. So the situation that 64 bit inodes are sent to a 32 bit client triggers two distinct problems. Inode number truncation is a kernel bug for which I just sent a patch to fuse maintainer Miklos Szeredi (you were Cc-d). The more serious issue, that the directory vanishes when you enter it, is a glusterfs bug. It's fixed in glusterfs 3.x series, but there is no easy fix. You will have to upgrade to 3.x to get it sorted out. That means 3.x everywhere, on clients and also bricks, because 2.x is not compatible with 3.x. Csaba