Created attachment 717
Hello, We are having issues with running phpBB on a gluster nfs share. Setup is as follows; - Replication between two servers using gluster - Web server mounting gluster by NFS - phpBB website running in apache The problem is related the to the cache folder within phpBB which is writeable by the webserver. The only way to get phpBB to function correctly is to mount -o bind a local folder onto the cache folder so that phpBB doesnt use gluster for its cache. When the cache folder in on gluster lots of .nfs files are created and the website stalls. Highlights of the error logs are; [2011-11-03 14:17:45.118450] D [posix.c:300:posix_lstat_with_gfid] (-->/usr/lib/glusterfs/3.2.4/xlator/features/access-control.so(posix_acl_readdirp+0x208) [0x7f1e6bf6ad28] (-->/usr/lib/glusterfs/3.2.4/xlator/storage/posix.so(posix_readdirp+0xf) [0x7f1e6c18085f] (-->/usr/lib/glusterfs/3.2.4/xlator/storage/posix.so(posix_do_readdir+0x655) [0x7f1e6c180795]))) 0-www-posix: failed to get gfid [2011-11-03 14:18:17.645251] D [posix.c:723:posix_do_chmod] 0-www-posix: /mnt/store/www2/www/phpbb/html/cache/sql_9dc91161799801b44926b824ccc5cf51.php (Function not implemented) [2011-11-03 14:18:17.647059] D [inodelk.c:293:__inode_unlock_lock] 0-www-locks: Matching lock found for unlock [2011-11-03 14:18:17.983591] D [server-resolve.c:256:resolve_path_deep] 0-/mnt/store/www2: RESOLVE RENAME() seeking deep resolution of /www/phpbb/html/cache/.nfs40b359053add4b8a00000091 [2011-11-03 14:18:17.983811] D [server-resolve.c:201:resolve_deep_cbk] 0-www-server: /www/phpbb/html/cache/.nfs40b359053add4b8a00000091: failed to resolve (No such file or directory) [2011-11-03 14:18:17.983825] D [server-resolve.c:170:resolve_deep_continue] 0-www-server: return value of resolve_*_simple 1 Attached are; - Brick debug log - volume info Our setup is; - Ubuntu 11.10 Server 64bit - Ext4 drive (tried ext3, no effect) - Linux 3.0.0-12-virtual #20-Ubuntu - Gluster 3.2.4 from .debs on your download page. - PHP 5.3.6 - Apache 2.2.20-1ubuntu1 Let me know what else i can provided to help you track it down. - Lee
Just to add that ive also replicated the problem on CentOS 5.7 server using ext3 as a backstore.
Hi Lee, Are webservers mounting from different NFS servers? If so can you mount them from the same NFS server and see if this problem still happens?
Hi Krishna, Thanks for your response. Yes they are mounting different nfs servers. This was done because we dont want the gluster/nfs server to be the single point of failure. I will run the test as requested and report back. Thanks.
Hello, I shutdown all other webservers, so only one webserver was up and had gluster nfs share mounted, but the problem still exists. Thanks.
Lee, By chance do any of the servers mount glusterfs using fuse? Or all access to the glusterfs is through NFS?
There is a FTP server which mounts using glusterfs fuse client. I have tested the issue while the FTP server is shutdown, but the problem still persists.
Lee, we will need access to the setup to dig further, let us know as much as possible on how to make the bug happen so that we can reproduce it live on your system.
(In reply to comment #8) > Lee, we will need access to the setup to dig further, let us know as much as > possible on how to make the bug happen so that we can reproduce it live on your > system. Also want to confirm that you are not creating files in the backend directly.
(In reply to comment #9) > Also want to confirm that you are not creating files in the backend directly. We are not creating anything directly, it is being done over a gluster nfs share. (In reply to comment #8) > Lee, we will need access to the setup to dig further, let us know as much as > possible on how to make the bug happen so that we can reproduce it live on your > system. Im not sure how we are going to arrange this. Im not sure if i will be able to give you access to the live system. I maybe able to recreate this issue in a development area and provide you access to that, but that will involve a fair bit of work. Im not sure of the best way to proceed.
Lee, Gluster has undergone changes (backend GFID changes) because of which this bug should not be seen anymore as there is no more hard-resolution of files from NFS. Can you see if you still see this problem? Thanks.
Confirmed that 3.3 resolves this issue. Thanks.