Bug 765511 (GLUSTER-3779)

Summary: [posix.c:723:posix_do_chmod] - (Function not implemented) PHPBB Cache folder on Gluster
Product: [Community] GlusterFS Reporter: Lee Simpson <gluster>
Component: nfsAssignee: Vinayaga Raman <vraman>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 3.2.4CC: gluster-bugs, rwheeler
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-16 12:55:14 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Brick debug logs
none
Volume info none

Description Lee Simpson 2011-11-03 12:47:55 UTC
Created attachment 717

Comment 1 Lee Simpson 2011-11-03 15:47:16 UTC
Hello,

We are having issues with running phpBB on a gluster nfs share. 

Setup is as follows;

- Replication between two servers using gluster
- Web server mounting gluster by NFS
- phpBB website running in apache

The problem is related the to the cache folder within phpBB which is writeable by the webserver. The only way to get phpBB to function correctly is to mount -o bind a local folder onto the cache folder so that phpBB doesnt use gluster for its cache.

When the cache folder in on gluster lots of .nfs files are created and the website stalls.

Highlights of the error logs are;

[2011-11-03 14:17:45.118450] D [posix.c:300:posix_lstat_with_gfid] (-->/usr/lib/glusterfs/3.2.4/xlator/features/access-control.so(posix_acl_readdirp+0x208) [0x7f1e6bf6ad28] (-->/usr/lib/glusterfs/3.2.4/xlator/storage/posix.so(posix_readdirp+0xf) [0x7f1e6c18085f] (-->/usr/lib/glusterfs/3.2.4/xlator/storage/posix.so(posix_do_readdir+0x655) [0x7f1e6c180795]))) 0-www-posix: failed to get gfid
[2011-11-03 14:18:17.645251] D [posix.c:723:posix_do_chmod] 0-www-posix: /mnt/store/www2/www/phpbb/html/cache/sql_9dc91161799801b44926b824ccc5cf51.php (Function not implemented)
[2011-11-03 14:18:17.647059] D [inodelk.c:293:__inode_unlock_lock] 0-www-locks:  Matching lock found for unlock
[2011-11-03 14:18:17.983591] D [server-resolve.c:256:resolve_path_deep] 0-/mnt/store/www2: RESOLVE RENAME() seeking deep resolution of /www/phpbb/html/cache/.nfs40b359053add4b8a00000091
[2011-11-03 14:18:17.983811] D [server-resolve.c:201:resolve_deep_cbk] 0-www-server: /www/phpbb/html/cache/.nfs40b359053add4b8a00000091: failed to resolve (No such file or directory)
[2011-11-03 14:18:17.983825] D [server-resolve.c:170:resolve_deep_continue] 0-www-server: return value of resolve_*_simple 1

Attached are;

- Brick debug log
- volume info

Our setup is;

- Ubuntu 11.10 Server 64bit
- Ext4 drive (tried ext3, no effect)
- Linux 3.0.0-12-virtual #20-Ubuntu
- Gluster 3.2.4 from .debs on your download page.
- PHP 5.3.6
- Apache 2.2.20-1ubuntu1


Let me know what else i can provided to help you track it down.

- Lee

Comment 2 Lee Simpson 2011-11-03 20:59:30 UTC
Just to add that ive also replicated the problem on CentOS 5.7 server using ext3 as a backstore.

Comment 3 Krishna Srinivas 2011-11-09 10:03:27 UTC
Hi Lee,

Are webservers mounting from different NFS servers? If so can you mount them from the same NFS server and see if this problem still happens?

Comment 4 Lee Simpson 2011-11-09 10:11:06 UTC
Hi Krishna,

Thanks for your response.

Yes they are mounting different nfs servers. This was done because we dont want the gluster/nfs server to be the single point of failure.

I will run the test as requested and report back. 

Thanks.

Comment 5 Lee Simpson 2011-11-09 10:43:27 UTC
Hello, 

I shutdown all other webservers, so only one webserver was up and had gluster nfs share mounted, but the problem still exists.

Thanks.

Comment 6 Krishna Srinivas 2011-11-10 03:28:20 UTC
Lee, By chance do any of the servers mount glusterfs using fuse? Or all access to the glusterfs is through NFS?

Comment 7 Lee Simpson 2011-11-10 10:27:04 UTC
There is a FTP server which mounts using glusterfs fuse client.

I have tested the issue while the FTP server is shutdown, but the problem still persists.

Comment 8 Krishna Srinivas 2011-11-15 06:55:59 UTC
Lee, we will need access to the setup to dig further, let us know as much as possible on how to make the bug happen so that we can reproduce it live on your system.

Comment 9 Krishna Srinivas 2011-11-15 08:05:57 UTC
(In reply to comment #8)
> Lee, we will need access to the setup to dig further, let us know as much as
> possible on how to make the bug happen so that we can reproduce it live on your
> system.

Also want to confirm that you are not creating files in the backend directly.

Comment 10 Lee Simpson 2011-11-15 08:13:48 UTC
(In reply to comment #9)
> Also want to confirm that you are not creating files in the backend directly.

We are not creating anything directly, it is being done over a gluster nfs share.


(In reply to comment #8)
> Lee, we will need access to the setup to dig further, let us know as much as
> possible on how to make the bug happen so that we can reproduce it live on your
> system.

Im not sure how we are going to arrange this. Im not sure if i will be able to give you access to the live system. I maybe able to recreate this issue in a development area and provide you access to that, but that will involve a fair bit of work.

Im not sure of the best way to proceed.

Comment 11 Krishna Srinivas 2012-08-14 12:16:07 UTC
Lee, Gluster has undergone changes (backend GFID changes) because of which this bug should not be seen anymore as there is no more hard-resolution of files from NFS. Can you see if you still see this problem? Thanks.

Comment 12 Lee Simpson 2012-08-16 12:55:14 UTC
Confirmed that 3.3 resolves this issue.

Thanks.