Description of problem: set up is like I am logged into a client with a non-root user. on a nfs mount I have a directory for this user. If I try to rename a file it fails. This is behaviour is seen when root squash is on. Volume Name: dr1 Type: Distributed-Replicate Volume ID: 40922fbc-a3a4-4760-a17e-73ce4cc4423f Status: Started Number of Bricks: 6 x 2 = 12 Transport-type: tcp Bricks: Brick1: rhsauto032.lab.eng.blr.redhat.com:/rhs/bricks/d1r1 Brick2: rhsauto033.lab.eng.blr.redhat.com:/rhs/bricks/d1r2 Brick3: rhsauto034.lab.eng.blr.redhat.com:/rhs/bricks/d2r1 Brick4: rhsauto035.lab.eng.blr.redhat.com:/rhs/bricks/d2r2 Brick5: rhsauto032.lab.eng.blr.redhat.com:/rhs/bricks/d3r1 Brick6: rhsauto033.lab.eng.blr.redhat.com:/rhs/bricks/d3r2 Brick7: rhsauto034.lab.eng.blr.redhat.com:/rhs/bricks/d4r1 Brick8: rhsauto035.lab.eng.blr.redhat.com:/rhs/bricks/d4r2 Brick9: rhsauto032.lab.eng.blr.redhat.com:/rhs/bricks/d5r1 Brick10: rhsauto033.lab.eng.blr.redhat.com:/rhs/bricks/d5r2 Brick11: rhsauto034.lab.eng.blr.redhat.com:/rhs/bricks/d6r1 Brick12: rhsauto035.lab.eng.blr.redhat.com:/rhs/bricks/d6r2 Options Reconfigured: server.root-squash: on features.alert-time: 1s features.quota-deem-statfs: on features.default-soft-limit: 50% features.quota: on Version-Release number of selected component (if applicable): glusterfs-3.4.0.31rhs-1.el6rhs.x86_64 How reproducible: always Steps to Reproduce: 1. create a volume, start it 2. mount it over nfs 3. create a dir. 4. chown the dir permission for non-root user 4a. enable root squash 5. login to system client as the non-root user 6. create a file 7. rename the file. Actual results: [saurabh@rhsauto036 saurabh-dir]$ touch f.n1 [saurabh@rhsauto036 saurabh-dir]$ mv f.n1 f.n1-rename mv: cannot move `f.n1' to `f.n1-rename': Permission denied Expected results: the rename should happen, as with root-squash off it does work Additional info: nfs.log reports this, [2013-09-06 08:52:40.024472] I [glusterfsd-mgmt.c:1559:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2013-09-06 08:52:40.028243] I [glusterfsd-mgmt.c:1559:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2013-09-06 08:52:40.029992] I [glusterfsd-mgmt.c:1559:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2013-09-06 08:52:52.056620] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-dr1-client-2: remote operation failed: Permission denied. Path: /saurabh-dir/f.n1 [2013-09-06 08:52:52.056902] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-dr1-client-3: remote operation failed: Permission denied. Path: /saurabh-dir/f.n1 [2013-09-06 08:52:52.057343] W [client-rpc-fops.c:2453:client3_3_link_cbk] 0-dr1-client-10: remote operation failed: Permission denied (/saurabh-dir/f.n1 -> /saurabh-dir/f.n1-rename) [2013-09-06 08:52:52.057477] W [client-rpc-fops.c:2453:client3_3_link_cbk] 0-dr1-client-11: remote operation failed: Permission denied (/saurabh-dir/f.n1 -> /saurabh-dir/f.n1-rename) [2013-09-06 08:52:52.062617] W [dht-rename.c:365:dht_rename_unlink_cbk] 0-dr1-dht: /saurabh-dir/f.n1: unlink on dr1-replicate-1 failed (No such file or directory) [2013-09-06 08:52:52.063807] W [dht-rename.c:365:dht_rename_unlink_cbk] 0-dr1-dht: /saurabh-dir/f.n1: unlink on dr1-replicate-5 failed (No such file or directory) [2013-09-06 08:52:52.064049] W [nfs3.c:3663:nfs3svc_rename_cbk] 0-nfs: 9236d348: rename /saurabh-dir/f.n1 -> /saurabh-dir/f.n1-rename => -1 (Permission denied)
I just checked the behavior, the rename is not failing all the time. It is failing only when the rename operation will cause the file to move from one brick to another due to DHT. Actually the file is not moved physically but a link file is created in the target brick. As per my discussion with Shishir the link file is created with root permission. If this link file creation is subjected to root-squashing then the rename will fail with "permission denied".
Similar issue [qa1@rhsauto015 ~]$ mv /mnt/nfs-test/qa1-user-dir/f.1 /mnt/nfs-test/qa1-user-dir/dir1/f.1-new mv: cannot move `/mnt/nfs-test/qa1-user-dir/f.1' to `/mnt/nfs-test/qa1-user-dir/dir1/f.1-new': Permission denied [qa1@rhsauto015 ~]$ Happens on glusterfs-3.6.0.33-1.el6rhs.x86_64