Description of problem: gluster fails to propagate permissions on the root of a gluster export when adding bricks. This causes unexpected behavior on the client side. Version-Release number of selected component (if applicable): How reproducible: Every time Steps to Reproduce: ++server config (three nodes with same config): [root@rhsa3 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda1 202:1 0 100G 0 disk / xvdb 202:16 0 150G 0 disk └─xvdb1 202:17 0 150G 0 part ├─vg_rhs-rhslvdemo (dm-0) 253:0 0 1G 0 lvm /storage ├─vg_rhs-brick1 (dm-1) 253:1 0 2G 0 lvm /bricks/1 ├─vg_rhs-brick2 (dm-2) 253:2 0 2G 0 lvm /bricks/2 ├─vg_rhs-brick3 (dm-3) 253:3 0 2G 0 lvm /bricks/3 ├─vg_rhs-brick4 (dm-4) 253:4 0 2G 0 lvm /bricks/4 ├─vg_rhs-brick5 (dm-5) 253:5 0 2G 0 lvm /bricks/5 ├─vg_rhs-brick6 (dm-6) 253:6 0 2G 0 lvm /bricks/6 ├─vg_rhs-brick7 (dm-7) 253:7 0 2G 0 lvm /bricks/7 └─vg_rhs-brick8 (dm-8) 253:8 0 2G 0 lvm /bricks/8 [root@rhsa3 ~]# xfs_info /bricks/5 <snip> [root@rhsa3 ~]# gluster volume create vol5 rhsa{1,2}:/bricks/5/brick volume create: vol5: success: please start the volume to access data [root@rhsa3 ~]# gluster volume start vol5 volume start: vol5: success ++On client: [student@rhsac vol2]$ sudo mkdir -p /mnt/native/vol5 [student@rhsac vol2]$ ls -ld /mnt/native/vol5 drwxr-xr-x. 2 root root 4096 Mar 21 12:52 /mnt/native/vol5 [student@rhsac vol2]$ sudo mount /mnt/native/vol5 [student@rhsac vol2]$ mount | grep vol5 rhsa1:/vol5 on /mnt/native/vol5 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) [student@rhsac vol2]$ sudo chmod a+rwx,+t /mnt/native/vol5 [student@rhsac vol2]$ ls -ld /mnt/native/vol5 drwxrwxrwt. 3 root root 46 Mar 21 12:49 /mnt/native/vol5 [student@rhsac vol2]$ echo 'hello world' > /mnt/native/vol5/x [student@rhsac vol2]$ ls -l /mnt/native/vol5 total 1 -rw-rw-r--. 1 student student 12 Mar 21 12:55 x ++On server: [root@rhsa3 ~]# ls -ld /bricks/5/brick drwxr-xr-x 2 root root 6 Mar 19 23:52 /bricks/5/brick [root@rhsa3 ~]# gluster volume add-brick vol5 rhsa3:/bricks/5/brick volume add-brick: success ++On client: [student@rhsac vol5]$ for x in `seq 10 22`; do dd if=/dev/zero of=./file_r$x bs=2K count=1; done 1+0 records in 1+0 records out 2048 bytes (2.0 kB) copied, 0.00178166 s, 1.1 MB/s 1+0 records in 1+0 records out 2048 bytes (2.0 kB) copied, 0.00129699 s, 1.6 MB/s dd: opening `./file_r12': Permission denied dd: opening `./file_r13': Permission denied dd: opening `./file_r14': Permission denied dd: opening `./file_r15': Permission denied dd: opening `./file_r16': Permission denied dd: opening `./file_r17': Permission denied dd: opening `./file_r18': Permission denied dd: opening `./file_r19': Permission denied dd: opening `./file_r20': Permission denied dd: opening `./file_r21': Permission denied dd: opening `./file_r22': Permission denied ++/brick/5/bricks on each server: rhsa1: drwxrwxrwt 3 root root 61 Mar 21 17:00 . rhsa2: drwxrwxrwt 3 root root 85 Mar 21 17:00 . rhsa3: drwxr-xr-x 3 root root 23 Mar 21 16:58 . ++gluster volume rebalance vol5 fix-layout start - no change in dir perms ++gluster volume rebalance vol5 start - no change in perms, files rebalanced with failures [root@rhsa3 ~]# gluster volume rebalance vol5 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 9 0 0 completed 0.00 rhsa1 0 0Bytes 0 1 0 failed 0.00 rhsa2 0 0Bytes 0 1 0 failed 0.00 volume rebalance: vol5: success: ++On client: Notice inconsistancy in affected directory -- [student@rhsac vol5]$ ls -ld . drwxr-xr-x. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxrwxrwt. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxrwxrwt. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxrwxrwt. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxrwxrwt. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxrwxrwt. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxrwxrwt. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxrwxrwt. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxrwxrwt. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxr-xr-x. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ls -ld . drwxr-xr-x. 3 root root 200 Mar 21 13:05 . [student@rhsac vol5]$ ++workaround: replace permissions on client/force rewrite [student@rhsac vol5]$ sudo chmod a+rwx,+t /mnt/native/vol5 [root@rhsa3 ~]# gluster volume rebalance vol5 start force volume rebalance: vol5: success: Starting rebalance on volume vol5 has been successful. ID: 12ae2478-7674-4b73-9d0e-ebdf6fbc03dd [root@rhsa3 ~]# gluster volume rebalance vol5 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 9 0 0 completed 0.00 rhsa1 0 0Bytes 10 0 0 completed 0.00 rhsa2 3 2.0KB 12 0 0 completed 0.00 volume rebalance: vol5: success:
Dev ack to 3.0 RHS BZs
*** Bug 1003966 has been marked as a duplicate of this bug. ***
Cloning this to 3.1.