Description of problem: add-brick command on volume with ACL different than root:root seriously breaks permissions. Version-Release number of selected component (if applicable): 3.4.0 ~ 3.5 also probably on master How reproducible: always Steps to Reproduce: 1. gluster volume create test pufo:/mnt/gluster/test1 pufo:/mnt/gluster/test2 force 2. gluster volume start test 3. mount -t glusterfs pufo:test /media/ 4. chown apache:apache /media 5. ls -lha /mnt/gluster/ total 0 drwxr-xr-x. 1 root root 20 Feb 11 14:38 . drwxr-xr-x. 1 root root 30 Feb 11 14:37 .. drwxr-xr-x. 1 apache apache 20 Feb 11 14:39 test1 drwxr-xr-x. 1 apache apache 20 Feb 11 14:39 test2 6. ls -lha /media/ total 0 drwxr-xr-x. 1 apache apache 120 Feb 11 14:40 . drwxr-xr-x. 1 root root 172 Feb 11 12:14 .. 7. gluster volume add-brick test pufo:/mnt/gluster/test3 pufo:/mnt/gluster/test4 force 8. ls -lha /mnt/gluster/ total 0 drwxr-xr-x. 1 root root 40 Feb 11 14:40 . drwxr-xr-x. 1 root root 30 Feb 11 14:37 .. drwxr-xr-x. 1 apache apache 20 Feb 11 14:39 test1 drwxr-xr-x. 1 apache apache 20 Feb 11 14:39 test2 drwxr-xr-x. 1 root root 20 Feb 11 14:40 test3 drwxr-xr-x. 1 root root 20 Feb 11 14:40 test4 9. ls -lha /media/ total 0 drwxr-xr-x. 1 apache apache 120 Feb 11 14:40 . drwxr-xr-x. 1 root root 172 Feb 11 12:14 .. Actual results: As you can see on first sight one would think permissions are OK as fuse mount caches them but the moment user apache tries to do something it'll hit permission denied caused by new bricks lacking the ACLs. Expected results: All bricks should have same ACL and it should stay unchanged by add-brick. Additional info:
Also present on master.
One can get it to somewhat work with gluster volume set test storage.owner-uid 48 but this is far from right. First of all the change should not require running gluster volume set command and second of all where are the acls?
so brick options has to be exended by posix xattrs and acls and setattr option should check for root or volume changes and update volume options? Question is what that would do with ~50 bricks trying to update volume op, will it slow down considerably? What about conflicts?
I don't think adding options is correct solution here: * options would need to be updated from xlators * options would have to be extended for xattrs which can grow quite much * options would have to be synced Gluster uses acl of underlying fs as an acl backend which means when brick is being added there should be start daemon > create path > join in some prepairing state > selfheal gfid 1 to copy permissions attributes and so on > switch to fully operating brick and now rebalance is optional but the cornercase of bricks root should not be solved by adding volume options :/
No the final conclusion is that this definetly is work for dht/afr self heal. There shouldn't be any brick-uid brick-gid option and permissions,acls/xattrs should be synced by self heal.
REVIEW: http://review.gluster.org/7178 (posix,glusterd: Deprecate brick-uid and brick-gid options) posted (#1) for review on master by Lukáš Bezdička (lukas.bezdicka)
REVIEW: http://review.gluster.org/7178 (posix,glusterd: Deprecate brick-uid and brick-gid options) posted (#2) for review on master by Lukáš Bezdička (lukas.bezdicka)
REVIEW: http://review.gluster.org/7178 (posix,glusterd: Deprecate brick-uid and brick-gid options) posted (#3) for review on master by Lukáš Bezdička (lukas.bezdicka)
Ok patched works better but still is quite broken, I'll add some tests to continue with.
REVIEW: http://review.gluster.org/7178 (posix,glusterd: Deprecate brick-uid and brick-gid options) posted (#4) for review on master by Lukáš Bezdička (lukas.bezdicka)
Here file oink was created while one brick was down and it got healed correctly [2014-03-04 13:44:58.885649] T [fuse-bridge.c:555:fuse_lookup_resume] 0-glusterfs-fuse: 78: LOOKUP /oink(aceae17a-e22c-4974-a16c-253f44840c3f) [2014-03-04 13:44:58.885956] T [dht-hashfn.c:97:dht_hash_compute] 1-patchy-dht: trying regex for oink [2014-03-04 13:44:58.886000] T [dht-common.c:1478:dht_lookup] 1-patchy-dht: incomplete layout failure for path=/oink [2014-03-04 13:44:58.889005] D [dht-common.c:448:dht_lookup_dir_cbk] 1-patchy-dht: lookup of /oink on patchy-client-2 returned error (No such file or directory) [2014-03-04 13:44:58.889143] D [dht-layout.c:670:dht_layout_normalize] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x90) [0x7f0861e090f0] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_3_lookup_cbk+0x6eb) [0x7f085b51d5bb] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_lookup_dir_cbk+0x474) [0x7f085b2c5cc4]))) 1-patchy-dht: path=/oink err=No such file or directory on subvol=patchy-client-2 [2014-03-04 13:44:58.889167] D [dht-common.c:496:dht_lookup_dir_cbk] 1-patchy-dht: fixing assignment on /oink [2014-03-04 13:44:58.889203] D [dht-selfheal.c:528:dht_selfheal_dir_mkdir] 1-patchy-dht: creating directory /oink on subvol patchy-client-2 [2014-03-04 13:44:58.890905] T [dht-selfheal.c:374:dht_selfheal_dir_setattr] 1-patchy-dht: setattr for /oink on subvol patchy-client-2 [2014-03-04 13:44:58.892109] T [dht-selfheal.c:288:dht_selfheal_dir_xattr] 1-patchy-dht: 1 subvolumes missing xattr for /oink [2014-03-04 13:44:58.892149] T [dht-selfheal.c:172:dht_selfheal_dir_xattr_persubvol] 1-patchy-dht: setting hash range 0 - 0 (type 0) on subvolume patchy-client-2 for /oink
REVIEW: http://review.gluster.org/7178 (posix,glusterd: Deprecate brick-uid and brick-gid options) posted (#5) for review on master by Lukáš Bezdička (lukas.bezdicka)
REVIEW: http://review.gluster.org/7178 (posix,glusterd: Deprecate brick-uid and brick-gid options) posted (#6) for review on master by Lukáš Bezdička (lukas.bezdicka)
REVIEW: http://review.gluster.org/7223 (cluster/dht: Make sure we also check directory xattrs for healing) posted (#1) for review on master by Lukáš Bezdička (lukas.bezdicka)
REVIEW: http://review.gluster.org/7223 (cluster/dht: Make sure we also check directory xattrs for healing) posted (#2) for review on master by Lukáš Bezdička (lukas.bezdicka)
REVIEW: http://review.gluster.org/7178 (posix,glusterd: Deprecate brick-uid and brick-gid options) posted (#7) for review on master by Lukáš Bezdička (lukas.bezdicka)
REVIEW: http://review.gluster.org/7223 (cluster/dht: Make sure we also check directory xattrs for healing) posted (#3) for review on master by Lukáš Bezdička (lukas.bezdicka)
GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5. This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "bugs". If there is no response by the end of the month, this bug will get automatically closed.
GlusterFS 3.4.x has reached end-of-life. If this bug still exists in a later release please reopen this and change the version or open a new bug.
GlusterFS 3.4.x has reached end-of-life.\ \ If this bug still exists in a later release please reopen this and change the version or open a new bug.