Bug 1317048 - RHGS3: Unable to set more than 21 POSIX ACLs to a file using the Native client
RHGS3: Unable to set more than 21 POSIX ACLs to a file using the Native client
Status: CLOSED NOTABUG
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
3.1
x86_64 Linux
urgent Severity urgent
: ---
: ---
Assigned To: Atin Mukherjee
storage-qa-internal@redhat.com
http://oss.sgi.com/archives/xfs/2007-...
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-11 14:26 EST by Rodrigo A B Freire
Modified: 2016-09-17 12:44 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-11 17:53:20 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2197311 None None None 2016-03-11 15:06 EST

  None (edit)
Description Rodrigo A B Freire 2016-03-11 14:26:29 EST
Description of problem:
setfacl is not setting more than 21 different acl permissions on directories under a GlusterFS volume.

Version-Release number of selected component (if applicable):
glusterfs-server-3.7.5-19.el6rhs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Try to set more than 21 ACLs in a RHGS file mounted by the native client

Actual results:
The setfacl will return the error:
[root@server ~]# setfacl -m u:user:rwx /mnt/sftp/acl_test/
setfacl: /mnt/sftp/acl_test: Invalid argument

Expected results:
The server should accept the ACL.

Additional info:
Reproducer:
# for i in a b c d e f g h i j k l m n o p q r s t u v w x y z ; do echo $i;  useradd $i ; setfacl -m u:"$i":rwx <destination file> ; done
Comment 6 Shyamsundar 2016-03-11 15:40:24 EST
The error in setting the ACL is coming from this message:
[posix-helpers.c:1177:posix_handle_pair] 0-sftp_rep_vol01-posix: <filename>: key:system.posix_acl_accessflags: 0 length:212 [Invalid argument]

This indicates a failure in setxattr syscall, which is returning the error EINVAL here.

Tried a few things on my Fedora box (although not representative of the actual problem), and here are the results.

1) Created ~30 users (with names testuser0$i)
2) Used setfacl -m u:<username>:rwx <dirpath>

- When <dirpath> belonged to an ext4 filesystem this passed
  - getfacl also passed
  - but poking around using getfattr to see how the xattr was stored, resulted in it showing no xattr for this name.
  - I assume ext4 stores this differently
- When <dirpath> belonged to an xfs filesystem
  - The set failed after 20 users were updated.
  - The error thought was "Argument list too long", possibly due to this being Fedora, and hence having more recent packages
  - On examining the xattrs on the XFS directory directly using getfattr, it shows
    - system.posix_acl_access="\000..."
    and,
    - trusted.SGI_ACL_FILE="\000..."

Glusterfs prescribes XFS as its backed, and the error here is presumed to have been on the same filesystem. It is possible that the ACL encoding is exceedng the xattr size limit, and hence we are getting an EINVAL error as stated.

Additionally, it looks like this is not an XFS error, just that we exceed the xattr size. See [1]

Checking the XFS fs data here is what I see,
# xfs_db -r /dev/mapper/fedora_dhcp--41--97-mntd
xfs_db> inode 67110757
xfs_db> print
core.magic = 0x494e
core.mode = 040775
core.version = 2
core.format = 1 (local)
...
a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,4194421,1,0]
xfs_db> ablock 0
xfs_db> p
hdr.info.forw = 0
hdr.info.back = 0
hdr.info.magic = 0xfbee
hdr.count = 4
hdr.usedbytes = 424
hdr.firstused = 3068
...
nvlist[0].valuelen = 16
nvlist[0].namelen = 4
nvlist[0].name = "gfid"
nvlist[0].value = "\3677\302LX3G@\251\320\033\'aQ\242\322"
nvlist[1].valuelen = 304
nvlist[1].namelen = 12
nvlist[1].name = "SGI_ACL_FILE"
nvlist[1].value = "\000\..."
nvlist[2].valuelen = 16
nvlist[2].namelen = 13
nvlist[2].name = "glusterfs.dht"
nvlist[2].value = "\000\000\000\001\000\000\000\000\277\377\377\375\377\377\377\377"
nvlist[3].valuelen = 37
nvlist[3].namelen = 7
nvlist[3].name = "selinux"
nvlist[3].value = "unconfined_u:object_r:unlabeled_t:s0\000"

The above does not seem to be 64K in size, so unsure what limits we are hitting, or if there is a simple setting that can overcome this limitation.

[1] http://oss.sgi.com/archives/xfs/2014-02/msg00024.html
Comment 7 Shyamsundar 2016-03-11 16:09:48 EST
On further discussion with Brian Foster in the XFS team, the issue is that on my Fedora machine, the XFS FS is at v4 superblock version, which supports only 25 ACLs. On counting the actual ACLs the failure is beyond 25 (there are a few default ACLs set already).

We need to following to ensure that the cu is on the same version:
xfs_info <mount> for the mount point being used as bricks.

The output would look as follows,
meta-data=/dev/mapper/fedora_dhcp--41--97-mntd isize=256    agcount=16, agsize=1474560 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=23592960, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=11520, version=2

In the output, if crc=0 then it is v4, if that reads crc=1 then it is v5, which is limited by 64K value for the xattrs, as the ACL limits as well.

NOTE: XFS v5 ships from RHEL7 onwards.

Marking this as NEEDINFO to validate the above.
Comment 8 Rodrigo A B Freire 2016-03-11 16:30:39 EST
http://oss.sgi.com/archives/xfs/2007-12/msg00466.html
Comment 9 Rodrigo A B Freire 2016-03-11 16:33:49 EST
RHEL6 git code:

[rfreire@rf rhel6]$ grep XFS_ACL_MAX_ENTRIES fs/xfs/xfs_acl.h
#define XFS_ACL_MAX_ENTRIES 25
	} acl_entry[XFS_ACL_MAX_ENTRIES];

RHEL7:
[rfreire@rf rhel7]$ grep XFS_ACL_MAX_ENTRIES fs/xfs/xfs_acl.h
[rfreire@rf rhel7]$
Comment 12 Rodrigo A B Freire 2016-03-11 17:53:20 EST
OK, I'm closing it as NOTABUG.

Use RHEL7 and create the filesystem using -m crc=1 and you will be able to use it. Otherwise, it will use the V4 superblock and behave like RHEL6.

Thanks Shyan for your help and patience!

[root@localhost ~]# dd if=/dev/zero of=XFS bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 12.0768 s, 88.9 MB/s
[root@localhost ~]# mkfs.xfs -m crc=1 XFS
meta-data=XFS                    isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@localhost ~]# losetup /dev/loop0 XFS
[root@localhost ~]# mount -o rw,noatime,nouuid,attr2,delaylog,inode64,noquota /dev/loop0 /mnt
[root@localhost ~]# xfs_info /mnt
meta-data=/dev/loop0             isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@localhost ~]# mkdir /mnt/bla
[root@localhost ~]# wc -l /etc/passwd
32
[root@localhost ~]# for i in `cat /etc/passwd | awk -F : '{print $1}' `; do echo $i ; setfacl -m u:"$i":rwx /mnt/bla && echo OK || echo FAIL ;  done
root
OK
bin
OK
daemon
OK
adm
OK
lp
OK
sync
OK
shutdown
OK
halt
OK
mail
OK
operator
OK
games
OK
ftp
OK
nobody
OK
avahi-autoipd
OK
systemd-bus-proxy
OK
systemd-network
OK
dbus
OK
polkitd
OK
tss
OK
postfix
OK
sshd
OK
rpc
OK
memcached
OK
apache
OK
ods
OK
rpcuser
OK
nfsnobody
OK
sssd
OK
ntp
OK
hsqldb
OK
tomcat
OK
pkiuser
OK

Note You need to log in before you can comment on or make changes to this bug.