Description of problem: We expect that if we specify average file size for arbiter volume, then arbiter brick should be able to create amount of files equal to "(volume_size / avg_file_size)". But, it is not so de-facto. Underlying volumes have expected size, but resulting size of XFS filesystem is significantly less than expected. Version-Release number of selected component (if applicable): Heketi Image: rhgs3/rhgs-volmanager-rhel7:v3.10 Heketi Image ID: docker-pullable://brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-volmanager-rhel7@sha256:51fabf34dfd0dd795fb47b9923a67d32cf866e4a1b8ed8482b0f83dfd9aa1fa8 $ oc version oc v3.10.14 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://vp-ansible-v310-14-2-master-0:8443 openshift v3.10.14 kubernetes v1.10.0+b81c8f8 sh-4.2# gluster --version glusterfs 3.8.4 built on Jul 12 2018 12:36:38 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. How reproducible: 100% Steps to Reproduce: 1. Create arbiter volume of 13Gb size specifying avg file size as 500kb: $ heketi-cli volume create --name=500kb-avg --size=13 --gluster-volume-options='user.heketi.arbiter true,user.heketi.average-file-size 500' 2. Go to the arbiter brick 3. Create as many files as you can Actual results: Only 14373 files was possible to create. Expected results: 27262 files are expected to be created. Comments: real available minimum space we can get for arbiter brick is 9.3Mb instead of expected about 16MB. It is true for arbiter volumes with 900kb+ avg file size. Additional info: $ heketi-cli volume list Id:9e56f7819a4d6a524cca496d269440ec Cluster:ed3af51a8e17862a2ba7b886d4f9ad9e Name:heketidbstorage $ heketi-cli volume create --name=500kb-avg --size=13 --gluster-volume-options='user.heketi.arbiter true,user.heketi.average-file-size 500' Name: 500kb-avg Size: 13 Volume Id: 92931fc40faad24a65e86f1b46ee5e2f Cluster Id: ed3af51a8e17862a2ba7b886d4f9ad9e Mount: 10.70.46.6:500kb-avg Mount Options: backup-volfile-servers=10.70.47.64,10.70.47.164 Block: false Free Size: 0 Reserved Size: 0 Block Volumes: [] Durability Type: replicate Distributed+Replica: 3 $ heketi-cli volume list Id:92931fc40faad24a65e86f1b46ee5e2f Cluster:ed3af51a8e17862a2ba7b886d4f9ad9e Name:500kb-avg Id:9e56f7819a4d6a524cca496d269440ec Cluster:ed3af51a8e17862a2ba7b886d4f9ad9e Name:heketidbstorage $ oc get pods NAME READY STATUS RESTARTS AGE glusterblock-cns-provisioner-dc-1-9wpzc 1/1 Running 0 13h glusterfs-cns-9fzvj 1/1 Running 0 13h glusterfs-cns-glz7g 1/1 Running 0 13h glusterfs-cns-pblkc 1/1 Running 0 13h heketi-cns-1-sqhh6 1/1 Running 0 13h $ oc rsh glusterfs-cns-9fzvj sh-4.2# gluster v info Volume Name: 500kb-avg Type: Replicate Volume ID: 450a54ee-2325-4228-89d2-b0124a12a30e Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: 10.70.47.64:/var/lib/heketi/mounts/vg_e503be91d5ab04a216fc7f2083233c25/brick_cbda8ce01f3d356d51a9ef57b1e099eb/brick Brick2: 10.70.47.164:/var/lib/heketi/mounts/vg_ac4f3b15e121b39e15a210d22e3b409c/brick_29972828bec6b8a6b7f917010cba01c0/brick Brick3: 10.70.46.6:/var/lib/heketi/mounts/vg_4b472ca392527038bbceaf6cb5c6c134/brick_c6ce97aa651332654482d01979e19e0c/brick (arbiter) Options Reconfigured: user.heketi.average-file-size: 500 user.heketi.arbiter: true transport.address-family: inet nfs.disable: on performance.client-io-threads: off cluster.brick-multiplex: on Volume Name: heketidbstorage Type: Replicate Volume ID: 993af0bc-24ae-4e4c-81c6-95bf6e9567c1 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.47.164:/var/lib/heketi/mounts/vg_f19d67fa640cc26e007b1fd1766aeff9/brick_ca13d9d9e7b122e39234772483e7ae3f/brick Brick2: 10.70.46.6:/var/lib/heketi/mounts/vg_ecf35005c0da09ab36325168e793334d/brick_fd370dd25f4f1a2c93aaea2d27f0d6eb/brick Brick3: 10.70.47.64:/var/lib/heketi/mounts/vg_d922736ca831714ed37240c05385c14d/brick_3ee556751d5dfcff2b526a1bd7bd21f9/brick Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off cluster.brick-multiplex: on sh-4.2# df -Th Filesystem Type Size Used Avail Use% Mounted on overlay overlay 40G 1.8G 39G 5% / /dev/sdc xfs 40G 42M 40G 1% /run devtmpfs devtmpfs 16G 0 16G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm /dev/mapper/docker--vol-dockerlv xfs 40G 1.8G 39G 5% /run/secrets /dev/mapper/rhel_dhcp46--210-root xfs 35G 2.4G 33G 7% /etc/ssl tmpfs tmpfs 16G 5.1M 16G 1% /run/lvm tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup tmpfs tmpfs 16G 16K 16G 1% /run/secrets/kubernetes.io/serviceaccount /dev/mapper/vg_ecf35005c0da09ab36325168e793334d-brick_fd370dd25f4f1a2c93aaea2d27f0d6eb xfs 2.0G 33M 2.0G 2% /var/lib/heketi/mounts/vg_ecf35005c0da09ab36325168e793334d/brick_fd370dd25f4f1a2c93aaea2d27f0d6eb /dev/mapper/vg_4b472ca392527038bbceaf6cb5c6c134-brick_c6ce97aa651332654482d01979e19e0c xfs 22M 1.6M 20M 8% /var/lib/heketi/mounts/vg_4b472ca392527038bbceaf6cb5c6c134/brick_c6ce97aa651332654482d01979e19e0c sh-4.2# cd /var/lib/heketi/mounts/vg_4b472ca392527038bbceaf6cb5c6c134/brick_c6ce97aa651332654482d01979e19e0c sh-4.2# touch file{1..14374} touch: cannot touch 'file14374': No space left on device sh-4.2# ls -l | grep -c file 14373 sh-4.2# pwd /var/lib/heketi/mounts/vg_4b472ca392527038bbceaf6cb5c6c134/brick_c6ce97aa651332654482d01979e19e0c sh-4.2# df -Th /var/lib/heketi/mounts/vg_4b472ca392527038bbceaf6cb5c6c134/brick_c6ce97aa651332654482d01979e19e0c Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_4b472ca392527038bbceaf6cb5c6c134-brick_c6ce97aa651332654482d01979e19e0c xfs 22M 9.1M 13M 43% /var/lib/heketi/mounts/vg_4b472ca392527038bbceaf6cb5c6c134/brick_c6ce97aa651332654482d01979e19e0c sh-4.2# lvdisplay /dev/mapper/vg_4b472ca392527038bbceaf6cb5c6c134-brick_c6ce97aa651332654482d01979e19e0c --- Logical volume --- LV Path /dev/vg_4b472ca392527038bbceaf6cb5c6c134/brick_c6ce97aa651332654482d01979e19e0c LV Name brick_c6ce97aa651332654482d01979e19e0c VG Name vg_4b472ca392527038bbceaf6cb5c6c134 LV UUID T0H0N0-fd1y-u5Df-uUSG-62Th-m1CY-kTW2vj LV Write Access read/write LV Creation host, time vp-ansible-v310-14-2-app-cns-2, 2018-08-03 11:14:17 +0000 LV Pool name tp_c6ce97aa651332654482d01979e19e0c LV Status available # open 1 LV Size 28.00 MiB Mapped size 54.46% Current LE 7 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:12
I am working on this.
Analysis: when a 13G volume with average size of 500 KB is created: we get an arbiter calculation of: bricksize: 13631488, filesize: 500, arbiterbricksize: 27262 mkfs command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_8b7da1a300b44823878dfab9c4112224-brick_bbc42f67c168b1b8d1e6f9b176088491 Hence, with each file taking up 1K in inode block, we should be able to create 27262 files. However, the stat info is File: ‘/var/lib/heketi/mounts/vg_8b7da1a300b44823878dfab9c4112224/brick_bbc42f67c168b1b8d1e6f9b176088491’ Size: 19 Blocks: 0 IO Block: 4096 directory Device: fc04h/64516d Inode: 14336 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:unlabeled_t:s0 Access: 1970-01-01 05:30:00.000000000 +0530 Modify: 2018-08-08 23:03:07.302437334 +0530 Change: 2018-08-08 23:03:07.302437334 +0530 Birth: - We can see only 14336 inodes available. Formatting the space partition with xfs inode option of maxpct=100 gives us about 38K inodes. 38K being greater than 27262 satisfies our requirement. We get more inodes than the calculation in heketi because in heketi we also accounted for blocks needed by those 27262 files' extended attributes. To test the same, I create a arbiter volume and created a file. Then I copied over the file to xfs partition with maxpct=100, 27262 times. This test worked. Other tests performed: I added a extended attribute to all files using setfattr -n user.rtalur1 -v a10byteval I added another set of ea to all files using setfattr -n user.rtalur2 -v a10byteval At this point, we still have about 75% of the partition free. I think this is a satisfactory test. Hence, I will create a patch which changes xfs partition options for arbiter brick.
(In reply to Raghavendra Talur from comment #3) > Analysis: > > when a 13G volume with average size of 500 KB is created: > we get an arbiter calculation of: bricksize: 13631488, filesize: 500, > arbiterbricksize: 27262 > mkfs command: mkfs.xfs -i size=512 -n size=8192 > /dev/mapper/vg_8b7da1a300b44823878dfab9c4112224- > brick_bbc42f67c168b1b8d1e6f9b176088491 > > Hence, with each file taking up 1K in inode block, we should be able to > create 27262 files. > However, the stat info is > > File: > ‘/var/lib/heketi/mounts/vg_8b7da1a300b44823878dfab9c4112224/ > brick_bbc42f67c168b1b8d1e6f9b176088491’ > Size: 19 Blocks: 0 IO Block: 4096 directory > Device: fc04h/64516d Inode: 14336 Links: 3 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Context: system_u:object_r:unlabeled_t:s0 > Access: 1970-01-01 05:30:00.000000000 +0530 > Modify: 2018-08-08 23:03:07.302437334 +0530 > Change: 2018-08-08 23:03:07.302437334 +0530 > Birth: - > > > We can see only 14336 inodes available. I should have pasted the df output, not stat. Below is df output showing about 14K inodes. xfs 14K 26 14K 1% 22M 1.6M 20M 8% - > > > Formatting the space partition with xfs inode option of maxpct=100 gives us > about 38K inodes. 38K being greater than 27262 satisfies our requirement. > > We get more inodes than the calculation in heketi because in heketi we also > accounted for blocks needed by those 27262 files' extended attributes. To > test the same, I create a arbiter volume and created a file. Then I copied > over the file to xfs partition with maxpct=100, 27262 times. This test > worked. > > Other tests performed: > I added a extended attribute to all files using setfattr -n user.rtalur1 -v > a10byteval > I added another set of ea to all files using setfattr -n user.rtalur2 -v > a10byteval > > At this point, we still have about 75% of the partition free. I think this > is a satisfactory test. > > Hence, I will create a patch which changes xfs partition options for arbiter > brick.
Patch posted at https://github.com/heketi/heketi/pull/1306
Verified it using "rhgs3/rhgs-volmanager-rhel7:3.4.0-3". Where we can create amount of files on arbiter bricks equal to "volume size / avg file size".
Updated doc text in the Doc Text field. Please review for technical accuracy.
Doc Text looks OK
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2686