Description of problem: ---------------------- Looks like we have regressed with small file rmdirs and delete-renamed with this build by almost 30%. Version-Release number of selected component (if applicable): ------------------------------------------------------------ glusterfs-3.7.9-2.el6rhs.x86_64 How reproducible: ---------------- 2/2 Steps to Reproduce: ------------------- 1. Mount the dist rep volume via FUSE/gNFS/SMB as the issue is seen across all the three access mechanisms 2. Run small file workload (specifically rmdir,delete-renamed) 3. Check for regression from 3.1.2 Actual results: --------------- The calculated IOPS with this build is beyond 10% from the baseline Expected results: ----------------- The expected IOPS value should be within +-10% of the basleine. Additional info: ---------------- OS : RHEL 6.8 **************** VOL CONFIGURTION **************** [root@gqas001 ~]# gluster v info Volume Name: testvol Type: Distributed-Replicate Volume ID: b59cdf1b-ee19-4558-84ad-6e0e88bf4289 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gqas001.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0 Brick2: gqas014.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1 Brick3: gqas015.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2 Brick4: gqas016.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3 Options Reconfigured: server.allow-insecure: on performance.stat-prefetch: off performance.readdir-ahead: on [root@gqas001 ~]# I am hitting this issue with and without "Small File Performance Enhancements".
********** With 3.1.2 ********** delete-renamed : 7229.080000 files/sec rmdir : 2761.420000 files/sec These numbers are from a "non tuned" volume ******************* With 3.1.3(3.7.9-2) ******************* delete-renamed : 4728.684409 files/sec rmdir : 1946.532115 files/sec Small file performance enhancements(lookup optimize enabled and setting client and server event threads to 4 ) doesn't help either with 3.1.3: delete-renamed : 3897.576407 files/sec rmdir : 2718.358495 files/sec
************** EXACT WORKLOAD ************** Order of operations : (Create -> Read -> Append -> Rename -> Delete-Renamed -> Mkdir -> Rmdir -> Cleanup)*3 CLI example: python /small-files/smallfile/smallfile_cli.py --operation rmdir --threads 8 --file-size 64 --files 10000 --top /gluster-mount --pause 1000 --host-set "`echo $CLIENT | tr ' ' ','`"