1404113 – [Perf] 12% Drop in sequential reads on SMB v1.0 on a *2 deployment

Bug 1404113 - [Perf] 12% Drop in sequential reads on SMB v1.0 on a *2 deployment

Summary: [Perf] 12% Drop in sequential reads on SMB v1.0 on a *2 deployment

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	io-threads
Sub Component:
Version:	rhgs-3.2
Hardware:	All
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Ravishankar N
QA Contact:	Nag Pavan Chilakam
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-12-13 06:13 UTC by Karan Sandha
Modified:	2018-11-09 03:54 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-11-09 03:54:53 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1429233	0	unspecified	CLOSED	[Perf] : 14% regression on SMBv1 Sequential Reads on plain distribute volumes.	2021-02-22 00:41:40 UTC

Internal Links: 1429233

Description Karan Sandha 2016-12-13 06:13:38 UTC

Description of problem:

There's a 12% regression on sequential reads on SMB v1.0 

3.1.3    :- 667526 KB/s 
3.8.4.5  :- 587465 KB/s

Version-Release number of selected component (if applicable):
3.8.4.5

How reproducible:
100%

Steps to Reproduce:
1. Create 4 servers and 4 clients
2. Mount the clients on SMB v1.0
3. Run the iozone command :-
#qeSyncDropCaches.sh
#iozone -+m /clients.ioz -+h $(hostname) -C -w -c -e -i 1 -+n -r 64k -s 8g -t 16
4. Compare the numbers with 3.1.3 

Actual results:
There is a regression on 12% on sequential reads

Expected results:
The shouldn't be any regression.

Additional info:
[root@gqas003 ~]# gluster v info
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: c16f6e36-0e41-4a35-9966-51ef44d8fc29
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas003.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
server.allow-insecure: on
performance.stat-prefetch: off
performance.readdir-ahead: on
Volume profile for 3.8.4.5 for SMB v1 reads


[root@gqas003 ~]# gluster volume profile testvol info
Brick: gqas003.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
----------------------------------------------------------------
Cumulative Stats:
   Block Size:             131072b+ 
 No. of Reads:               470658 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              3  RELEASEDIR
      0.00     146.00 us     129.00 us     156.00 us              4    GETXATTR
      0.01      96.29 us      57.00 us     157.00 us             38       FSTAT
      0.03     204.38 us     185.00 us     387.00 us             72      LOOKUP
     99.96     180.90 us      71.00 us   41153.00 us         233461        READ
 
    Duration: 565 seconds
   Data Read: 61690085376 bytes
Data Written: 0 bytes
 
Interval 0 Stats:
   Block Size:             131072b+ 
 No. of Reads:               470658 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              3  RELEASEDIR
      0.00     146.00 us     129.00 us     156.00 us              4    GETXATTR
      0.01      96.29 us      57.00 us     157.00 us             38       FSTAT
      0.03     204.38 us     185.00 us     387.00 us             72      LOOKUP
     99.96     180.90 us      71.00 us   41153.00 us         233461        READ
 
    Duration: 565 seconds
   Data Read: 61690085376 bytes
Data Written: 0 bytes
 
Brick: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
----------------------------------------------------------------
Cumulative Stats:
   Block Size:             131072b+ 
 No. of Reads:               627420 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              3  RELEASEDIR
      0.00     149.60 us     131.00 us     163.00 us              5    GETXATTR
      0.01      92.42 us      55.00 us     139.00 us             43       FSTAT
      0.02     208.16 us     190.00 us     294.00 us             56      LOOKUP
     99.97     178.00 us      71.00 us   64396.00 us         311581        READ
 
    Duration: 565 seconds
   Data Read: 82237194240 bytes
Data Written: 0 bytes
 
Interval 0 Stats:
   Block Size:             131072b+ 
 No. of Reads:               627420 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              3  RELEASEDIR
      0.00     149.60 us     131.00 us     163.00 us              5    GETXATTR
      0.01      92.42 us      55.00 us     139.00 us             43       FSTAT
      0.02     208.16 us     190.00 us     294.00 us             56      LOOKUP
     99.97     178.01 us      71.00 us   64396.00 us         311582        READ
 
    Duration: 565 seconds
   Data Read: 82237194240 bytes
Data Written: 0 bytes
 
Brick: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
----------------------------------------------------------------
Cumulative Stats:
   Block Size:             131072b+ 
 No. of Reads:               234180 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              3  RELEASEDIR
      0.00     146.50 us     136.00 us     157.00 us              2    GETXATTR
      0.01      97.17 us      58.00 us     139.00 us             18       FSTAT
      0.05     188.45 us     150.00 us     339.00 us             56      LOOKUP
     99.94     186.35 us      72.00 us   19972.00 us         114990        READ
 
    Duration: 565 seconds
   Data Read: 30694440960 bytes
Data Written: 0 bytes
 
Interval 0 Stats:
   Block Size:             131072b+ 
 No. of Reads:               234180 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              3  RELEASEDIR
      0.00     146.50 us     136.00 us     157.00 us              2    GETXATTR
      0.01      97.17 us      58.00 us     139.00 us             18       FSTAT
      0.05     188.45 us     150.00 us     339.00 us             56      LOOKUP
     99.94     186.35 us      72.00 us   19972.00 us         114990        READ
 
    Duration: 565 seconds
   Data Read: 30694440960 bytes
Data Written: 0 bytes
 
Brick: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
----------------------------------------------------------------
Cumulative Stats:
   Block Size:             131072b+ 
 No. of Reads:               533214 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              3  RELEASEDIR
      0.00     150.60 us     138.00 us     192.00 us              5    GETXATTR
      0.01      94.38 us      57.00 us     132.00 us             50       FSTAT
      0.03     204.81 us     133.00 us     311.00 us             72      LOOKUP
     99.96     178.41 us      61.00 us   34804.00 us         262891        READ
 
    Duration: 565 seconds
   Data Read: 69889425408 bytes
Data Written: 0 bytes
 
Interval 0 Stats:
   Block Size:             131072b+ 
 No. of Reads:               533214 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              3  RELEASEDIR
      0.00     150.60 us     138.00 us     192.00 us              5    GETXATTR
      0.01      94.38 us      57.00 us     132.00 us             50       FSTAT
      0.03     204.81 us     133.00 us     311.00 us             72      LOOKUP
     99.96     178.41 us      61.00 us   34804.00 us         262891        READ
 
    Duration: 565 seconds
   Data Read: 69889425408 bytes
Data Written: 0 bytes
 






Volume profile for 3.1.3 smb v1 reads:-


[root@gqas003 ~]# gluster volume profile testvol info
Brick: gqas003.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
----------------------------------------------------------------
Cumulative Stats:
   Block Size:              65536b+              131072b+ 
 No. of Reads:                    0               1012239 
No. of Writes:               100748                539446 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             29     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            118  RELEASEDIR
      0.00       3.00 us       3.00 us       3.00 us              2     OPENDIR
      0.00     131.25 us      21.00 us     191.00 us             12    GETXATTR
      0.00     739.25 us     298.00 us    1263.00 us              4     READDIR
      0.01      79.98 us      51.00 us     118.00 us             61       FSTAT
      0.02     190.55 us      22.00 us     491.00 us             74      LOOKUP
     99.97     168.24 us      68.00 us   21108.00 us         503315        READ
 
    Duration: 34902 seconds
   Data Read: 132676190208 bytes
Data Written: 77309411328 bytes
 
Interval 0 Stats:
   Block Size:              65536b+              131072b+ 
 No. of Reads:                    0               1012239 
No. of Writes:               100748                539446 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             29     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            118  RELEASEDIR
      0.00       3.00 us       3.00 us       3.00 us              2     OPENDIR
      0.00     131.25 us      21.00 us     191.00 us             12    GETXATTR
      0.00     739.25 us     298.00 us    1263.00 us              4     READDIR
      0.01      79.98 us      51.00 us     118.00 us             61       FSTAT
      0.02     190.55 us      22.00 us     491.00 us             74      LOOKUP
     99.97     168.24 us      68.00 us   21108.00 us         503315        READ
 
    Duration: 34902 seconds
   Data Read: 132676190208 bytes
Data Written: 77309411328 bytes
 
Brick: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
----------------------------------------------------------------
Cumulative Stats:
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     0                288381 
No. of Writes:                    1                 82543                417478 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             26     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            118  RELEASEDIR
      0.00       3.50 us       3.00 us       4.00 us              2     OPENDIR
      0.00     119.67 us      18.00 us     167.00 us              6    GETXATTR
      0.01      98.00 us      51.00 us     135.00 us             17       FSTAT
      0.01     751.00 us     347.00 us    1234.00 us              4     READDIR
      0.04     186.31 us      34.00 us     280.00 us             58      LOOKUP
     99.94     178.51 us      69.00 us   18705.00 us         142597        READ
 
    Duration: 34902 seconds
   Data Read: 37798674432 bytes
Data Written: 60129542144 bytes
 
Interval 0 Stats:
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     0                288381 
No. of Writes:                    1                 82543                417478 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             26     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            118  RELEASEDIR
      0.00       3.50 us       3.00 us       4.00 us              2     OPENDIR
      0.00     119.67 us      18.00 us     167.00 us              6    GETXATTR
      0.01      98.00 us      51.00 us     135.00 us             17       FSTAT
      0.01     751.00 us     347.00 us    1234.00 us              4     READDIR
      0.04     186.31 us      34.00 us     280.00 us             58      LOOKUP
     99.94     178.51 us      69.00 us   18705.00 us         142597        READ
 
    Duration: 34902 seconds
   Data Read: 37798674432 bytes
Data Written: 60129542144 bytes
 
Brick: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
----------------------------------------------------------------
Cumulative Stats:
   Block Size:              65536b+              131072b+ 
 No. of Reads:                    0                     0 
No. of Writes:               100748                539446 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             29     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            118  RELEASEDIR
      0.04       3.00 us       3.00 us       3.00 us              2     OPENDIR
      1.97     108.00 us      89.00 us     140.00 us              3    GETXATTR
     18.38     754.25 us     308.00 us    1410.00 us              4     READDIR
     79.61     176.64 us      89.00 us     249.00 us             74      LOOKUP
 
    Duration: 34902 seconds
   Data Read: 0 bytes
Data Written: 77309411328 bytes
 
Interval 0 Stats:
   Block Size:              65536b+              131072b+ 
 No. of Reads:                    0                     0 
No. of Writes:               100748                539446 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             29     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            118  RELEASEDIR
      0.04       3.00 us       3.00 us       3.00 us              2     OPENDIR
      1.97     108.00 us      89.00 us     140.00 us              3    GETXATTR
     18.38     754.25 us     308.00 us    1410.00 us              4     READDIR
     79.61     176.64 us      89.00 us     249.00 us             74      LOOKUP
 
    Duration: 34902 seconds
   Data Read: 0 bytes
Data Written: 77309411328 bytes
 
Brick: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
----------------------------------------------------------------
Cumulative Stats:
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     0                377236 
No. of Writes:                    1                 82543                417478 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             26     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            118  RELEASEDIR
      0.00       3.50 us       3.00 us       4.00 us              2     OPENDIR
      0.00     130.86 us      83.00 us     158.00 us              7    GETXATTR
      0.01     103.42 us      66.00 us     146.00 us             24       FSTAT
      0.01     746.00 us     361.00 us    1261.00 us              4     READDIR
      0.03     191.74 us      83.00 us     330.00 us             58      LOOKUP
     99.95     182.41 us      69.00 us   16281.00 us         186660        READ
 
    Duration: 34902 seconds
   Data Read: 49445076992 bytes
Data Written: 60129542144 bytes
 
Interval 0 Stats:
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     0                377236 
No. of Writes:                    1                 82543                417478 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             26     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            118  RELEASEDIR
      0.00       3.50 us       3.00 us       4.00 us              2     OPENDIR
      0.00     130.86 us      83.00 us     158.00 us              7    GETXATTR
      0.01     103.42 us      66.00 us     146.00 us             24       FSTAT
      0.01     746.00 us     361.00 us    1261.00 us              4     READDIR
      0.03     191.74 us      83.00 us     330.00 us             58      LOOKUP
     99.95     182.41 us      69.00 us   16281.00 us         186660        READ
 
    Duration: 34902 seconds
   Data Read: 49445076992 bytes
Data Written: 60129542144 bytes
 
[root@gqas003 ~]#

Comment 3 Michael Adam 2016-12-13 14:08:56 UTC

How does that relate to Bug #1397854 ?

Comment 16 Atin Mukherjee 2018-11-08 16:27:15 UTC

Is this still an issue? If not can we close this bug?

Comment 17 Ravishankar N 2018-11-09 03:54:53 UTC

I don't think we would be working on fixing rhgs-3.2 any more.Closing the BZ for now. Please re-open if needed or perf issues are seen on the latest rhgs version.

Note You need to log in before you can comment on or make changes to this bug.