Bug 1438706

Summary: Sharding: Fix a performance bug
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Atin Mukherjee <amukherj>
Component: shardingAssignee: Krutika Dhananjay <kdhananj>
Status: CLOSED ERRATA QA Contact: RamaKasturi <knarra>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.2CC: bugs, kdhananj, rcyriac, rhs-bugs, rnalakka, sasundar, storage-qa-internal
Target Milestone: ---Keywords: Triaged
Target Release: RHGS 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8.4-22 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1436739
: 1439731 (view as bug list) Environment:
Last Closed: 2017-09-21 04:35:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1436739    
Bug Blocks: 1417151, 1437326, 1437330, 1439731    

Description Atin Mukherjee 2017-04-04 08:59:59 UTC 
+++ This bug was initially created as a clone of Bug #1436739 +++

Description of problem:

As per Sanjay Rao's inputs, there was a performance drop in random reads fio workload when run through vms hosted on sharded volumes.

Volume profile indicated a big difference between the number of lookups sent by FUSE and number of lookups received by individual bricks.

Through code reading, it was found that there is a performance bug in shard which was causing the translator to trigger unusually high number of lookups for cache invalidation even when there was no modification to the file.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Worker Ant on 2017-03-28 10:23:33 EDT ---

REVIEW: https://review.gluster.org/16961 (features/shard: Pass the correct iatt for cache invalidation) posted (#1) for review on master by Krutika Dhananjay (kdhananj)

--- Additional comment from Worker Ant on 2017-03-30 01:48:38 EDT ---

REVIEW: https://review.gluster.org/16961 (features/shard: Pass the correct iatt for cache invalidation) posted (#2) for review on master by Krutika Dhananjay (kdhananj)

--- Additional comment from Worker Ant on 2017-03-30 02:15:59 EDT ---

COMMIT: https://review.gluster.org/16961 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit dd5ada1f11d76b4c55c7c55d23718617f11a6c12
Author: Krutika Dhananjay <kdhananj>
Date:   Tue Mar 28 19:26:41 2017 +0530

    features/shard: Pass the correct iatt for cache invalidation
    
    This fixes a performance issue with shard which was causing
    the translator to trigger unusually high number of lookups
    for cache invalidation even when there was no modification to
    the file.
    
    In shard_common_stat_cbk(), it is local->prebuf that contains the
    aggregated size and block count as opposed to buf which only holds the
    attributes for the physical copy of base shard. Passing buf for
    inode_ctx invalidation would always set refresh to true since the file
    size in inode ctx contains the aggregated size and would never be same
    as @buf->ia_size. This was leading to every write/read being preceded
    by a lookup on the base shard even when the file underwent no
    modification.
    
    Change-Id: Ib0349291d2d01f3782d6d0bdd90c6db5e0609210
    BUG: 1436739
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: https://review.gluster.org/16961
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
Comment 2 Krutika Dhananjay 2017-04-04 09:54:06 UTC 
https://code.engineering.redhat.com/gerrit/#/c/102390/
Comment 7 RamaKasturi 2017-06-02 10:25:02 UTC 
Had HC installed with glusterfs 3.2 bits and observed that lookups sent on the brick  were greater than the one sent on fuse mount. Used fio tools and ran 100 random read workload.

Lookup on fuse mount : 741
Lookup sent over the brick : 15540

Updated 3.2 bits to 3.3 and ran 100 read workload on the vms. I have run the workload thrice and i have got similar results twice but the first time lookups sent on the brick were greater.

First run:
=====================

=== Interval 0 stats ===
      Duration : 1021 secs
     BytesRead : 149981696
  BytesWritten : 49664

Block Size   :             512B+            1024B+            2048B+
Read Count   :                29                 0                 2
Write Count  :                59                 3                 0

Block Size   :            4096B+            8192B+           16384B+
Read Count   :               110             12007                 2
Write Count  :                 0                 0                 1

Block Size   :           32768B+           65536B+          131072B+
Read Count   :                 2                11               384
Write Count  :                 0                 0                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               11987      569.88 us      190.00 us    31860.00 us
OPEN                  31       52.48 us       27.00 us      104.00 us
READ               12103    12023.84 us      341.00 us   209385.00 us
WRITE                 15     7500.80 us     1334.00 us    78435.00 us
STATFS                43      535.09 us      341.00 us     4031.00 us
FLUSH                 48       31.56 us       12.00 us       68.00 us
FSYNC                 18      912.50 us      423.00 us     1979.00 us
REMOVEXATTR            7      961.29 us      894.00 us     1103.00 us
OPENDIR               31      618.26 us      492.00 us     1432.00 us
FSTAT                 42     1106.86 us      256.00 us    29577.00 us
LOOKUP               269     2146.18 us      482.00 us    87316.00 us
SETATTR                7      856.14 us      749.00 us     1078.00 us
READDIRP              62     1465.13 us       16.00 us    43345.00 us
RELEASE              186           0 us           0 us           0 us
RELEASEDIR           189           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

=== Interval 0 stats ===
      Duration : 1022 secs
     BytesRead : 842203648
  BytesWritten : 5270016

Block Size   :             512B+            1024B+            2048B+
Read Count   :               321                 0                 3
Write Count  :                63                 3                 0

Block Size   :            4096B+            8192B+           16384B+
Read Count   :               409             12008                10
Write Count  :                 3                 0                 1

Block Size   :           32768B+           65536B+          131072B+
Read Count   :                 6               507              5250
Write Count  :                 1                 7                34

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               12012      688.10 us      191.00 us    29323.00 us
OPEN                 177       46.47 us       22.00 us       92.00 us
READ               13041    10628.98 us      315.00 us   279819.00 us
WRITE                 14     3515.00 us     1525.00 us    22023.00 us
STATFS                41      467.46 us      297.00 us      782.00 us
FLUSH                280       32.73 us       12.00 us       72.00 us
FSYNC                 18      894.11 us      447.00 us     1864.00 us
REMOVEXATTR            7      905.86 us      826.00 us      994.00 us
OPENDIR               33      789.85 us      430.00 us     5189.00 us
FSTAT                 59      521.12 us      224.00 us     8359.00 us
LOOKUP               542     3737.25 us      487.00 us    59769.00 us
SETATTR                7      823.14 us      649.00 us      998.00 us
READDIRP              66      857.58 us       15.00 us    36928.00 us
RELEASE             1051           0 us           0 us           0 us
RELEASEDIR           204           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

=== Interval 0 stats ===
      Duration : 1022 secs
     BytesRead : 151019520
  BytesWritten : 2146816

Block Size   :             512B+            1024B+            2048B+
Read Count   :                76                 0                 1
Write Count  :                59                 3                 0

Block Size   :            4096B+            8192B+           16384B+
Read Count   :               249             12013                 5
Write Count  :                 3                 0                 1

Block Size   :           32768B+           65536B+          131072B+
Read Count   :                 3                13               384
Write Count  :                 1                 4                12

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               11985      678.82 us      198.00 us    24093.00 us
OPEN                  29       45.83 us       28.00 us       69.00 us
READ               12099    10853.84 us      330.00 us   218767.00 us
WRITE                 15     5502.40 us     1318.00 us    53345.00 us
STATFS                42      444.74 us      349.00 us      537.00 us
FLUSH                 46       33.83 us       14.00 us       61.00 us
FSYNC                 18      750.39 us      422.00 us     1604.00 us
REMOVEXATTR            6      917.33 us      854.00 us      978.00 us
OPENDIR               30      673.97 us      461.00 us     4076.00 us
FSTAT                 56      435.66 us      296.00 us     2610.00 us
LOOKUP               265     1870.12 us      473.00 us    25081.00 us
SETATTR                6      802.50 us      741.00 us      883.00 us
READDIRP              60     1277.52 us       18.00 us    35013.00 us
RELEASE              184           0 us           0 us           0 us
RELEASEDIR           191           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
vol_profile_output:
===========================
Interval 0 Stats:
   Block Size:                256b+                 512b+                1024b+
 No. of Reads:                   19                     0                     0
No. of Writes:                    0                    30                     9

   Block Size:               8192b+               16384b+               65536b+
 No. of Reads:                12007                     0                     6
No. of Writes:                    0                     3                     0

   Block Size:             131072b+
 No. of Reads:                   64
No. of Writes:                    0
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             93  RELEASEDIR
      0.00     108.00 us     108.00 us     108.00 us              1        OPEN
      0.00      37.50 us      22.00 us      73.00 us              4     READDIR
      0.00      57.42 us      28.00 us     104.00 us             84    FINODELK
      0.00      73.95 us       2.00 us     117.00 us             93     OPENDIR
      0.01      66.79 us      26.00 us     118.00 us            124      STATFS
      0.01     309.07 us      62.00 us     824.00 us             54       FSYNC
      0.02     763.58 us      35.00 us   29177.00 us             43       FSTAT
      0.04     614.07 us     117.00 us   16973.00 us             84    FXATTROP
      0.04    1365.17 us     162.00 us   24131.00 us             42       WRITE
      0.06    1575.48 us      36.00 us   43256.00 us             52    READDIRP
      0.10     165.39 us      22.00 us   11109.00 us            830     SETATTR
      0.13     231.57 us      42.00 us   36982.00 us            811    SETXATTR
      0.14     248.70 us      41.00 us   21395.00 us            830 REMOVEXATTR
      0.27     328.41 us      22.00 us   37087.00 us           1167    GETXATTR
      0.37     456.10 us     113.00 us   30304.00 us           1163     XATTROP
      0.79      94.76 us      35.00 us   31630.00 us          11987        STAT
      1.01     604.74 us      18.00 us   64453.00 us           2411     INODELK
      3.10    1000.71 us      24.00 us   63019.00 us           4462      LOOKUP
     93.92   11193.88 us     142.00 us  209114.00 us          12096        READ

    Duration: 173 seconds
   Data Read: 107151813 bytes
Data Written: 73728 bytes

Lookup sent on fuse mount : 1076
Lookup sent over brick : 4462

Second Run:
====================
=== Interval 1 stats ===
      Duration : 404 secs
     BytesRead : 119500800
  BytesWritten : 10240

Block Size   :             512B+            4096B+            8192B+
Read Count   :                 0                41             12007
Write Count  :                20                 0                 0

Block Size   :          131072B+
Read Count   :               160
Write Count  :                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               11988      554.58 us      204.00 us     1794.00 us
OPEN                  31       46.52 us       28.00 us       75.00 us
READ               12107    10435.64 us      344.00 us   415124.00 us
WRITE                 10     3082.40 us     2172.00 us     7542.00 us
STATFS                44      509.98 us      380.00 us     1104.00 us
FLUSH                 50       28.78 us       11.00 us       60.00 us
REMOVEXATTR            6     2358.50 us      887.00 us     9236.00 us
OPENDIR               33     1843.15 us      508.00 us    14872.00 us
FSTAT                 46      421.09 us      233.00 us     1191.00 us
LOOKUP               277     2001.56 us      514.00 us    30868.00 us
SETATTR                6      912.17 us      837.00 us      953.00 us
READDIRP              66      766.29 us       14.00 us    29581.00 us
RELEASE               68           0 us           0 us           0 us
RELEASEDIR            71           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

=== Interval 1 stats ===
      Duration : 404 secs
     BytesRead : 395698176
  BytesWritten : 10240

Block Size   :             512B+            4096B+            8192B+
Read Count   :                72                41             12002
Write Count  :                20                 0                 0

Block Size   :           65536B+          131072B+
Read Count   :               196              2108
Write Count  :                 0                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               12014      717.24 us      202.00 us    36196.00 us
OPEN                 188       48.29 us       23.00 us      109.00 us
READ               13119    10006.90 us      275.00 us   269229.00 us
WRITE                  9     2375.11 us     2191.00 us     2555.00 us
STATFS                44     1183.20 us      376.00 us    19794.00 us
FLUSH                298       33.81 us       11.00 us       76.00 us
REMOVEXATTR            6      991.00 us      864.00 us     1263.00 us
OPENDIR               31      634.19 us      481.00 us     1094.00 us
FSTAT                 60      406.83 us      200.00 us      770.00 us
LOOKUP               583     3492.63 us      504.00 us    32151.00 us
SETATTR                6      809.17 us      754.00 us      917.00 us
READDIRP              62      311.15 us       16.00 us     1003.00 us
RELEASE              408           0 us           0 us           0 us
RELEASEDIR            69           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

=== Interval 1 stats ===
      Duration : 404 secs
     BytesRead : 119484416
  BytesWritten : 10240

Block Size   :             512B+            4096B+            8192B+
Read Count   :                 0                41             12005
Write Count  :                20                 0                 0

Block Size   :          131072B+
Read Count   :               160
Write Count  :                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               11979      686.02 us      195.00 us    33833.00 us
OPEN                  31       38.81 us       25.00 us       70.00 us
READ               12105    10773.69 us      342.00 us   213621.00 us
WRITE                 10     2432.90 us     2245.00 us     2622.00 us
STATFS                44      426.73 us      335.00 us      674.00 us
FLUSH                 50       31.84 us       12.00 us       60.00 us
REMOVEXATTR            6      857.00 us      803.00 us      932.00 us
OPENDIR               32      573.72 us      443.00 us     1296.00 us
FSTAT                 64      484.00 us      276.00 us     3078.00 us
LOOKUP               269     1648.59 us      488.00 us    33870.00 us
SETATTR                6      750.33 us      703.00 us      840.00 us
READDIRP              64     1220.69 us       15.00 us    59162.00 us
RELEASE               68           0 us           0 us           0 us
RELEASEDIR            71           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

volume profile:
=============================
Interval 0 Stats:
   Block Size:                256b+                 512b+                8192b+
 No. of Reads:                   18                     0                 12007
No. of Writes:                    0                    24                     0

   Block Size:             131072b+
 No. of Reads:                   64
No. of Writes:                    0
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             86  RELEASEDIR
      0.00     377.33 us     253.00 us     724.00 us              6     READDIR
      0.00      58.69 us      29.00 us      93.00 us             48    FINODELK
      0.00      74.67 us      38.00 us     144.00 us             40       FSTAT
      0.01      73.49 us       3.00 us     143.00 us             86     OPENDIR
      0.01      67.92 us      33.00 us     118.00 us            116      STATFS
      0.01     168.48 us     103.00 us     234.00 us             48    FXATTROP
      0.01     107.80 us      69.00 us     153.00 us            164 REMOVEXATTR
      0.02     132.43 us      89.00 us     201.00 us            148    SETXATTR
      0.02     122.78 us      31.00 us     193.00 us            164     SETATTR
      0.02    1087.46 us     769.00 us    5997.00 us             24       WRITE
      0.02      65.09 us      24.00 us     113.00 us            404     INODELK
      0.03     178.39 us      24.00 us     457.00 us            175    GETXATTR
      0.03     215.85 us     160.00 us     521.00 us            170     XATTROP
      0.03     760.12 us      37.00 us   29202.00 us             51    READDIRP
      0.21     188.61 us      36.00 us   15933.00 us           1412      LOOKUP
      0.76      78.78 us      29.00 us     723.00 us          11987        STAT
     98.82   10132.91 us     135.00 us  414844.00 us          12089        READ

    Duration: 167 seconds
   Data Read: 106758142 bytes
Data Written: 12288 bytes


Lookup sent over fuse : 1129
Lookup sent over brick : 1412

Third run:
==============================
=== Interval 2 stats ===
      Duration : 892 secs
     BytesRead : 143814656
  BytesWritten : 22016

Block Size   :             512B+            4096B+            8192B+
Read Count   :                 0                91             12006
Write Count  :                43                 0                 0

Block Size   :          131072B+
Read Count   :               344
Write Count  :                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               11981      557.32 us      209.00 us     1811.00 us
OPEN                  34       43.24 us       22.00 us       71.00 us
READ               12099    11715.72 us      379.00 us   218366.00 us
WRITE                  9     2543.67 us     2284.00 us     2803.00 us
STATFS                45      497.62 us      357.00 us      713.00 us
FLUSH                 54       32.30 us       13.00 us       66.00 us
REMOVEXATTR            7     1033.14 us      822.00 us     1228.00 us
OPENDIR               33      616.15 us      513.00 us      741.00 us
FSTAT                 61      416.98 us      230.00 us     1320.00 us
LOOKUP               278     1807.43 us      532.00 us    12786.00 us
SETATTR                7      969.14 us      771.00 us     1368.00 us
READDIRP              66      303.11 us       15.00 us      820.00 us
RELEASE              152           0 us           0 us           0 us
RELEASEDIR           155           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

=== Interval 2 stats ===
      Duration : 892 secs
     BytesRead : 752958464
  BytesWritten : 22016

Block Size   :             512B+            4096B+            8192B+
Read Count   :               158                90             12006
Write Count  :                43                 0                 0

Block Size   :           65536B+          131072B+
Read Count   :               433              4639
Write Count  :                 0                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               12017      550.86 us      201.00 us    12317.00 us
OPEN                 190       45.58 us       22.00 us      129.00 us
READ               13130    10153.46 us      270.00 us   260762.00 us
WRITE                  9     3066.78 us     2346.00 us     7145.00 us
STATFS                45      535.87 us      354.00 us     2304.00 us
FLUSH                302       34.14 us       11.00 us       71.00 us
REMOVEXATTR            7      996.14 us      848.00 us     1119.00 us
OPENDIR               36      685.11 us      488.00 us     1982.00 us
FSTAT                 63      385.44 us      230.00 us      635.00 us
LOOKUP               578     3320.98 us      489.00 us    36146.00 us
SETATTR                7      878.71 us      615.00 us     1027.00 us
READDIRP              72      297.44 us       17.00 us      895.00 us
RELEASE              902           0 us           0 us           0 us
RELEASEDIR           160           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

=== Interval 2 stats ===
      Duration : 892 secs
     BytesRead : 143806464
  BytesWritten : 22016

Block Size   :             512B+            4096B+            8192B+
Read Count   :                 0                91             12005
Write Count  :                43                 0                 0

Block Size   :          131072B+
Read Count   :               344
Write Count  :                 0

Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
---           ----------    -----------    -----------    -----------
STAT               11977      714.93 us      193.00 us    63057.00 us
OPEN                  32       39.69 us       22.00 us       71.00 us
READ               12098    10851.17 us      352.00 us   268565.00 us
WRITE                  9     2406.11 us     2243.00 us     2701.00 us
STATFS                44      435.57 us      303.00 us     1176.00 us
FLUSH                 52       29.65 us       12.00 us       49.00 us
REMOVEXATTR            6      843.67 us      762.00 us      912.00 us
OPENDIR               31      865.48 us      458.00 us     9475.00 us
FSTAT                 61      409.67 us      257.00 us      627.00 us
LOOKUP               282     1754.85 us      460.00 us    25927.00 us
SETATTR                6      749.17 us      638.00 us      847.00 us
READDIRP              62      304.05 us       14.00 us      776.00 us
RELEASE              154           0 us           0 us           0 us
RELEASEDIR           155           0 us           0 us           0 us
------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----

volume profile:
=======================
Interval 0 Stats:
   Block Size:                256b+                 512b+                8192b+
 No. of Reads:                   18                     0                 12006
No. of Writes:                    0                    24                     0

   Block Size:             131072b+
 No. of Reads:                   64
No. of Writes:                    0
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             92  RELEASEDIR
      0.00      37.75 us      21.00 us      76.00 us              4     READDIR
      0.00      62.33 us      35.00 us      93.00 us             48    FINODELK
      0.00      79.50 us      33.00 us     119.00 us             58       FSTAT
      0.00      74.51 us       3.00 us     133.00 us             92     OPENDIR
      0.01      64.51 us      25.00 us     410.00 us            121      STATFS
      0.01     176.33 us     132.00 us     233.00 us             48    FXATTROP
      0.01     169.34 us      40.00 us     416.00 us             56    READDIRP
      0.01     103.52 us      69.00 us     147.00 us            173 REMOVEXATTR
      0.01     124.14 us      83.00 us     185.00 us            155    SETXATTR
      0.02     121.36 us      31.00 us     189.00 us            173     SETATTR
      0.02     899.67 us     747.00 us    1118.00 us             24       WRITE
      0.02      68.47 us      24.00 us    2359.00 us            435     INODELK
      0.02     175.49 us      21.00 us     270.00 us            185    GETXATTR
      0.03     210.42 us     154.00 us     281.00 us            181     XATTROP
      0.17     165.59 us      24.00 us     580.00 us           1451      LOOKUP
      0.67      78.49 us      38.00 us     442.00 us          11981        STAT
     98.99   11416.08 us     130.00 us  218011.00 us          12088        READ

    Duration: 169 seconds
   Data Read: 106749950 bytes
Data Written: 12288 bytes

Lookup sent over fuse mount : 1138
Lookup sent over brick : 1451

Any idea why this was more first time? I ran same workload which is 100% random read.
Comment 8 Krutika Dhananjay 2017-06-05 05:29:57 UTC 
(In reply to RamaKasturi from comment #7)
> Had HC installed with glusterfs 3.2 bits and observed that lookups sent on
> the brick  were greater than the one sent on fuse mount. Used fio tools and
> ran 100 random read workload.
> 
> Lookup on fuse mount : 741
> Lookup sent over the brick : 15540
> 
> Updated 3.2 bits to 3.3 and ran 100 read workload on the vms. I have run the
> workload thrice and i have got similar results twice but the first time
> lookups sent on the brick were greater.
> 
> First run:
> =====================
> 
> === Interval 0 stats ===
>       Duration : 1021 secs
>      BytesRead : 149981696
>   BytesWritten : 49664
> 
> Block Size   :             512B+            1024B+            2048B+
> Read Count   :                29                 0                 2
> Write Count  :                59                 3                 0
> 
> Block Size   :            4096B+            8192B+           16384B+
> Read Count   :               110             12007                 2
> Write Count  :                 0                 0                 1
> 
> Block Size   :           32768B+           65536B+          131072B+
> Read Count   :                 2                11               384
> Write Count  :                 0                 0                 0
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               11987      569.88 us      190.00 us    31860.00 us
> OPEN                  31       52.48 us       27.00 us      104.00 us
> READ               12103    12023.84 us      341.00 us   209385.00 us
> WRITE                 15     7500.80 us     1334.00 us    78435.00 us
> STATFS                43      535.09 us      341.00 us     4031.00 us
> FLUSH                 48       31.56 us       12.00 us       68.00 us
> FSYNC                 18      912.50 us      423.00 us     1979.00 us
> REMOVEXATTR            7      961.29 us      894.00 us     1103.00 us
> OPENDIR               31      618.26 us      492.00 us     1432.00 us
> FSTAT                 42     1106.86 us      256.00 us    29577.00 us
> LOOKUP               269     2146.18 us      482.00 us    87316.00 us
> SETATTR                7      856.14 us      749.00 us     1078.00 us
> READDIRP              62     1465.13 us       16.00 us    43345.00 us
> RELEASE              186           0 us           0 us           0 us
> RELEASEDIR           189           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> 
> === Interval 0 stats ===
>       Duration : 1022 secs
>      BytesRead : 842203648
>   BytesWritten : 5270016
> 
> Block Size   :             512B+            1024B+            2048B+
> Read Count   :               321                 0                 3
> Write Count  :                63                 3                 0
> 
> Block Size   :            4096B+            8192B+           16384B+
> Read Count   :               409             12008                10
> Write Count  :                 3                 0                 1
> 
> Block Size   :           32768B+           65536B+          131072B+
> Read Count   :                 6               507              5250
> Write Count  :                 1                 7                34
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               12012      688.10 us      191.00 us    29323.00 us
> OPEN                 177       46.47 us       22.00 us       92.00 us
> READ               13041    10628.98 us      315.00 us   279819.00 us
> WRITE                 14     3515.00 us     1525.00 us    22023.00 us
> STATFS                41      467.46 us      297.00 us      782.00 us
> FLUSH                280       32.73 us       12.00 us       72.00 us
> FSYNC                 18      894.11 us      447.00 us     1864.00 us
> REMOVEXATTR            7      905.86 us      826.00 us      994.00 us
> OPENDIR               33      789.85 us      430.00 us     5189.00 us
> FSTAT                 59      521.12 us      224.00 us     8359.00 us
> LOOKUP               542     3737.25 us      487.00 us    59769.00 us
> SETATTR                7      823.14 us      649.00 us      998.00 us
> READDIRP              66      857.58 us       15.00 us    36928.00 us
> RELEASE             1051           0 us           0 us           0 us
> RELEASEDIR           204           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> 
> === Interval 0 stats ===
>       Duration : 1022 secs
>      BytesRead : 151019520
>   BytesWritten : 2146816
> 
> Block Size   :             512B+            1024B+            2048B+
> Read Count   :                76                 0                 1
> Write Count  :                59                 3                 0
> 
> Block Size   :            4096B+            8192B+           16384B+
> Read Count   :               249             12013                 5
> Write Count  :                 3                 0                 1
> 
> Block Size   :           32768B+           65536B+          131072B+
> Read Count   :                 3                13               384
> Write Count  :                 1                 4                12
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               11985      678.82 us      198.00 us    24093.00 us
> OPEN                  29       45.83 us       28.00 us       69.00 us
> READ               12099    10853.84 us      330.00 us   218767.00 us
> WRITE                 15     5502.40 us     1318.00 us    53345.00 us
> STATFS                42      444.74 us      349.00 us      537.00 us
> FLUSH                 46       33.83 us       14.00 us       61.00 us
> FSYNC                 18      750.39 us      422.00 us     1604.00 us
> REMOVEXATTR            6      917.33 us      854.00 us      978.00 us
> OPENDIR               30      673.97 us      461.00 us     4076.00 us
> FSTAT                 56      435.66 us      296.00 us     2610.00 us
> LOOKUP               265     1870.12 us      473.00 us    25081.00 us
> SETATTR                6      802.50 us      741.00 us      883.00 us
> READDIRP              60     1277.52 us       18.00 us    35013.00 us
> RELEASE              184           0 us           0 us           0 us
> RELEASEDIR           191           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> vol_profile_output:
> ===========================
> Interval 0 Stats:
>    Block Size:                256b+                 512b+               
> 1024b+
>  No. of Reads:                   19                     0                   
> 0
> No. of Writes:                    0                    30                   
> 9
> 
>    Block Size:               8192b+               16384b+              
> 65536b+
>  No. of Reads:                12007                     0                   
> 6
> No. of Writes:                    0                     3                   
> 0
> 
>    Block Size:             131072b+
>  No. of Reads:                   64
> No. of Writes:                    0
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls        
> Fop
>  ---------   -----------   -----------   -----------   ------------       
> ----
>       0.00       0.00 us       0.00 us       0.00 us             93 
> RELEASEDIR
>       0.00     108.00 us     108.00 us     108.00 us              1       
> OPEN
>       0.00      37.50 us      22.00 us      73.00 us              4    
> READDIR
>       0.00      57.42 us      28.00 us     104.00 us             84   
> FINODELK
>       0.00      73.95 us       2.00 us     117.00 us             93    
> OPENDIR
>       0.01      66.79 us      26.00 us     118.00 us            124     
> STATFS
>       0.01     309.07 us      62.00 us     824.00 us             54      
> FSYNC
>       0.02     763.58 us      35.00 us   29177.00 us             43      
> FSTAT
>       0.04     614.07 us     117.00 us   16973.00 us             84   
> FXATTROP
>       0.04    1365.17 us     162.00 us   24131.00 us             42      
> WRITE
>       0.06    1575.48 us      36.00 us   43256.00 us             52   
> READDIRP
>       0.10     165.39 us      22.00 us   11109.00 us            830    
> SETATTR
>       0.13     231.57 us      42.00 us   36982.00 us            811   
> SETXATTR
>       0.14     248.70 us      41.00 us   21395.00 us            830
> REMOVEXATTR
>       0.27     328.41 us      22.00 us   37087.00 us           1167   
> GETXATTR
>       0.37     456.10 us     113.00 us   30304.00 us           1163    
> XATTROP
>       0.79      94.76 us      35.00 us   31630.00 us          11987       
> STAT
>       1.01     604.74 us      18.00 us   64453.00 us           2411    
> INODELK
>       3.10    1000.71 us      24.00 us   63019.00 us           4462     
> LOOKUP
>      93.92   11193.88 us     142.00 us  209114.00 us          12096       
> READ
> 
>     Duration: 173 seconds
>    Data Read: 107151813 bytes
> Data Written: 73728 bytes
> 
> Lookup sent on fuse mount : 1076
> Lookup sent over brick : 4462
> 
> Second Run:
> ====================
> === Interval 1 stats ===
>       Duration : 404 secs
>      BytesRead : 119500800
>   BytesWritten : 10240
> 
> Block Size   :             512B+            4096B+            8192B+
> Read Count   :                 0                41             12007
> Write Count  :                20                 0                 0
> 
> Block Size   :          131072B+
> Read Count   :               160
> Write Count  :                 0
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               11988      554.58 us      204.00 us     1794.00 us
> OPEN                  31       46.52 us       28.00 us       75.00 us
> READ               12107    10435.64 us      344.00 us   415124.00 us
> WRITE                 10     3082.40 us     2172.00 us     7542.00 us
> STATFS                44      509.98 us      380.00 us     1104.00 us
> FLUSH                 50       28.78 us       11.00 us       60.00 us
> REMOVEXATTR            6     2358.50 us      887.00 us     9236.00 us
> OPENDIR               33     1843.15 us      508.00 us    14872.00 us
> FSTAT                 46      421.09 us      233.00 us     1191.00 us
> LOOKUP               277     2001.56 us      514.00 us    30868.00 us
> SETATTR                6      912.17 us      837.00 us      953.00 us
> READDIRP              66      766.29 us       14.00 us    29581.00 us
> RELEASE               68           0 us           0 us           0 us
> RELEASEDIR            71           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> 
> === Interval 1 stats ===
>       Duration : 404 secs
>      BytesRead : 395698176
>   BytesWritten : 10240
> 
> Block Size   :             512B+            4096B+            8192B+
> Read Count   :                72                41             12002
> Write Count  :                20                 0                 0
> 
> Block Size   :           65536B+          131072B+
> Read Count   :               196              2108
> Write Count  :                 0                 0
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               12014      717.24 us      202.00 us    36196.00 us
> OPEN                 188       48.29 us       23.00 us      109.00 us
> READ               13119    10006.90 us      275.00 us   269229.00 us
> WRITE                  9     2375.11 us     2191.00 us     2555.00 us
> STATFS                44     1183.20 us      376.00 us    19794.00 us
> FLUSH                298       33.81 us       11.00 us       76.00 us
> REMOVEXATTR            6      991.00 us      864.00 us     1263.00 us
> OPENDIR               31      634.19 us      481.00 us     1094.00 us
> FSTAT                 60      406.83 us      200.00 us      770.00 us
> LOOKUP               583     3492.63 us      504.00 us    32151.00 us
> SETATTR                6      809.17 us      754.00 us      917.00 us
> READDIRP              62      311.15 us       16.00 us     1003.00 us
> RELEASE              408           0 us           0 us           0 us
> RELEASEDIR            69           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> 
> === Interval 1 stats ===
>       Duration : 404 secs
>      BytesRead : 119484416
>   BytesWritten : 10240
> 
> Block Size   :             512B+            4096B+            8192B+
> Read Count   :                 0                41             12005
> Write Count  :                20                 0                 0
> 
> Block Size   :          131072B+
> Read Count   :               160
> Write Count  :                 0
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               11979      686.02 us      195.00 us    33833.00 us
> OPEN                  31       38.81 us       25.00 us       70.00 us
> READ               12105    10773.69 us      342.00 us   213621.00 us
> WRITE                 10     2432.90 us     2245.00 us     2622.00 us
> STATFS                44      426.73 us      335.00 us      674.00 us
> FLUSH                 50       31.84 us       12.00 us       60.00 us
> REMOVEXATTR            6      857.00 us      803.00 us      932.00 us
> OPENDIR               32      573.72 us      443.00 us     1296.00 us
> FSTAT                 64      484.00 us      276.00 us     3078.00 us
> LOOKUP               269     1648.59 us      488.00 us    33870.00 us
> SETATTR                6      750.33 us      703.00 us      840.00 us
> READDIRP              64     1220.69 us       15.00 us    59162.00 us
> RELEASE               68           0 us           0 us           0 us
> RELEASEDIR            71           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> 
> volume profile:
> =============================
> Interval 0 Stats:
>    Block Size:                256b+                 512b+               
> 8192b+
>  No. of Reads:                   18                     0                
> 12007
> No. of Writes:                    0                    24                   
> 0
> 
>    Block Size:             131072b+
>  No. of Reads:                   64
> No. of Writes:                    0
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls        
> Fop
>  ---------   -----------   -----------   -----------   ------------       
> ----
>       0.00       0.00 us       0.00 us       0.00 us             86 
> RELEASEDIR
>       0.00     377.33 us     253.00 us     724.00 us              6    
> READDIR
>       0.00      58.69 us      29.00 us      93.00 us             48   
> FINODELK
>       0.00      74.67 us      38.00 us     144.00 us             40      
> FSTAT
>       0.01      73.49 us       3.00 us     143.00 us             86    
> OPENDIR
>       0.01      67.92 us      33.00 us     118.00 us            116     
> STATFS
>       0.01     168.48 us     103.00 us     234.00 us             48   
> FXATTROP
>       0.01     107.80 us      69.00 us     153.00 us            164
> REMOVEXATTR
>       0.02     132.43 us      89.00 us     201.00 us            148   
> SETXATTR
>       0.02     122.78 us      31.00 us     193.00 us            164    
> SETATTR
>       0.02    1087.46 us     769.00 us    5997.00 us             24      
> WRITE
>       0.02      65.09 us      24.00 us     113.00 us            404    
> INODELK
>       0.03     178.39 us      24.00 us     457.00 us            175   
> GETXATTR
>       0.03     215.85 us     160.00 us     521.00 us            170    
> XATTROP
>       0.03     760.12 us      37.00 us   29202.00 us             51   
> READDIRP
>       0.21     188.61 us      36.00 us   15933.00 us           1412     
> LOOKUP
>       0.76      78.78 us      29.00 us     723.00 us          11987       
> STAT
>      98.82   10132.91 us     135.00 us  414844.00 us          12089       
> READ
> 
>     Duration: 167 seconds
>    Data Read: 106758142 bytes
> Data Written: 12288 bytes
> 
> 
> Lookup sent over fuse : 1129
> Lookup sent over brick : 1412
> 
> Third run:
> ==============================
> === Interval 2 stats ===
>       Duration : 892 secs
>      BytesRead : 143814656
>   BytesWritten : 22016
> 
> Block Size   :             512B+            4096B+            8192B+
> Read Count   :                 0                91             12006
> Write Count  :                43                 0                 0
> 
> Block Size   :          131072B+
> Read Count   :               344
> Write Count  :                 0
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               11981      557.32 us      209.00 us     1811.00 us
> OPEN                  34       43.24 us       22.00 us       71.00 us
> READ               12099    11715.72 us      379.00 us   218366.00 us
> WRITE                  9     2543.67 us     2284.00 us     2803.00 us
> STATFS                45      497.62 us      357.00 us      713.00 us
> FLUSH                 54       32.30 us       13.00 us       66.00 us
> REMOVEXATTR            7     1033.14 us      822.00 us     1228.00 us
> OPENDIR               33      616.15 us      513.00 us      741.00 us
> FSTAT                 61      416.98 us      230.00 us     1320.00 us
> LOOKUP               278     1807.43 us      532.00 us    12786.00 us
> SETATTR                7      969.14 us      771.00 us     1368.00 us
> READDIRP              66      303.11 us       15.00 us      820.00 us
> RELEASE              152           0 us           0 us           0 us
> RELEASEDIR           155           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> 
> === Interval 2 stats ===
>       Duration : 892 secs
>      BytesRead : 752958464
>   BytesWritten : 22016
> 
> Block Size   :             512B+            4096B+            8192B+
> Read Count   :               158                90             12006
> Write Count  :                43                 0                 0
> 
> Block Size   :           65536B+          131072B+
> Read Count   :               433              4639
> Write Count  :                 0                 0
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               12017      550.86 us      201.00 us    12317.00 us
> OPEN                 190       45.58 us       22.00 us      129.00 us
> READ               13130    10153.46 us      270.00 us   260762.00 us
> WRITE                  9     3066.78 us     2346.00 us     7145.00 us
> STATFS                45      535.87 us      354.00 us     2304.00 us
> FLUSH                302       34.14 us       11.00 us       71.00 us
> REMOVEXATTR            7      996.14 us      848.00 us     1119.00 us
> OPENDIR               36      685.11 us      488.00 us     1982.00 us
> FSTAT                 63      385.44 us      230.00 us      635.00 us
> LOOKUP               578     3320.98 us      489.00 us    36146.00 us
> SETATTR                7      878.71 us      615.00 us     1027.00 us
> READDIRP              72      297.44 us       17.00 us      895.00 us
> RELEASE              902           0 us           0 us           0 us
> RELEASEDIR           160           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> 
> === Interval 2 stats ===
>       Duration : 892 secs
>      BytesRead : 143806464
>   BytesWritten : 22016
> 
> Block Size   :             512B+            4096B+            8192B+
> Read Count   :                 0                91             12005
> Write Count  :                43                 0                 0
> 
> Block Size   :          131072B+
> Read Count   :               344
> Write Count  :                 0
> 
> Fop           Call Count    Avg-Latency    Min-Latency    Max-Latency
> ---           ----------    -----------    -----------    -----------
> STAT               11977      714.93 us      193.00 us    63057.00 us
> OPEN                  32       39.69 us       22.00 us       71.00 us
> READ               12098    10851.17 us      352.00 us   268565.00 us
> WRITE                  9     2406.11 us     2243.00 us     2701.00 us
> STATFS                44      435.57 us      303.00 us     1176.00 us
> FLUSH                 52       29.65 us       12.00 us       49.00 us
> REMOVEXATTR            6      843.67 us      762.00 us      912.00 us
> OPENDIR               31      865.48 us      458.00 us     9475.00 us
> FSTAT                 61      409.67 us      257.00 us      627.00 us
> LOOKUP               282     1754.85 us      460.00 us    25927.00 us
> SETATTR                6      749.17 us      638.00 us      847.00 us
> READDIRP              62      304.05 us       14.00 us      776.00 us
> RELEASE              154           0 us           0 us           0 us
> RELEASEDIR           155           0 us           0 us           0 us
> ------ ----- ----- ----- ----- ----- ----- -----  ----- ----- ----- -----
> 
> volume profile:
> =======================
> Interval 0 Stats:
>    Block Size:                256b+                 512b+               
> 8192b+
>  No. of Reads:                   18                     0                
> 12006
> No. of Writes:                    0                    24                   
> 0
> 
>    Block Size:             131072b+
>  No. of Reads:                   64
> No. of Writes:                    0
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls        
> Fop
>  ---------   -----------   -----------   -----------   ------------       
> ----
>       0.00       0.00 us       0.00 us       0.00 us             92 
> RELEASEDIR
>       0.00      37.75 us      21.00 us      76.00 us              4    
> READDIR
>       0.00      62.33 us      35.00 us      93.00 us             48   
> FINODELK
>       0.00      79.50 us      33.00 us     119.00 us             58      
> FSTAT
>       0.00      74.51 us       3.00 us     133.00 us             92    
> OPENDIR
>       0.01      64.51 us      25.00 us     410.00 us            121     
> STATFS
>       0.01     176.33 us     132.00 us     233.00 us             48   
> FXATTROP
>       0.01     169.34 us      40.00 us     416.00 us             56   
> READDIRP
>       0.01     103.52 us      69.00 us     147.00 us            173
> REMOVEXATTR
>       0.01     124.14 us      83.00 us     185.00 us            155   
> SETXATTR
>       0.02     121.36 us      31.00 us     189.00 us            173    
> SETATTR
>       0.02     899.67 us     747.00 us    1118.00 us             24      
> WRITE
>       0.02      68.47 us      24.00 us    2359.00 us            435    
> INODELK
>       0.02     175.49 us      21.00 us     270.00 us            185   
> GETXATTR
>       0.03     210.42 us     154.00 us     281.00 us            181    
> XATTROP
>       0.17     165.59 us      24.00 us     580.00 us           1451     
> LOOKUP
>       0.67      78.49 us      38.00 us     442.00 us          11981       
> STAT
>      98.99   11416.08 us     130.00 us  218011.00 us          12088       
> READ
> 
>     Duration: 169 seconds
>    Data Read: 106749950 bytes
> Data Written: 12288 bytes
> 
> Lookup sent over fuse mount : 1138
> Lookup sent over brick : 1451
> 
> Any idea why this was more first time? I ran same workload which is 100%
> random read.

Assuming the first run was following a fresh mount of the volume, my guess is that shard has looked up some shards before reading from them. Earlier it used to be sending MKNODs before LOOKUP. But ever since we found the add-brick + vm corruption issue, we had to change the order in this patch - https://review.gluster.org/17010

Note that this is a one-time lookup. Upon first lookup, shard xl remembers the shard-to-inode mapping in memory. Subsequent reads/writes in that region won't require this lookup.

-Krutika
Comment 9 RamaKasturi 2017-06-05 10:31:53 UTC 
Thanks krutika. Based on my observations and krutikas input on why lookups were more first time marking this bug to verified state as everything works as expected.
Comment 11 errata-xmlrpc 2017-09-21 04:35:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774