Bug 1461286 - [Disperse]poor write performance when copy small files to an EC volume from windows 7 client.
[Disperse]poor write performance when copy small files to an EC volume from w...
Status: CLOSED NOTABUG
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: disperse (Show other bugs)
3.2
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Ashish Pandey
nchilaka
: ZStream
: 1458985 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-14 02:51 EDT by WenhanShi
Modified: 2017-07-18 03:06 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-18 03:06:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description WenhanShi 2017-06-14 02:51:50 EDT
Description of problem:
A poor performance can be observed when copy/create a lot of small files to an EC volume. From profile I saw READDIRP has a large Avg-latency which is 30-40%.


Version-Release number of selected component (if applicable):
gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64     
gluster-nagios-common-0.2.4-1.el7rhgs.noarch     
glusterfs-3.8.4-18.el7rhgs.x86_64                
glusterfs-api-3.8.4-18.el7rhgs.x86_64            
glusterfs-cli-3.8.4-18.el7rhgs.x86_64            
glusterfs-client-xlators-3.8.4-18.el7rhgs.x86_64 
glusterfs-fuse-3.8.4-18.el7rhgs.x86_64           
glusterfs-ganesha-3.8.4-18.el7rhgs.x86_64        
glusterfs-geo-replication-3.8.4-18.el7rhgs.x86_64
glusterfs-libs-3.8.4-18.el7rhgs.x86_64           
glusterfs-rdma-3.8.4-18.el7rhgs.x86_64           
glusterfs-server-3.8.4-18.el7rhgs.x86_64         
nfs-ganesha-gluster-2.4.1-9.el7rhgs.x86_64       
python-gluster-3.8.4-18.el7rhgs.noarch           
samba-vfs-glusterfs-4.4.6-5.el7rhgs.x86_64       
vdsm-gluster-4.17.33-1.1.el7rhgs.noarch          


How reproducible:
1. every time

Steps to Reproduce:
1. Create a 4+2 EC volume on 3 gluster nodes.
2. mount the volume by FUSE and export the mount point to samba
3. access the volume from windows client, and copy 60000 files(7G totally ) to the volume by xcopy command.

Actual results:
will cost 40+ minutes 

Expected results:
want it to get faster.

Additional info:

# gluster volume info dispersed1

Volume Name: dispersed1
Type: Disperse
Volume ID: 2af122a6-1549-4b38-90ab-06dbae284e45
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: 10.0.0.71:/gluster/brick1
Brick2: 10.0.0.71:/gluster/brick2
Brick3: 10.0.0.72:/gluster/brick1
Brick4: 10.0.0.72:/gluster/brick2
Brick5: 10.0.0.73:/gluster/brick1
Brick6: 10.0.0.73:/gluster/brick2
Options Reconfigured:
performance.read-ahead: off
client.event-threads: 4
performance.client-io-threads: on
cluster.lookup-optimize: on
cluster.readdir-optimize: on
network.inode-lru-limit: 100000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
storage.batch-fsync-delay-usec: 0
performance.cache-samba-metadata: on
server.allow-insecure: on
performance.stat-prefetch: on
performance.readdir-ahead: off
nfs.disable: on

[wenshi@collab-shell x-text]$ cat vol_profile_0614-3.txt 
Brick: 10.0.0.71:/gluster/brick2
--------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1469                   985                  4610 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  710                   577                   727 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  242                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54807      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117077  RELEASEDIR
      0.00     209.00 us     209.00 us     209.00 us              1      RENAME
      0.00      71.00 us      43.00 us     127.00 us              6        OPEN
      0.04     393.07 us     145.00 us    1104.00 us            288     READDIR
      0.04      56.57 us      16.00 us     257.00 us           2452      STATFS
      0.23      76.24 us      20.00 us     164.00 us           9319        READ
      0.77      40.53 us       7.00 us     135.00 us          60090       FLUSH
      1.30    2293.09 us     123.00 us  926327.00 us           1776       MKDIR
      1.60      91.02 us      21.00 us     606.00 us          55304     XATTROP
      1.73     104.05 us      36.00 us     462.00 us          52177   FTRUNCATE
      2.07      61.75 us      15.00 us     239.00 us         105165       FSTAT
      2.35      89.41 us      23.00 us     232.00 us          82738     SETATTR
      2.46     114.54 us      27.00 us     582.00 us          67530       WRITE
      3.22      45.21 us       8.00 us    1578.00 us         223709     INODELK
      3.32      89.15 us       1.00 us    2131.00 us         117077     OPENDIR
      3.92     232.73 us     100.00 us  163513.00 us          52982      CREATE
      5.32     102.06 us       9.00 us  655677.00 us         163884    FXATTROP
      5.45     205.62 us      14.00 us  386240.00 us          83347    GETXATTR
      7.19     136.04 us      31.00 us    1222.00 us         166050    SETXATTR
     20.52     100.37 us      11.00 us 1251401.00 us         642623      LOOKUP
     38.49    3647.02 us      18.00 us   71724.00 us          33180    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 45759488 bytes
Data Written: 2036296192 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1469                   985                  4610 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  710                   577                   727 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  242                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54807      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117077  RELEASEDIR
      0.00     209.00 us     209.00 us     209.00 us              1      RENAME
      0.00      71.00 us      43.00 us     127.00 us              6        OPEN
      0.04     393.07 us     145.00 us    1104.00 us            288     READDIR
      0.04      56.57 us      16.00 us     257.00 us           2452      STATFS
      0.23      76.24 us      20.00 us     164.00 us           9319        READ
      0.77      40.53 us       7.00 us     135.00 us          60090       FLUSH
      1.30    2293.09 us     123.00 us  926327.00 us           1776       MKDIR
      1.60      91.02 us      21.00 us     606.00 us          55304     XATTROP
      1.73     104.05 us      36.00 us     462.00 us          52177   FTRUNCATE
      2.07      61.75 us      15.00 us     239.00 us         105165       FSTAT
      2.35      89.41 us      23.00 us     232.00 us          82738     SETATTR
      2.46     114.54 us      27.00 us     582.00 us          67530       WRITE
      3.22      45.21 us       8.00 us    1578.00 us         223709     INODELK
      3.32      89.15 us       1.00 us    2131.00 us         117077     OPENDIR
      3.92     232.73 us     100.00 us  163513.00 us          52982      CREATE
      5.32     102.06 us       9.00 us  655677.00 us         163884    FXATTROP
      5.45     205.62 us      14.00 us  386240.00 us          83347    GETXATTR
      7.19     136.04 us      31.00 us    1222.00 us         166050    SETXATTR
     20.52     100.37 us      11.00 us 1251401.00 us         642623      LOOKUP
     38.49    3647.02 us      18.00 us   71724.00 us          33180    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 45759488 bytes
Data Written: 2036296192 bytes
 
Brick: 10.0.0.71:/gluster/brick1
--------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1484                   976                  4554 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  728                   536                   701 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  238                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54807      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117078  RELEASEDIR
      0.00     164.00 us     164.00 us     164.00 us              1      RENAME
      0.00      75.67 us      50.00 us     104.00 us              6        OPEN
      0.04     408.52 us     144.00 us     829.00 us            290     READDIR
      0.05      55.47 us      17.00 us     249.00 us           2452      STATFS
      0.25      76.35 us      21.00 us     158.00 us           9217        READ
      0.50     781.25 us     126.00 us  826464.00 us           1776       MKDIR
      0.90      41.71 us       7.00 us     167.00 us          60090       FLUSH
      1.86      93.58 us      22.00 us     669.00 us          55304     XATTROP
      2.01     107.30 us      40.00 us     250.00 us          52177   FTRUNCATE
      2.43      64.21 us      16.00 us     234.00 us         105165       FSTAT
      2.79      93.63 us      21.00 us     626.00 us          82738     SETATTR
      2.85     117.08 us      28.00 us    4808.00 us          67530       WRITE
      3.78      89.75 us       1.00 us    2253.00 us         117078     OPENDIR
      3.90      48.41 us       8.00 us    1802.00 us         223708     INODELK
      5.01     262.64 us     102.00 us 1368286.00 us          52982      CREATE
      5.65      95.74 us      10.00 us  502047.00 us         163885    FXATTROP
      6.02     201.00 us      14.00 us  345298.00 us          83217    GETXATTR
      8.29     138.66 us      33.00 us    1078.00 us         166050    SETXATTR
     23.68     102.38 us      12.00 us  390815.00 us         642625      LOOKUP
     30.00    3805.81 us      19.00 us  616226.00 us          21902    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 44471296 bytes
Data Written: 2036296192 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1484                   976                  4554 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  728                   536                   701 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  238                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54807      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117078  RELEASEDIR
      0.00     164.00 us     164.00 us     164.00 us              1      RENAME
      0.00      75.67 us      50.00 us     104.00 us              6        OPEN
      0.04     408.52 us     144.00 us     829.00 us            290     READDIR
      0.05      55.47 us      17.00 us     249.00 us           2452      STATFS
      0.25      76.35 us      21.00 us     158.00 us           9217        READ
      0.50     781.25 us     126.00 us  826464.00 us           1776       MKDIR
      0.90      41.71 us       7.00 us     167.00 us          60090       FLUSH
      1.86      93.58 us      22.00 us     669.00 us          55304     XATTROP
      2.01     107.30 us      40.00 us     250.00 us          52177   FTRUNCATE
      2.43      64.21 us      16.00 us     234.00 us         105165       FSTAT
      2.79      93.63 us      21.00 us     626.00 us          82738     SETATTR
      2.85     117.08 us      28.00 us    4808.00 us          67530       WRITE
      3.78      89.75 us       1.00 us    2253.00 us         117078     OPENDIR
      3.90      48.41 us       8.00 us    1802.00 us         223708     INODELK
      5.01     262.64 us     102.00 us 1368286.00 us          52982      CREATE
      5.65      95.74 us      10.00 us  502047.00 us         163885    FXATTROP
      6.02     201.00 us      14.00 us  345298.00 us          83217    GETXATTR
      8.29     138.66 us      33.00 us    1078.00 us         166050    SETXATTR
     23.68     102.38 us      12.00 us  390815.00 us         642625      LOOKUP
     30.00    3805.81 us      19.00 us  616226.00 us          21902    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 44471296 bytes
Data Written: 2036296192 bytes
 
Brick: 10.0.0.72:/gluster/brick1
--------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1480                   976                  4648 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  704                   599                   747 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  251                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54810      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117080  RELEASEDIR
      0.00     188.00 us     188.00 us     188.00 us              1      RENAME
      0.00      92.67 us      74.00 us     143.00 us              6        OPEN
      0.03     399.89 us     145.00 us     851.00 us            294     READDIR
      0.04      58.29 us      16.00 us     154.00 us           2452      STATFS
      0.21      76.92 us      22.00 us     168.00 us           9404        READ
      0.35     667.04 us     130.00 us  327543.00 us           1776       MKDIR
      0.76      42.30 us       8.00 us     157.00 us          60090       FLUSH
      1.51      91.79 us      19.00 us     333.00 us          55304     XATTROP
      1.66     107.19 us      38.00 us     318.00 us          52177   FTRUNCATE
      2.05      65.53 us      16.00 us     346.00 us         105165       FSTAT
      2.31      93.82 us      22.00 us     317.00 us          82738     SETATTR
      2.39     119.04 us      26.00 us    1006.00 us          67530       WRITE
      3.21      92.13 us       1.00 us    1733.00 us         117080     OPENDIR
      3.28      49.34 us       9.00 us    1315.00 us         223710     INODELK
      4.20     266.79 us     102.00 us  680697.00 us          52982      CREATE
      4.90     198.24 us      14.00 us  379080.00 us          83089    GETXATTR
      4.93     101.29 us       9.00 us 1004809.00 us         163885    FXATTROP
      6.79     137.65 us      33.00 us    1025.00 us         166050    SETXATTR
     19.65     102.88 us      17.00 us  836278.00 us         642627      LOOKUP
     41.73    3442.23 us      19.00 us  308537.00 us          40788    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 46831616 bytes
Data Written: 2036296192 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1480                   976                  4648 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  704                   599                   747 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  251                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54810      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117080  RELEASEDIR
      0.00     188.00 us     188.00 us     188.00 us              1      RENAME
      0.00      92.67 us      74.00 us     143.00 us              6        OPEN
      0.03     399.89 us     145.00 us     851.00 us            294     READDIR
      0.04      58.29 us      16.00 us     154.00 us           2452      STATFS
      0.21      76.92 us      22.00 us     168.00 us           9404        READ
      0.35     667.04 us     130.00 us  327543.00 us           1776       MKDIR
      0.76      42.30 us       8.00 us     157.00 us          60090       FLUSH
      1.51      91.79 us      19.00 us     333.00 us          55304     XATTROP
      1.66     107.19 us      38.00 us     318.00 us          52177   FTRUNCATE
      2.05      65.53 us      16.00 us     346.00 us         105165       FSTAT
      2.31      93.82 us      22.00 us     317.00 us          82738     SETATTR
      2.39     119.04 us      26.00 us    1006.00 us          67530       WRITE
      3.21      92.13 us       1.00 us    1733.00 us         117080     OPENDIR
      3.28      49.34 us       9.00 us    1315.00 us         223710     INODELK
      4.20     266.79 us     102.00 us  680697.00 us          52982      CREATE
      4.90     198.24 us      14.00 us  379080.00 us          83089    GETXATTR
      4.93     101.29 us       9.00 us 1004809.00 us         163885    FXATTROP
      6.79     137.65 us      33.00 us    1025.00 us         166050    SETXATTR
     19.65     102.88 us      17.00 us  836278.00 us         642627      LOOKUP
     41.73    3442.23 us      19.00 us  308537.00 us          40788    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 46831616 bytes
Data Written: 2036296192 bytes
 
Brick: 10.0.0.72:/gluster/brick2
--------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1396                   889                  4500 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  693                   599                   745 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  229                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54805      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117079  RELEASEDIR
      0.00     177.00 us     177.00 us     177.00 us              1      RENAME
      0.00      94.83 us      81.00 us     114.00 us              6        OPEN
      0.04     362.67 us     122.00 us     805.00 us            292     READDIR
      0.05      52.47 us      15.00 us     155.00 us           2452      STATFS
      0.23      73.00 us      18.00 us     526.00 us           9051        READ
      0.85    1361.63 us     123.00 us  837548.00 us           1776       MKDIR
      0.85      40.53 us       8.00 us     158.00 us          60090       FLUSH
      1.73      89.38 us      22.00 us     570.00 us          55304     XATTROP
      1.90     103.92 us      38.00 us     365.00 us          52177   FTRUNCATE
      2.28      61.97 us      15.00 us     162.00 us         105165       FSTAT
      2.60      89.48 us      25.00 us     551.00 us          82738     SETATTR
      2.77     116.88 us      27.00 us    1363.00 us          67530       WRITE
      3.61      87.98 us       1.00 us    1588.00 us         117079     OPENDIR
      3.76      47.91 us       8.00 us    1356.00 us         223708     INODELK
      4.53     243.82 us     102.00 us  447399.00 us          52982      CREATE
      5.31      92.38 us       9.00 us  487535.00 us         163884    FXATTROP
      5.54     190.14 us      12.00 us  524210.00 us          83142    GETXATTR
      7.82     134.42 us      35.00 us     705.00 us         166050    SETXATTR
     22.01      97.68 us      13.00 us   91750.00 us         642629      LOOKUP
     34.13    3531.25 us      19.00 us  626560.00 us          27568    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 45528064 bytes
Data Written: 2036296192 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1396                   889                  4500 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  693                   599                   745 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  229                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54805      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117079  RELEASEDIR
      0.00     177.00 us     177.00 us     177.00 us              1      RENAME
      0.00      94.83 us      81.00 us     114.00 us              6        OPEN
      0.04     362.67 us     122.00 us     805.00 us            292     READDIR
      0.05      52.47 us      15.00 us     155.00 us           2452      STATFS
      0.23      73.00 us      18.00 us     526.00 us           9051        READ
      0.85    1361.63 us     123.00 us  837548.00 us           1776       MKDIR
      0.85      40.53 us       8.00 us     158.00 us          60090       FLUSH
      1.73      89.38 us      22.00 us     570.00 us          55304     XATTROP
      1.90     103.92 us      38.00 us     365.00 us          52177   FTRUNCATE
      2.28      61.97 us      15.00 us     162.00 us         105165       FSTAT
      2.60      89.48 us      25.00 us     551.00 us          82738     SETATTR
      2.77     116.88 us      27.00 us    1363.00 us          67530       WRITE
      3.61      87.98 us       1.00 us    1588.00 us         117079     OPENDIR
      3.76      47.91 us       8.00 us    1356.00 us         223708     INODELK
      4.53     243.82 us     102.00 us  447399.00 us          52982      CREATE
      5.31      92.38 us       9.00 us  487535.00 us         163884    FXATTROP
      5.54     190.14 us      12.00 us  524210.00 us          83142    GETXATTR
      7.82     134.42 us      35.00 us     705.00 us         166050    SETXATTR
     22.01      97.68 us      13.00 us   91750.00 us         642628      LOOKUP
     34.13    3531.25 us      19.00 us  626560.00 us          27568    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 45528064 bytes
Data Written: 2036296192 bytes
 
Brick: 10.0.0.73:/gluster/brick1
--------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1322                   826                  4242 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  704                   549                   696 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  217                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54806      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117078  RELEASEDIR
      0.00     267.00 us     267.00 us     267.00 us              1      RENAME
      0.00      90.17 us      49.00 us     115.00 us              6        OPEN
      0.04     317.28 us     137.00 us     600.00 us            290     READDIR
      0.05      43.56 us      18.00 us     129.00 us           2452      STATFS
      0.23      62.12 us      20.00 us     125.00 us           8556        READ
      0.85      32.56 us       8.00 us     152.00 us          60090       FLUSH
      1.06    1378.58 us     127.00 us  635595.00 us           1776       MKDIR
      1.76      73.61 us      20.00 us     258.00 us          55304     XATTROP
      2.01      89.21 us      40.00 us     218.00 us          52177   FTRUNCATE
      2.33      51.32 us      14.00 us     145.00 us         105165       FSTAT
      2.68      91.75 us      27.00 us     698.00 us          67530       WRITE
      2.74      76.75 us      26.00 us     552.00 us          82738     SETATTR
      3.55      36.77 us       9.00 us    1323.00 us         223710     INODELK
      3.57      70.59 us       1.00 us    1679.00 us         117078     OPENDIR
      4.72     206.06 us     106.00 us  377976.00 us          52982      CREATE
      5.19      73.35 us      10.00 us  402215.00 us         163885    FXATTROP
      6.18     172.41 us      14.00 us  395059.00 us          83023    GETXATTR
      7.77     108.40 us      32.00 us     855.00 us         166050    SETXATTR
     23.34      84.07 us      14.00 us  323796.00 us         642625      LOOKUP
     31.94    3798.80 us      20.00 us   72906.00 us          19464    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 42773504 bytes
Data Written: 2036296192 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1322                   826                  4242 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  704                   549                   696 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  217                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54806      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117078  RELEASEDIR
      0.00     267.00 us     267.00 us     267.00 us              1      RENAME
      0.00      90.17 us      49.00 us     115.00 us              6        OPEN
      0.04     317.28 us     137.00 us     600.00 us            290     READDIR
      0.05      43.56 us      18.00 us     129.00 us           2452      STATFS
      0.23      62.12 us      20.00 us     125.00 us           8556        READ
      0.85      32.56 us       8.00 us     152.00 us          60090       FLUSH
      1.06    1378.58 us     127.00 us  635595.00 us           1776       MKDIR
      1.76      73.61 us      20.00 us     258.00 us          55304     XATTROP
      2.01      89.21 us      40.00 us     218.00 us          52177   FTRUNCATE
      2.33      51.32 us      14.00 us     145.00 us         105165       FSTAT
      2.68      91.75 us      27.00 us     698.00 us          67530       WRITE
      2.74      76.75 us      26.00 us     552.00 us          82738     SETATTR
      3.55      36.77 us       9.00 us    1323.00 us         223710     INODELK
      3.57      70.59 us       1.00 us    1679.00 us         117078     OPENDIR
      4.72     206.06 us     106.00 us  377976.00 us          52982      CREATE
      5.19      73.35 us      10.00 us  402215.00 us         163885    FXATTROP
      6.18     172.41 us      14.00 us  395059.00 us          83023    GETXATTR
      7.77     108.40 us      32.00 us     855.00 us         166050    SETXATTR
     23.34      84.07 us      14.00 us  323796.00 us         642625      LOOKUP
     31.94    3798.80 us      20.00 us   72906.00 us          19464    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 42773504 bytes
Data Written: 2036296192 bytes
 
Brick: 10.0.0.73:/gluster/brick2
--------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1421                   904                  4334 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  733                   508                   672 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  235                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54810      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117079  RELEASEDIR
      0.00     257.00 us     257.00 us     257.00 us              1      RENAME
      0.00      88.50 us      56.00 us     126.00 us              6        OPEN
      0.04     312.10 us     117.00 us     637.00 us            292     READDIR
      0.04      41.65 us      14.00 us     137.00 us           2452      STATFS
      0.20      59.91 us      22.00 us     125.00 us           8807        READ
      0.64     932.89 us     127.00 us 1089879.00 us           1776       MKDIR
      0.70      30.30 us       8.00 us     114.00 us          60090       FLUSH
      1.47      68.59 us      23.00 us     255.00 us          55304     XATTROP
      1.72      85.47 us      40.00 us     310.00 us          52177   FTRUNCATE
      1.98      48.73 us      16.00 us     193.00 us         105165       FSTAT
      2.26      86.63 us      29.00 us     788.00 us          67530       WRITE
      2.32      72.60 us      26.00 us     201.00 us          82738     SETATTR
      3.06      67.62 us       1.00 us    1242.00 us         117079     OPENDIR
      3.09      35.78 us      10.00 us    1277.00 us         223709     INODELK
      3.94     192.76 us     107.00 us  250262.00 us          52982      CREATE
      4.73      74.73 us      10.00 us  608318.00 us         163885    FXATTROP
      5.62     175.29 us      12.00 us  905368.00 us          83028    GETXATTR
      6.62     103.16 us      35.00 us     792.00 us         166050    SETXATTR
     20.10      80.99 us      12.00 us  150047.00 us         642623      LOOKUP
     41.46    3573.02 us      20.00 us   65494.00 us          30042    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 42788864 bytes
Data Written: 2036296192 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1421                   904                  4334 
No. of Writes:                16535                 10979                 12741 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                  733                   508                   672 
No. of Writes:                 4942                  2889                  3024 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                  235                     0                     0 
No. of Writes:                 4540                  4738                  3815 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                 3327 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          54810      FORGET
      0.00       0.00 us       0.00 us       0.00 us          52988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         117079  RELEASEDIR
      0.00     257.00 us     257.00 us     257.00 us              1      RENAME
      0.00      88.50 us      56.00 us     126.00 us              6        OPEN
      0.04     312.10 us     117.00 us     637.00 us            292     READDIR
      0.04      41.65 us      14.00 us     137.00 us           2452      STATFS
      0.20      59.91 us      22.00 us     125.00 us           8807        READ
      0.64     932.89 us     127.00 us 1089879.00 us           1776       MKDIR
      0.70      30.30 us       8.00 us     114.00 us          60090       FLUSH
      1.47      68.59 us      23.00 us     255.00 us          55304     XATTROP
      1.72      85.47 us      40.00 us     310.00 us          52177   FTRUNCATE
      1.98      48.73 us      16.00 us     193.00 us         105165       FSTAT
      2.26      86.63 us      29.00 us     788.00 us          67530       WRITE
      2.32      72.60 us      26.00 us     201.00 us          82738     SETATTR
      3.06      67.62 us       1.00 us    1242.00 us         117079     OPENDIR
      3.09      35.78 us      10.00 us    1277.00 us         223709     INODELK
      3.94     192.76 us     107.00 us  250262.00 us          52982      CREATE
      4.73      74.73 us      10.00 us  608318.00 us         163885    FXATTROP
      5.62     175.29 us      12.00 us  905368.00 us          83028    GETXATTR
      6.62     103.16 us      35.00 us     792.00 us         166050    SETXATTR
     20.10      80.99 us      12.00 us  150047.00 us         642623      LOOKUP
     41.46    3573.02 us      20.00 us   65494.00 us          30042    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            820      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            785     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             85    CI_XATTR
      0.00       0.00 us       0.00 us       0.00 us             35   CI_FORGET
 
    Duration: 2928 seconds
   Data Read: 42788864 bytes
Data Written: 2036296192 bytes
Comment 7 WenhanShi 2017-06-16 00:05:34 EDT
*** Bug 1458985 has been marked as a duplicate of this bug. ***
Comment 20 Ashish Pandey 2017-07-04 02:11:09 EDT
Hi Wenhan,

We did not see any issue with it RHEl 7.3 when we tried the test on our side.
As per the comment #17 also it looks like issue with window 7 + xcopy

Ashish

Note You need to log in before you can comment on or make changes to this bug.