Bug 1399105 - possible memory leak on client when writing to a file while another client issues a truncate
Summary: possible memory leak on client when writing to a file while another client i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: disperse
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: RHGS 3.2.0
Assignee: Ashish Pandey
QA Contact: nchilaka
URL:
Whiteboard:
Depends On:
Blocks: 1351528 1400818 1400833
TreeView+ depends on / blocked
 
Reported: 2016-11-28 09:59 UTC by nchilaka
Modified: 2017-03-23 05:51 UTC (History)
4 users (show)

Fixed In Version: glusterfs-3.8.4-7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1400818 (view as bug list)
Environment:
Last Closed: 2017-03-23 05:51:50 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description nchilaka 2016-11-28 09:59:36 UTC
Description of problem:
===========================
on an ec volume 2x(4+2), I mounted the volume on two fuse clients
From one client i wrote a file using dd of size 1GB, Now while this action started I did a truncate --size 2Gb from the second client.
I observed the resident memory of the fuse process increases and doesn't get released.
Also, the free -h show increase in used up memory significantly

Whereas if we just let the file create with dd, the res memory increases minimally which is acceptable.

Version-Release number of selected component (if applicable):
========================
3.8.4-5


How reproducible:
===============
consistently

Steps to Reproduce:
1.create a 2x(4+2) ec volume
2.mount on two fuse clients
3.keep tracking the top and free comamnd for the fuse mount(where you will be doing dd)
4. from one client dd a file of size 1Gb and as soon as this starts, from 2nd client issue a truncate of size 2GB

5. This was noticed consistently that the resident memory increases and doesn't get released at all

Comment 2 nchilaka 2016-11-28 10:00:22 UTC
root@dhcp35-37 ~]# gluster v status
Status of volume: disperse
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.37:/rhs/brick2/disperse      49153     0          Y       32685
Brick 10.70.35.116:/rhs/brick2/disperse     49152     0          Y       13804
Brick 10.70.35.239:/rhs/brick2/disperse     49152     0          Y       6677 
Brick 10.70.35.135:/rhs/brick2/disperse     49153     0          Y       6698 
Brick 10.70.35.8:/rhs/brick2/disperse       49153     0          Y       6617 
Brick 10.70.35.196:/rhs/brick2/disperse     49153     0          Y       6700 
Brick 10.70.35.37:/rhs/brick3/disperse      49154     0          Y       32704
Brick 10.70.35.116:/rhs/brick3/disperse     49154     0          Y       13823
Brick 10.70.35.239:/rhs/brick3/disperse     49154     0          Y       6696 
Brick 10.70.35.135:/rhs/brick3/disperse     49154     0          Y       6717 
Brick 10.70.35.8:/rhs/brick3/disperse       49154     0          Y       6636 
Brick 10.70.35.196:/rhs/brick3/disperse     49154     0          Y       6719 
Self-heal Daemon on localhost               N/A       N/A        Y       32724
Self-heal Daemon on 10.70.35.239            N/A       N/A        Y       6716 
Self-heal Daemon on 10.70.35.8              N/A       N/A        Y       6656 
Self-heal Daemon on 10.70.35.196            N/A       N/A        Y       6742 
Self-heal Daemon on 10.70.35.135            N/A       N/A        Y       6737 
Self-heal Daemon on 10.70.35.116            N/A       N/A        Y       13843
 
Task Status of Volume disperse
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : e401b7fd-1ef6-4d1f-a818-228cba36a128
Status               : completed           
 
[root@dhcp35-37 ~]# gluster v info disperse
 
Volume Name: disperse
Type: Distributed-Disperse
Volume ID: 9620f72c-f8bb-4e05-98e4-d1a2132db8a7
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.35.37:/rhs/brick2/disperse
Brick2: 10.70.35.116:/rhs/brick2/disperse
Brick3: 10.70.35.239:/rhs/brick2/disperse
Brick4: 10.70.35.135:/rhs/brick2/disperse
Brick5: 10.70.35.8:/rhs/brick2/disperse
Brick6: 10.70.35.196:/rhs/brick2/disperse
Brick7: 10.70.35.37:/rhs/brick3/disperse
Brick8: 10.70.35.116:/rhs/brick3/disperse
Brick9: 10.70.35.239:/rhs/brick3/disperse
Brick10: 10.70.35.135:/rhs/brick3/disperse
Brick11: 10.70.35.8:/rhs/brick3/disperse
Brick12: 10.70.35.196:/rhs/brick3/disperse
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
[root@dhcp35-37 ~]#

Comment 5 nchilaka 2016-11-28 11:02:13 UTC
Starting volume profile on disperse has been successful 
[root@dhcp35-37 ~]# gluster v profile disperse info
Brick: 10.70.35.37:/rhs/brick3/disperse
---------------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               901380                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.50      97.00 us      97.00 us      97.00 us              1     OPENDIR
      1.53      99.00 us      99.00 us      99.00 us              1    GETXATTR
      7.77     252.00 us      43.00 us     461.00 us              2     READDIR
     34.84     150.67 us      76.00 us     600.00 us             15      LOOKUP
     54.37     117.57 us      23.00 us     835.00 us             30     INODELK
 
    Duration: 15820 seconds
   Data Read: 461506560 bytes
Data Written: 3447035392 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               901380                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.50      97.00 us      97.00 us      97.00 us              1     OPENDIR
      1.53      99.00 us      99.00 us      99.00 us              1    GETXATTR
      7.77     252.00 us      43.00 us     461.00 us              2     READDIR
     34.84     150.67 us      76.00 us     600.00 us             15      LOOKUP
     54.37     117.57 us      23.00 us     835.00 us             30     INODELK
 
    Duration: 15820 seconds
   Data Read: 461506560 bytes
Data Written: 3447035392 bytes
 
Brick: 10.70.35.37:/rhs/brick2/disperse
---------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2064583                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                117870 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1951  RELEASEDIR
      1.74      51.00 us      29.00 us      73.00 us              2     READDIR
      7.63     448.00 us     448.00 us     448.00 us              1     OPENDIR
      8.31     488.00 us     488.00 us     488.00 us              1    GETXATTR
     35.12     137.47 us      67.00 us     295.00 us             15      LOOKUP
     47.21      92.40 us      23.00 us     695.00 us             30     INODELK
 
    Duration: 15820 seconds
   Data Read: 4919430656 bytes
Data Written: 6718739968 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2064583                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                117870 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1951  RELEASEDIR
      1.74      51.00 us      29.00 us      73.00 us              2     READDIR
      7.63     448.00 us     448.00 us     448.00 us              1     OPENDIR
      8.31     488.00 us     488.00 us     488.00 us              1    GETXATTR
     35.12     137.47 us      67.00 us     295.00 us             15      LOOKUP
     47.21      92.40 us      23.00 us     695.00 us             30     INODELK
 
    Duration: 15820 seconds
   Data Read: 4919430656 bytes
Data Written: 6718739968 bytes
 
Brick: 10.70.35.116:/rhs/brick2/disperse
----------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2070328                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                116906 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1951  RELEASEDIR
      2.06      84.00 us      84.00 us      84.00 us              1     OPENDIR
      3.60      73.50 us      67.00 us      80.00 us              2     READDIR
      4.38     179.00 us     179.00 us     179.00 us              1    GETXATTR
     42.42      57.77 us      39.00 us      97.00 us             30     INODELK
     47.54     129.47 us      85.00 us     227.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4890783744 bytes
Data Written: 6718739968 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2070328                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                116906 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1951  RELEASEDIR
      2.06      84.00 us      84.00 us      84.00 us              1     OPENDIR
      3.60      73.50 us      67.00 us      80.00 us              2     READDIR
      4.38     179.00 us     179.00 us     179.00 us              1    GETXATTR
     42.42      57.77 us      39.00 us      97.00 us             30     INODELK
     47.54     129.47 us      85.00 us     227.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4890783744 bytes
Data Written: 6718739968 bytes
 
Brick: 10.70.35.116:/rhs/brick3/disperse
----------------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               902198                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.62      30.50 us      27.00 us      34.00 us              2     READDIR
      1.73      65.00 us      65.00 us      65.00 us              1     OPENDIR
      3.62     136.00 us     136.00 us     136.00 us              1    GETXATTR
     42.87     107.40 us      74.00 us     156.00 us             15      LOOKUP
     50.16      62.83 us      28.00 us     291.00 us             30     INODELK
 
    Duration: 15819 seconds
   Data Read: 461925376 bytes
Data Written: 3447035392 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               902198                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.62      30.50 us      27.00 us      34.00 us              2     READDIR
      1.73      65.00 us      65.00 us      65.00 us              1     OPENDIR
      3.62     136.00 us     136.00 us     136.00 us              1    GETXATTR
     42.87     107.40 us      74.00 us     156.00 us             15      LOOKUP
     50.16      62.83 us      28.00 us     291.00 us             30     INODELK
 
    Duration: 15819 seconds
   Data Read: 461925376 bytes
Data Written: 3447035392 bytes
 
Brick: 10.70.35.239:/rhs/brick3/disperse
----------------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               902906                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.82      67.00 us      67.00 us      67.00 us              1     OPENDIR
      3.08      56.50 us      40.00 us      73.00 us              2     READDIR
      3.76     138.00 us     138.00 us     138.00 us              1    GETXATTR
     45.06      55.17 us      22.00 us     107.00 us             30     INODELK
     46.28     113.33 us      75.00 us     180.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 462287872 bytes
Data Written: 3447035392 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               902906                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.82      67.00 us      67.00 us      67.00 us              1     OPENDIR
      3.08      56.50 us      40.00 us      73.00 us              2     READDIR
      3.76     138.00 us     138.00 us     138.00 us              1    GETXATTR
     45.06      55.17 us      22.00 us     107.00 us             30     INODELK
     46.28     113.33 us      75.00 us     180.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 462287872 bytes
Data Written: 3447035392 bytes
 
Brick: 10.70.35.239:/rhs/brick2/disperse
----------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2062666                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                117681 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1950  RELEASEDIR
      2.00      38.50 us      36.00 us      41.00 us              2     READDIR
      2.23      86.00 us      86.00 us      86.00 us              1     OPENDIR
      4.51     174.00 us     174.00 us     174.00 us              1    GETXATTR
     44.27      56.87 us      22.00 us     162.00 us             30     INODELK
     46.99     120.73 us      60.00 us     234.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4912256000 bytes
Data Written: 6718739968 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2062666                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                117681 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1950  RELEASEDIR
      2.00      38.50 us      36.00 us      41.00 us              2     READDIR
      2.23      86.00 us      86.00 us      86.00 us              1     OPENDIR
      4.51     174.00 us     174.00 us     174.00 us              1    GETXATTR
     44.27      56.87 us      22.00 us     162.00 us             30     INODELK
     46.99     120.73 us      60.00 us     234.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4912256000 bytes
Data Written: 6718739968 bytes
 
Brick: 10.70.35.135:/rhs/brick3/disperse
----------------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               902157                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.63      69.00 us      69.00 us      69.00 us              1     OPENDIR
      1.89      40.00 us      26.00 us      54.00 us              2     READDIR
      5.69     241.00 us     241.00 us     241.00 us              1    GETXATTR
     41.60      58.77 us      21.00 us     287.00 us             30     INODELK
     49.20     139.00 us      78.00 us     276.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 461904384 bytes
Data Written: 3447035392 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               902157                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.63      69.00 us      69.00 us      69.00 us              1     OPENDIR
      1.89      40.00 us      26.00 us      54.00 us              2     READDIR
      5.69     241.00 us     241.00 us     241.00 us              1    GETXATTR
     41.60      58.77 us      21.00 us     287.00 us             30     INODELK
     49.20     139.00 us      78.00 us     276.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 461904384 bytes
Data Written: 3447035392 bytes
 
Brick: 10.70.35.135:/rhs/brick2/disperse
----------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2065363                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                116863 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1951  RELEASEDIR
      1.93      36.00 us      25.00 us      47.00 us              2     READDIR
      2.07      77.00 us      77.00 us      77.00 us              1     OPENDIR
      5.64     210.00 us     210.00 us     210.00 us              1    GETXATTR
     41.32      51.27 us      22.00 us     254.00 us             30     INODELK
     49.03     121.67 us      60.00 us     386.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4886832640 bytes
Data Written: 6718739968 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2065363                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                116863 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1951  RELEASEDIR
      1.93      36.00 us      25.00 us      47.00 us              2     READDIR
      2.07      77.00 us      77.00 us      77.00 us              1     OPENDIR
      5.64     210.00 us     210.00 us     210.00 us              1    GETXATTR
     41.32      51.27 us      22.00 us     254.00 us             30     INODELK
     49.03     121.67 us      60.00 us     386.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4886832640 bytes
Data Written: 6718739968 bytes
 
Brick: 10.70.35.8:/rhs/brick2/disperse
--------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2061857                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                116019 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1951  RELEASEDIR
      2.52     111.00 us     111.00 us     111.00 us              1     OPENDIR
      3.57      78.50 us      73.00 us      84.00 us              2     READDIR
      7.43     327.00 us     327.00 us     327.00 us              1    GETXATTR
     42.72      62.70 us      23.00 us     172.00 us             30     INODELK
     43.77     128.47 us      60.00 us     274.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4857381376 bytes
Data Written: 6718739968 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2061857                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                116019 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1951  RELEASEDIR
      2.52     111.00 us     111.00 us     111.00 us              1     OPENDIR
      3.57      78.50 us      73.00 us      84.00 us              2     READDIR
      7.43     327.00 us     327.00 us     327.00 us              1    GETXATTR
     42.72      62.70 us      23.00 us     172.00 us             30     INODELK
     43.77     128.47 us      60.00 us     274.00 us             15      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4857381376 bytes
Data Written: 6718739968 bytes
 
Brick: 10.70.35.8:/rhs/brick3/disperse
--------------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               900576                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.45      28.00 us      26.00 us      30.00 us              2     READDIR
      1.66      64.00 us      64.00 us      64.00 us              1     OPENDIR
      3.80     147.00 us     147.00 us     147.00 us              1    GETXATTR
     45.51      58.67 us      22.00 us     176.00 us             30     INODELK
     47.58     122.67 us      67.00 us     249.00 us             15      LOOKUP
 
    Duration: 15818 seconds
   Data Read: 461094912 bytes
Data Written: 3447035392 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               900576                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1957  RELEASEDIR
      1.45      28.00 us      26.00 us      30.00 us              2     READDIR
      1.66      64.00 us      64.00 us      64.00 us              1     OPENDIR
      3.80     147.00 us     147.00 us     147.00 us              1    GETXATTR
     45.51      58.67 us      22.00 us     176.00 us             30     INODELK
     47.58     122.67 us      67.00 us     249.00 us             15      LOOKUP
 
    Duration: 15818 seconds
   Data Read: 461094912 bytes
Data Written: 3447035392 bytes
 
Brick: 10.70.35.196:/rhs/brick2/disperse
----------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2053415                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                117801 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1958  RELEASEDIR
      1.66      39.50 us       9.00 us      70.00 us              2     OPENDIR
      9.25     147.00 us      52.00 us     246.00 us              3    GETXATTR
     17.71     211.00 us      94.00 us     389.00 us              4     READDIR
     33.88      53.83 us      25.00 us     150.00 us             30     INODELK
     37.51     111.75 us      61.00 us     258.00 us             16      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4911451648 bytes
Data Written: 6718739968 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                 512b+                1024b+ 
 No. of Reads:                    0               2053415                     0 
No. of Writes:                    3                 99280               2270930 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:              1699103                  6563                  5390 
 
   Block Size:              16384b+               32768b+ 
 No. of Reads:                    0                117801 
No. of Writes:                 2437                  1291 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             57      FORGET
      0.00       0.00 us       0.00 us       0.00 us           2144     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1958  RELEASEDIR
      1.66      39.50 us       9.00 us      70.00 us              2     OPENDIR
      9.25     147.00 us      52.00 us     246.00 us              3    GETXATTR
     17.71     211.00 us      94.00 us     389.00 us              4     READDIR
     33.88      53.83 us      25.00 us     150.00 us             30     INODELK
     37.51     111.75 us      61.00 us     258.00 us             16      LOOKUP
 
    Duration: 15819 seconds
   Data Read: 4911451648 bytes
Data Written: 6718739968 bytes
 
Brick: 10.70.35.196:/rhs/brick3/disperse
----------------------------------------
Cumulative Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               900507                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1964  RELEASEDIR
      1.52      35.00 us       2.00 us      68.00 us              2     OPENDIR
      7.44     114.33 us      63.00 us     142.00 us              3    GETXATTR
     15.80     182.25 us      38.00 us     426.00 us              4     READDIR
     35.46      54.53 us      29.00 us     141.00 us             30     INODELK
     39.78     114.69 us      81.00 us     198.00 us             16      LOOKUP
 
    Duration: 15818 seconds
   Data Read: 461059584 bytes
Data Written: 3447035392 bytes
 
Interval 0 Stats:
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:               900507                     0                     0 
No. of Writes:                70203               1693147                573317 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3144                  2175                   656 
 
   Block Size:              32768b+ 
 No. of Reads:                    0 
No. of Writes:                  262 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             62      FORGET
      0.00       0.00 us       0.00 us       0.00 us           1175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1964  RELEASEDIR
      1.52      35.00 us       2.00 us      68.00 us              2     OPENDIR
      7.44     114.33 us      63.00 us     142.00 us              3    GETXATTR
     15.80     182.25 us      38.00 us     426.00 us              4     READDIR
     35.46      54.53 us      29.00 us     141.00 us             30     INODELK
     39.78     114.69 us      81.00 us     198.00 us             16      LOOKUP
 
    Duration: 15818 seconds
   Data Read: 461059584 bytes
Data Written: 3447035392 bytes

Comment 9 Atin Mukherjee 2016-12-02 12:22:28 UTC
upstream mainline patch http://review.gluster.org/16006 posted for review.

Comment 10 Atin Mukherjee 2016-12-02 12:24:43 UTC
(In reply to Atin Mukherjee from comment #9)
> upstream mainline patch http://review.gluster.org/16006 posted for review.

Correction, this is a 3.9 patch, mainline patch is http://review.gluster.org/#/c/16007

Comment 11 Atin Mukherjee 2016-12-04 05:03:10 UTC
downstream patch : https://code.engineering.redhat.com/gerrit/92048

Comment 13 nchilaka 2016-12-20 11:02:00 UTC
on_qa validation
ran the below case and found the memory consumption constant. Hence marking as passed.
validated on 3.8.4-8

1.create a 2x(4+2) ec volume
2.mount on two fuse clients
3.keep tracking the top and free comamnd for the fuse mount(where you will be doing dd)
4. from one client dd a file of size 2Gb and as soon as this starts, from 2nd client issue a truncate of size 5GB


I didn't notice any spike
client1 where dd was happening:
[root@rhs-client45 ~]# for i in {1..10000};do top -n 1 -b|egrep "RES|6967";sleep 30;done
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  51384   3136 S   0.0  0.3   0:00.07 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  51384   3136 S   0.0  0.3   0:00.07 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  51824   3136 S   0.0  0.3   0:00.07 gluster+
==========>started dd
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  57104   3824 S  93.8  0.4   0:28.92 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59232   3840 S 106.2  0.4   0:58.07 gluster+
====>  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59232   3840 S  93.8  0.4   1:27.33 gluster+
======>started trucate
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59232   3840 S  93.8  0.4   1:53.77 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59236   3840 S 100.0  0.4   2:23.98 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59244   3840 S 100.0  0.4   2:53.03 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59244   3840 S 106.2  0.4   3:22.03 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59244   3840 S 106.2  0.4   3:50.41 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59244   3840 S  93.8  0.4   4:19.49 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59244   3840 S  93.8  0.4   4:47.89 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  649480  59244   3840 S   0.0  0.4   5:09.92 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  848144  60228   3904 S  68.8  0.4   5:17.60 gluster+
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6967 root      20   0  848144  60248   3908 S  43.8  0.4   5:34.62 gluster+
=======================================


client2 where truncate was issued
[root@rhs-client24 ~]# for i in {1..10000};do top -n 1 -b|egrep "RES|6428";sleep 30;done
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  638m  46m 3220 S  0.0  0.3   0:00.12 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  638m  47m 3220 S  0.0  0.3   0:00.13 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  638m  47m 3220 S  0.0  0.3   0:00.13 glusterfs       
==========>started dd
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  48m 3484 S  0.0  0.3   0:00.16 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  48m 3484 S  0.0  0.3   0:00.16 glusterfs       
======>started trucate
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.17 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.18 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.18 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.18 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.18 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.18 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.18 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.18 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  50m 3760 S  0.0  0.3   0:00.20 glusterfs       
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  54m 3920 S 30.8  0.3   0:00.49 glusterfs       
===============================
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
 6428 root      20   0  702m  55m 3920 S 36.5  0.3   0:11.82 glusterfs

Comment 15 errata-xmlrpc 2017-03-23 05:51:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.