Bug 1781550

Summary: glusterfs process memory leak in ior test
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: hari gowtham <hgowtham>
Component: read-aheadAssignee: Xavi Hernandez <jahernan>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.5CC: bugs, jahernan, nchilaka, pasik, pprakash, puebele, rhs-bugs, rkothiya, sheggodu, shujun.huang, zz.sh.cynthia
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: RHGS 3.5.z Batch Update 3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-6.0-38 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1779055 Environment:
Last Closed: 2020-12-17 04:50:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1779055    
Bug Blocks:    

Description hari gowtham 2019-12-10 09:17:24 UTC
+++ This bug was initially created as a clone of Bug #1779055 +++

Description of problem:
when test with ior tool glusterfs client process memory leak found
no else op is carry on , only do io through gluster client process.
glusterfs client process eat up more and more memory
Version-Release number of selected component (if applicable):
# glusterfs -V
glusterfs 7.0
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.


How reproducible:


Steps to Reproduce:
1.begin io with ior tool :
python /opt/tool/ior -mfx -s 1000 -n 100 -t 10 /mnt/testvol/testdir
2.statedump of the glusterfs cleint process
3.from statedump the glusterfs client process memory is increasing,even after stop test and delete all created files

Actual results:
memory goes up never go back even after remove all created files

Expected results:
memory back to normal

Additional info:
from statedump,xlator.mount.fuse.itable.active_size seems keep increasing
from enclosed statedump you can find this.

--- Additional comment from zhou lin on 2019-12-03 07:57:23 UTC ---

# gluster v info config
 
Volume Name: config
Type: Replicate
Volume ID: e4690308-7345-4e32-8d31-b13e10e87112
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 169.254.0.30:/mnt/bricks/config/brick
Brick2: 169.254.0.28:/mnt/bricks/config/brick
Options Reconfigured:
performance.client-io-threads: off
server.allow-insecure: on
network.frame-timeout: 180
network.ping-timeout: 42
cluster.consistent-metadata: off
cluster.favorite-child-policy: mtime
cluster.server-quorum-type: none
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
cluster.server-quorum-ratio: 51

--- Additional comment from zhou lin on 2019-12-04 02:54:53 UTC ---

from statedump it is quite obvious that
xlator.mount.fuse.itable.active_size keeps growing
thera are more and more follwing sections appearing in statedump

[xlator.mount.fuse.itable.active.1]
gfid=924e4dde-79a5-471b-9a6e-7d769f0bae61
nlookup=0
fd-count=0
active-fd-count=0
ref=100
invalidate-sent=0
ia_type=2
ref_by_xl:.hsjvol-client-0=1
ref_by_xl:.hsjvol-readdir-ahead=99

--- Additional comment from Worker Ant on 2019-12-05 05:55:21 UTC ---

REVIEW: https://review.gluster.org/23811 (To fix readdir-ahead memory leak) posted (#1) for review on master by None

--- Additional comment from Worker Ant on 2019-12-05 06:37:29 UTC ---

REVIEW: https://review.gluster.org/23812 (To fix readdir-ahead memory leak) posted (#1) for review on master by None

--- Additional comment from Worker Ant on 2019-12-05 07:05:12 UTC ---

REVIEW: https://review.gluster.org/23813 (To fix readdir-ahead memory leak) posted (#1) for review on master by None

--- Additional comment from Worker Ant on 2019-12-05 08:08:58 UTC ---

REVIEW: https://review.gluster.org/23815 (To fix readdir-ahead memory leak) posted (#1) for review on master by None

--- Additional comment from Worker Ant on 2019-12-10 05:01:24 UTC ---

REVIEW: https://review.gluster.org/23815 (To fix readdir-ahead memory leak) merged (#2) on master by Amar Tumballi

Comment 13 Manisha Saini 2020-11-11 08:11:41 UTC
Verified this BZ with 

# rpm -qa | grep gluster
glusterfs-libs-6.0-46.el7rhgs.x86_64
glusterfs-api-6.0-46.el7rhgs.x86_64
glusterfs-geo-replication-6.0-46.el7rhgs.x86_64
glusterfs-6.0-46.el7rhgs.x86_64
glusterfs-fuse-6.0-46.el7rhgs.x86_64
glusterfs-cli-6.0-46.el7rhgs.x86_64
python2-gluster-6.0-46.el7rhgs.x86_64
glusterfs-client-xlators-6.0-46.el7rhgs.x86_64
glusterfs-server-6.0-46.el7rhgs.x86_64


Created multiple files on mount point.
Also Ran fio and crefi to create multiple files on mount point.Deleted the files from mount point.No spike in memory was observed.Moving this BZ to verified state.

Comment 15 errata-xmlrpc 2020-12-17 04:50:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (glusterfs bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5603