Bug 1535281
| Summary: | possible memleak in glusterfsd process with brick multiplexing on | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | krishnaram Karthick <kramdoss> | |
| Component: | core | Assignee: | Mohit Agrawal <moagrawa> | |
| Status: | CLOSED ERRATA | QA Contact: | Bala Konda Reddy M <bmekala> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | cns-3.6 | CC: | abhishku, amukherj, kramdoss, nchilaka, nravinas, pprakash, rcyriac, rhinduja, rhs-bugs, storage-qa-internal, vbellur | |
| Target Milestone: | --- | |||
| Target Release: | RHGS 3.4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | brick-multiplexing | |||
| Fixed In Version: | glusterfs-3.12.2-8 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1535784 1544090 1549473 (view as bug list) | Environment: | ||
| Last Closed: | 2018-09-04 06:40:51 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1544090, 1549501 | |||
| Bug Blocks: | 1503137, 1535784, 1549473 | |||
Build : 3.12.2-8
On a three node brick-mux enabled setup. Created a base 2X3 volume. In a loop created two volumes, started, stopped and deleted. Did it for 3500 iterations.
At the end glusterfsd memory increased to 2.2G earlier it went to 14G. Compared to it memleak reduced very much.
As discussed with mohit, it is acceptable. Here is the output of the memory consumption.
###############1 iteration #############
total used free shared buff/cache available
Mem: 7.6G 246M 7.2G 8.8M 207M 7.1G
Swap: 2.0G 0B 2.0G
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1614 root 20 0 606860 9040 4112 S 0.0 0.1 0:00.44 glusterd
2325 root 20 0 1869420 21684 3684 S 0.0 0.3 0:00.09 glusterfsd
###############3613 iteration #############
total used free shared buff/cache available
Mem: 7.6G 2.5G 4.1G 88M 1.0G 4.7G
Swap: 2.0G 0B 2.0G
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1614 root 20 0 615440 43916 4404 S 0.0 0.5 11:38.95 glusterd
2325 root 20 0 86.9g 2.2g 4256 S 0.0 28.8 11:38.02 glusterfsd
Hence marking it as verified.
*** Bug 1619369 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607 |
Description of problem: With brick multiplexing on, when volume creation and deletion was run continuously for ~12 hours, glusterfsd process on each of the three nodes consumes close to 14gb of memory with a single volume in the system. This is quite high. Please note that throughout the test, heketidb volume is not deleted and hence the same brick process remain throughout the test. Version-Release number of selected component (if applicable): sh-4.2# rpm -qa | grep 'gluster' glusterfs-libs-3.8.4-54.el7rhgs.x86_64 glusterfs-3.8.4-54.el7rhgs.x86_64 glusterfs-api-3.8.4-54.el7rhgs.x86_64 glusterfs-cli-3.8.4-54.el7rhgs.x86_64 glusterfs-fuse-3.8.4-54.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-54.el7rhgs.x86_64 glusterfs-client-xlators-3.8.4-54.el7rhgs.x86_64 glusterfs-server-3.8.4-54.el7rhgs.x86_64 gluster-block-0.2.1-14.el7rhgs.x86_64 How reproducible: Always Steps to Reproduce: 1. on a CNS setup, run the following script for 12 hours. while true; do for i in {1..5}; do heketi-cli volume create --size=1; done; heketi-cli volume list | awk '{print $1}' | cut -c 4- >> vollist; while read i; do heketi-cli volume delete $i; sleep 2; done<vollist; rm vollist; done Actual results: glusterfsd process consumes ~14 gb with 1 volume Expected results: typically, glusterfsd would consume < 1gb for a volume Additional info: