Bug 1736830

Summary: issuing an interrupt(ctrl+c) post lookup hung issue(BZ#1731896 ) is leaking memory and finally fuse mount gets OOM killed
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nag Pavan Chilakam <nchilaka>
Component: fuseAssignee: Csaba Henk <csaba>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: urgent Docs Contact:
Priority: high    
Version: rhgs-3.5CC: amukherj, csaba, nbalacha, rhs-bugs, sheggodu, storage-qa-internal
Target Milestone: ---Flags: csaba: needinfo-
Target Release: RHGS 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0-11 Doc Type: Bug Fix
Doc Text:
Previously, dynamically allocated memory was not freed correctly, which led to an increase in memory consumption and out-of-memory management on gluster clients. Memory is now freed correctly so that memory overruns do not occur.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-30 12:22:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1696809    
Attachments:
Description Flags
top-output.txt none

Description Nag Pavan Chilakam 2019-08-02 06:58:22 UTC
Description of problem:
========================
raising this issue to track the memory leak seperately from the lookup hung as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1731896#c21


Version-Release number of selected component (if applicable):
===========
6.0.9


How reproducible:
===============
2/2 quite consistent

Steps to Reproduce:
====================
1.created a 3 node cluster , enabled  brickmux
2. created a 1x3 volume "repvol"and 21x(4+2) ecvolume "ecv" {each server hosts 2 bricks of each ec set)
3. set below options to ecvol
server.event-threads: 8
client.event-threads: 8
disperse.shd-max-threads: 24
4. mounted both ecv and repvol on 6 different clients
5. top o/p being continously captured for each client in repvol
6. IOs triggered on ecv in below pattern
    a). linux untar from 2 clients for 50times
    b). crefi from 2 clients "#for j in {1..20};do for i in {create,chmod,chown,chgrp,symlink,truncate,rename,hardlink}; do crefi --multi -n 5 -b 20 -d 10 --max=1K --min=50 --random -T 2 -t text --fop=$i /mnt/cvlt-ecv/IOs/crefi/$HOSTNAME/ ; sleep 10 ; done;rm -rf /mnt/cvlt-ecv/IOs/crefi/$HOSTNAME/*;done"
    c). lookups (find *|xargs stat) from all 6 clients


after about half a day, the lookups hang on the fuse client for which it was reported in BZ#1731896

Now, if a user tries to interrupt the hang state, by doing ctrl+c couple of times, the fuse mount starts to leak memory and finally the fuse mount gets OOM killed once no more memory is available


[Wed Jul 31 11:34:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:34:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:34:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:34:49 2019] Call Trace:
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:34:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:34:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:34:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:34:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:34:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:36:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:36:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:36:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:36:49 2019] Call Trace:
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:36:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:36:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:36:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:36:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:36:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:38:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:38:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:38:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:38:49 2019] Call Trace:
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:38:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:38:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:38:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:38:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:38:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:40:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:40:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:40:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:40:49 2019] Call Trace:
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:40:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:40:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:40:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:40:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:40:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:42:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:42:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:42:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:42:49 2019] Call Trace:
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:42:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:42:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:42:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:42:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:42:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:44:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:44:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:44:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:44:49 2019] Call Trace:
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:44:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:44:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:44:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:44:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:44:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:46:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:46:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:46:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:46:49 2019] Call Trace:
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:46:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:46:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:46:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:46:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:46:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:48:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:48:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:48:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:48:49 2019] Call Trace:
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:48:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:48:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:48:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:48:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:48:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:50:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:50:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:50:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:50:49 2019] Call Trace:
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:50:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:50:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:50:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:50:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:50:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:52:49 2019] INFO: task find:11295 blocked for more than 120 seconds.
[Wed Jul 31 11:52:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jul 31 11:52:49 2019] find            D ffff8a7ff5e5d230     0 11295  30513 0x00000084
[Wed Jul 31 11:52:49 2019] Call Trace:
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf57f1c9>] schedule+0x29/0x70
[Wed Jul 31 11:52:49 2019]  [<ffffffffc08244d5>] __fuse_request_send+0xf5/0x2e0 [fuse]
[Wed Jul 31 11:52:49 2019]  [<ffffffffaeec61c0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jul 31 11:52:49 2019]  [<ffffffffc08246d2>] fuse_request_send+0x12/0x20 [fuse]
[Wed Jul 31 11:52:49 2019]  [<ffffffffc082a562>] fuse_readdir+0x2d2/0x730 [fuse]
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf05dd37>] iterate_dir+0x97/0x130
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf05e252>] SyS_getdents+0xa2/0x130
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf05ddd0>] ? iterate_dir+0x130/0x130
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf58be15>] ? system_call_after_swapgs+0xa2/0x146
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf58bede>] system_call_fastpath+0x25/0x2a
[Wed Jul 31 11:52:49 2019]  [<ffffffffaf58be21>] ? system_call_after_swapgs+0xae/0x146
[Fri Aug  2 00:28:14 2019] irqbalance invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[Fri Aug  2 00:28:14 2019] irqbalance cpuset=/ mems_allowed=0-1
[Fri Aug  2 00:28:14 2019] CPU: 0 PID: 1518 Comm: irqbalance Kdump: loaded Not tainted 3.10.0-1061.el7.x86_64 #1
[Fri Aug  2 00:28:14 2019] Hardware name: Supermicro X9DRW-3LN4F+/X9DRW-3TF+/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 1.0b 05/29/2012
[Fri Aug  2 00:28:14 2019] Call Trace:
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf579262>] dump_stack+0x19/0x1b
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf573c04>] dump_header+0x90/0x229
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf10825b>] ? cred_has_capability+0x6b/0x120
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefbfd74>] oom_kill_process+0x254/0x3e0
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf10833e>] ? selinux_capable+0x2e/0x40
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefc05c6>] out_of_memory+0x4b6/0x4f0
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf57471c>] __alloc_pages_slowpath+0x5d6/0x724
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefc6b84>] __alloc_pages_nodemask+0x404/0x420
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf014c68>] alloc_pages_current+0x98/0x110
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefbbc77>] __page_cache_alloc+0x97/0xb0
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefbe838>] filemap_fault+0x298/0x490
[Fri Aug  2 00:28:14 2019]  [<ffffffffc050357e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
[Fri Aug  2 00:28:14 2019]  [<ffffffffc050377c>] xfs_filemap_fault+0x2c/0x30 [xfs]
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefea29a>] __do_fault.isra.61+0x8a/0x100
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefea84c>] do_read_fault.isra.63+0x4c/0x1b0
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefef2fa>] handle_pte_fault+0x22a/0xe20
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefebc37>] ? arch_tlb_finish_mmu+0x47/0x80
[Fri Aug  2 00:28:14 2019]  [<ffffffffaefebd53>] ? tlb_finish_mmu+0x23/0x30
[Fri Aug  2 00:28:14 2019]  [<ffffffffaeff5544>] ? unmap_region+0xf4/0x140
[Fri Aug  2 00:28:14 2019]  [<ffffffffaeff200d>] handle_mm_fault+0x39d/0x9b0
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf586633>] __do_page_fault+0x213/0x500
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf586955>] do_page_fault+0x35/0x90
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf582ab6>] ? error_swapgs+0xa7/0xbd
[Fri Aug  2 00:28:14 2019]  [<ffffffffaf582768>] page_fault+0x28/0x30
[Fri Aug  2 00:28:14 2019] Mem-Info:
[Fri Aug  2 00:28:14 2019] active_anon:3372927 inactive_anon:512780 isolated_anon:0
 active_file:477 inactive_file:1296 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 slab_reclaimable:8871 slab_unreclaimable:22324
 mapped:1245 shmem:43107 pagetables:13079 bounce:0
 free:32594 free_pcp:5015 free_cma:0
[Fri Aug  2 00:28:14 2019] Node 0 DMA free:15848kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15932kB managed:15848kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[Fri Aug  2 00:28:14 2019] lowmem_reserve[]: 0 1778 7777 7777
[Fri Aug  2 00:28:14 2019] Node 0 DMA32 free:34008kB min:10112kB low:12640kB high:15168kB active_anon:1305548kB inactive_anon:435304kB active_file:16kB inactive_file:2504kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052108kB managed:1821676kB mlocked:0kB dirty:0kB writeback:0kB mapped:300kB shmem:17176kB slab_reclaimable:4528kB slab_unreclaimable:10072kB kernel_stack:304kB pagetables:6028kB unstable:0kB bounce:0kB free_pcp:5736kB local_pcp:380kB free_cma:0kB writeback_tmp:0kB pages_scanned:4185 all_unreclaimable? yes
[Fri Aug  2 00:28:14 2019] lowmem_reserve[]: 0 0 5998 5998
[Fri Aug  2 00:28:14 2019] Node 0 Normal free:33836kB min:34080kB low:42600kB high:51120kB active_anon:5171612kB inactive_anon:738892kB active_file:300kB inactive_file:1228kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6291456kB managed:6142164kB mlocked:0kB dirty:0kB writeback:0kB mapped:3124kB shmem:3660kB slab_reclaimable:17008kB slab_unreclaimable:44212kB kernel_stack:3232kB pagetables:22700kB unstable:0kB bounce:0kB free_pcp:7180kB local_pcp:584kB free_cma:0kB writeback_tmp:0kB pages_scanned:2486 all_unreclaimable? yes
[Fri Aug  2 00:28:14 2019] lowmem_reserve[]: 0 0 0 0
[Fri Aug  2 00:28:14 2019] Node 1 Normal free:46684kB min:45828kB low:57284kB high:68740kB active_anon:7014548kB inactive_anon:876924kB active_file:1592kB inactive_file:1452kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8255016kB mlocked:0kB dirty:0kB writeback:0kB mapped:1556kB shmem:151592kB slab_reclaimable:13948kB slab_unreclaimable:35012kB kernel_stack:2528kB pagetables:23588kB unstable:0kB bounce:0kB free_pcp:7144kB local_pcp:160kB free_cma:0kB writeback_tmp:0kB pages_scanned:4110 all_unreclaimable? no
[Fri Aug  2 00:28:14 2019] lowmem_reserve[]: 0 0 0 0
[Fri Aug  2 00:28:14 2019] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15848kB
[Fri Aug  2 00:28:14 2019] Node 0 DMA32: 914*4kB (U) 438*8kB (UE) 226*16kB (UE) 208*32kB (UEM) 99*64kB (UE) 44*128kB (UEM) 18*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 34008kB
[Fri Aug  2 00:28:14 2019] Node 0 Normal: 2256*4kB (UE) 871*8kB (UE) 603*16kB (UE) 267*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 34184kB
[Fri Aug  2 00:28:14 2019] Node 1 Normal: 1208*4kB (UEM) 568*8kB (UEM) 274*16kB (UEM) 250*32kB (UEM) 137*64kB (UEM) 105*128kB (UEM) 11*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46784kB
[Fri Aug  2 00:28:14 2019] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Fri Aug  2 00:28:14 2019] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Fri Aug  2 00:28:14 2019] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Fri Aug  2 00:28:14 2019] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Fri Aug  2 00:28:14 2019] 53912 total pagecache pages
[Fri Aug  2 00:28:14 2019] 8748 pages in swap cache
[Fri Aug  2 00:28:14 2019] Swap cache stats: add 2218430, delete 2208656, find 69999/81445
[Fri Aug  2 00:28:14 2019] Free swap  = 0kB
[Fri Aug  2 00:28:14 2019] Total swap = 8257532kB
[Fri Aug  2 00:28:14 2019] 4187026 pages RAM
[Fri Aug  2 00:28:14 2019] 0 pages HighMem/MovableOnly
[Fri Aug  2 00:28:14 2019] 128350 pages reserved
[Fri Aug  2 00:28:14 2019] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[Fri Aug  2 00:28:14 2019] [  951]     0   951     9770     1163      25       46             0 systemd-journal
[Fri Aug  2 00:28:14 2019] [  983]     0   983    50275        0      32      905             0 lvmetad
[Fri Aug  2 00:28:14 2019] [  987]     0   987    12156        2      25      720         -1000 systemd-udevd
[Fri Aug  2 00:28:14 2019] [ 1475]     0  1475    13882       26      26       85         -1000 auditd
[Fri Aug  2 00:28:14 2019] [ 1494]   998  1494   153252      132      62     1423             0 polkitd
[Fri Aug  2 00:28:14 2019] [ 1496]    81  1496    16638      110      33       85          -900 dbus-daemon
[Fri Aug  2 00:28:14 2019] [ 1518]     0  1518     5444      127      15       41             0 irqbalance
[Fri Aug  2 00:28:14 2019] [ 1524]     0  1524     6653       41      20       85             0 systemd-logind
[Fri Aug  2 00:28:14 2019] [ 1531]     0  1531    31572       27      18      130             0 crond
[Fri Aug  2 00:28:14 2019] [ 1534]   997  1534    29451       29      29       86             0 chronyd
[Fri Aug  2 00:28:14 2019] [ 1548]     0  1548    90685      235      97     6421             0 firewalld
[Fri Aug  2 00:28:14 2019] [ 1551]     0  1551    27527        2      11       32             0 agetty
[Fri Aug  2 00:28:14 2019] [ 1556]     0  1556   137030      311      82      287             0 NetworkManager
[Fri Aug  2 00:28:14 2019] [ 1707]     0  1707    25724       58      52      459             0 dhclient
[Fri Aug  2 00:28:14 2019] [ 1947]     0  1947   146649      139     104     3264             0 tuned
[Fri Aug  2 00:28:14 2019] [ 1951]     0  1951    28230       28      59      230         -1000 sshd
[Fri Aug  2 00:28:14 2019] [ 1953]     0  1953    28928       23      13       17             0 rhsmcertd
[Fri Aug  2 00:28:14 2019] [ 2002]     0  2002    26993       32       9        6             0 rhnsd
[Fri Aug  2 00:28:14 2019] [ 2475]     0  2475    22424       19      42      240             0 master
[Fri Aug  2 00:28:14 2019] [ 2491]    89  2491    22467       14      46      239             0 qmgr
[Fri Aug  2 00:28:14 2019] [15662]     0 15662    93588        0     147     5229          -900 rhsmd
[Fri Aug  2 00:28:14 2019] [24624]     0 24624    54105      274      40      112             0 rsyslogd
[Fri Aug  2 00:28:14 2019] [28685]     0 28685    38712       58      79      296             0 sshd
[Fri Aug  2 00:28:14 2019] [28692]     0 28692    28889       96      13       39             0 bash
[Fri Aug  2 00:28:14 2019] [28952]     0 28952   142588     1414      68     1983             0 glusterfs
[Fri Aug  2 00:28:14 2019] [29027]     0 29027  6011388  3823343   11396  1965302             0 glusterfs
[Fri Aug  2 00:28:14 2019] [29313]     0 29313    31976       66      18       71             0 screen
[Fri Aug  2 00:28:14 2019] [29314]     0 29314    28894        4      13      115             0 bash
[Fri Aug  2 00:28:14 2019] [29332]     0 29332    33580      113      20     4693             0 bash
[Fri Aug  2 00:28:14 2019] [30512]     0 30512    31976        1      17      164             0 screen
[Fri Aug  2 00:28:14 2019] [30513]     0 30513    28894        2      13      118             0 bash
[Fri Aug  2 00:28:14 2019] [11295]     0 11295    30057        2      16       77             0 find
[Fri Aug  2 00:28:14 2019] [21704]     0 21704    38712       55      80      306             0 sshd
[Fri Aug  2 00:28:14 2019] [21712]     0 21712    28862       83      13       34             0 bash
[Fri Aug  2 00:28:14 2019] [21801]     0 21801     5010       62      15        0             0 tmux
[Fri Aug  2 00:28:14 2019] [21803]     0 21803     7174     1706      19        0             0 tmux
[Fri Aug  2 00:28:14 2019] [21804]     0 21804    35882      547      25        0             0 zsh
[Fri Aug  2 00:28:14 2019] [22023]     0 22023    35764      425      25        0             0 zsh
[Fri Aug  2 00:28:14 2019] [ 3036]     0  3036    35701      394      26        0             0 zsh
[Fri Aug  2 00:28:14 2019] [ 3728]     0  3728    34120     1413      22        0             0 ruby
[Fri Aug  2 00:28:14 2019] [ 4339]     0  4339    35118      320      26        0             0 zsh
[Fri Aug  2 00:28:14 2019] [ 4419]     0  4419    26989       26       9        0             0 tee
[Fri Aug  2 00:28:14 2019] [ 4420]     0  4420    35118      320      23        0             0 zsh
[Fri Aug  2 00:28:14 2019] [ 5588]     0  5588    27578       41      11        0             0 less
[Fri Aug  2 00:28:14 2019] [19554]    89 19554    22450      253      45        0             0 pickup
[Fri Aug  2 00:28:14 2019] [20465]     0 20465    39838     1312      33        0             0 rhsmcertd-worke
[Fri Aug  2 00:28:14 2019] [20904]     0 20904     1058       50       7        0             0 tee
[Fri Aug  2 00:28:14 2019] [20927]     0 20927    38205       83      30        0             0 ps
[Fri Aug  2 00:28:14 2019] Out of memory: Kill process 29027 (glusterfs) score 918 or sacrifice child
[Fri Aug  2 00:28:14 2019] Killed process 29027 (glusterfs), UID 0, total-vm:24045552kB, anon-rss:15293372kB, file-rss:0kB, shmem-rss:0kB

Comment 3 Nag Pavan Chilakam 2019-08-02 07:06:34 UTC
it can be seen below, that as long as IOs were going on the resident memory was more or less at 1.2gb, Once I did a ctrl+c, ie at around "11 AM Wed Jul 31 " then the memory started to increase till oom kill

look at attachment for top o/p every 2 min
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.65 glusterfs
Wed Jul 31 10:39:24 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.66 glusterfs
Wed Jul 31 10:41:24 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.67 glusterfs
Wed Jul 31 10:43:25 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.68 glusterfs
Wed Jul 31 10:45:25 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.69 glusterfs
Wed Jul 31 10:47:25 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.70 glusterfs
Wed Jul 31 10:49:25 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.71 glusterfs
Wed Jul 31 10:51:25 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.72 glusterfs
Wed Jul 31 10:53:26 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.73 glusterfs
Wed Jul 31 10:55:26 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.74 glusterfs
Wed Jul 31 10:57:26 IST 2019
29027 root      20   0 2156528   1.2g   4456 S   0.0  7.5 280:33.74 glusterfs
Wed Jul 31 11:35:30 IST 2019
29027 root      20   0 2222064   1.2g   4500 S   0.0  7.6 280:44.36 glusterfs
Wed Jul 31 11:37:30 IST 2019
29027 root      20   0 2222064   1.2g   4500 S   0.0  7.7 280:45.46 glusterfs
Wed Jul 31 11:39:30 IST 2019
29027 root      20   0 2222064   1.2g   4500 S   0.0  7.9 280:46.50 glusterfs
Wed Jul 31 11:41:30 IST 2019
29027 root      20   0 2222064   1.2g   4500 S   0.0  8.0 280:47.57 glusterfs
Wed Jul 31 11:43:31 IST 2019
29027 root      20   0 2287600   1.3g   4500 S   0.0  8.1 280:48.80 glusterfs
Wed Jul 31 11:45:31 IST 2019
29027 root      20   0 2287600   1.3g   4500 S   0.0  8.2 280:50.02 glusterfs
Wed Jul 31 11:47:31 IST 2019
29027 root      20   0 2287600   1.3g   4500 S   0.0  8.4 280:51.21 glusterfs
Wed Jul 31 11:49:31 IST 2019
29027 root      20   0 2353136   1.3g   4500 S   0.0  8.5 280:52.56 glusterfs
Wed Jul 31 11:51:32 IST 2019
29027 root      20   0 2353136   1.3g   4500 S   0.0  8.6 280:53.70 glusterfs
Wed Jul 31 11:53:32 IST 2019
29027 root      20   0 2353136   1.4g   4512 S   0.0  8.8 281:03.09 glusterfs
Wed Jul 31 11:55:32 IST 2019
29027 root      20   0 2418672   1.4g   4512 S   0.0  8.8 281:03.98 glusterfs
Wed Jul 31 11:57:32 IST 2019
29027 root      20   0 2418672   1.4g   4512 S   6.2  8.9 281:04.98 glusterfs
Wed Jul 31 11:59:32 IST 2019
29027 root      20   0 2418672   1.4g   4512 S   0.0  9.1 281:06.08 glusterfs
Wed Jul 31 12:01:33 IST 2019
29027 root      20   0 2418672   1.4g   4512 S   0.0  9.2 281:07.21 glusterfs
Wed Jul 31 12:03:33 IST 2019
29027 root      20   0 2484208   1.4g   4512 S   0.0  9.3 281:08.27 glusterfs
Wed Jul 31 12:05:33 IST 2019
29027 root      20   0 2484208   1.5g   4512 S   0.0  9.4 281:09.38 glusterfs
Wed Jul 31 12:07:33 IST 2019
29027 root      20   0 2484208   1.5g   4512 S   0.0  9.5 281:10.51 glusterfs
Wed Jul 31 12:09:33 IST 2019
29027 root      20   0 2549744   1.5g   4512 S   0.0  9.7 281:11.68 glusterfs
Wed Jul 31 12:11:34 IST 2019
29027 root      20   0 2549744   1.5g   4512 S   0.0  9.8 281:12.87 glusterfs
Wed Jul 31 12:13:34 IST 2019
29027 root      20   0 2549744   1.5g   4512 S   6.2  9.9 281:14.00 glusterfs
Wed Jul 31 12:15:34 IST 2019
29027 root      20   0 2549744   1.6g   4512 S   0.0 10.0 281:15.06 glusterfs
Wed Jul 31 12:17:34 IST 2019
29027 root      20   0 2615280   1.6g   4512 S   0.0 10.2 281:16.44 glusterfs
Wed Jul 31 12:19:34 IST 2019
29027 root      20   0 2615280   1.6g   4512 S   0.0 10.3 281:17.72 glusterfs
Wed Jul 31 12:21:35 IST 2019
29027 root      20   0 2680816   1.6g   4512 S   0.0 10.5 281:18.90 glusterfs
Wed Jul 31 12:23:35 IST 2019
29027 root      20   0 2680816   1.6g   4512 S   0.0 10.6 281:20.20 glusterfs
Wed Jul 31 12:25:35 IST 2019
29027 root      20   0 2680816   1.7g   4512 S   0.0 10.8 281:21.48 glusterfs
Wed Jul 31 12:27:35 IST 2019
29027 root      20   0 2746352   1.7g   4512 S   0.0 10.9 281:22.70 glusterfs
Wed Jul 31 12:29:35 IST 2019
29027 root      20   0 2746352   1.7g   4512 S   0.0 11.0 281:23.59 glusterfs
Wed Jul 31 12:31:36 IST 2019
29027 root      20   0 2746352   1.7g   4512 S   0.0 11.1 281:24.67 glusterfs
Wed Jul 31 12:33:36 IST 2019
29027 root      20   0 2746352   1.7g   4512 S   0.0 11.2 281:25.79 glusterfs
Wed Jul 31 12:35:36 IST 2019
29027 root      20   0 2811888   1.8g   4512 S   6.7 11.3 281:26.88 glusterfs
Wed Jul 31 12:37:36 IST 2019
29027 root      20   0 2811888   1.8g   4512 S   0.0 11.4 281:27.88 glusterfs
Wed Jul 31 12:39:36 IST 2019
29027 root      20   0 2811888   1.8g   4512 S   0.0 11.5 281:28.93 glusterfs
Wed Jul 31 12:41:37 IST 2019
29027 root      20   0 2877424   1.8g   4512 S   0.0 11.7 281:30.06 glusterfs
Wed Jul 31 12:43:37 IST 2019
29027 root      20   0 2877424   1.8g   4512 S   0.0 11.8 281:31.14 glusterfs
Wed Jul 31 12:45:37 IST 2019
29027 root      20   0 2877424   1.8g   4512 S   0.0 11.9 281:32.17 glusterfs
Wed Jul 31 12:47:37 IST 2019
29027 root      20   0 2877424   1.9g   4512 S   0.0 12.0 281:33.22 glusterfs
Wed Jul 31 12:49:37 IST 2019
29027 root      20   0 2942960   1.9g   4512 S   0.0 12.1 281:34.29 glusterfs
Wed Jul 31 12:51:38 IST 2019
29027 root      20   0 2942960   1.9g   4512 S   0.0 12.3 281:35.39 glusterfs
Wed Jul 31 12:53:38 IST 2019
29027 root      20   0 2942960   1.9g   4512 S  12.5 12.4 281:36.50 glusterfs
Wed Jul 31 12:55:38 IST 2019
29027 root      20   0 3008496   1.9g   4512 S   0.0 12.5 281:37.62 glusterfs
Wed Jul 31 12:57:38 IST 2019
29027 root      20   0 3008496   2.0g   4512 S   0.0 12.6 281:38.66 glusterfs
Wed Jul 31 12:59:38 IST 2019
29027 root      20   0 3008496   2.0g   4512 S   0.0 12.8 281:39.69 glusterfs
Wed Jul 31 13:01:39 IST 2019
29027 root      20   0 3074032   2.0g   4512 S   0.0 12.9 281:40.73 glusterfs
Wed Jul 31 13:03:39 IST 2019
29027 root      20   0 3074032   2.0g   4512 S   0.0 13.0 281:41.92 glusterfs
Wed Jul 31 13:05:39 IST 2019
29027 root      20   0 3074032   2.0g   4512 S   0.0 13.2 281:43.20 glusterfs
Wed Jul 31 13:07:39 IST 2019
29027 root      20   0 3139568   2.1g   4512 S   0.0 13.3 281:44.39 glusterfs
Wed Jul 31 13:09:39 IST 2019
29027 root      20   0 3139568   2.1g   4512 S   0.0 13.4 281:45.45 glusterfs
Wed Jul 31 13:11:40 IST 2019
29027 root      20   0 3139568   2.1g   4512 S   0.0 13.6 281:46.52 glusterfs
Wed Jul 31 13:13:40 IST 2019
29027 root      20   0 3205104   2.1g   4512 S   0.0 13.7 281:47.59 glusterfs
Wed Jul 31 13:15:40 IST 2019
29027 root      20   0 3205104   2.1g   4512 S   0.0 13.8 281:48.64 glusterfs
Wed Jul 31 13:17:40 IST 2019
29027 root      20   0 3205104   2.2g   4512 S   0.0 13.9 281:49.74 glusterfs
Wed Jul 31 13:19:40 IST 2019
29027 root      20   0 3205104   2.2g   4512 S   0.0 14.0 281:50.68 glusterfs
Wed Jul 31 13:21:41 IST 2019
29027 root      20   0 3270640   2.2g   4512 S   0.0 14.1 281:51.69 glusterfs
Wed Jul 31 13:23:41 IST 2019
29027 root      20   0 3270640   2.2g   4512 S   0.0 14.2 281:53.05 glusterfs
Wed Jul 31 13:25:41 IST 2019
29027 root      20   0 3270640   2.2g   4512 S   0.0 14.4 281:54.49 glusterfs
Wed Jul 31 13:27:41 IST 2019
29027 root      20   0 3336176   2.3g   4512 S   0.0 14.6 281:55.98 glusterfs
Wed Jul 31 13:29:42 IST 2019
29027 root      20   0 3336176   2.3g   4512 S   0.0 14.7 281:57.19 glusterfs
Wed Jul 31 13:31:42 IST 2019
29027 root      20   0 3336176   2.3g   4512 S   0.0 14.8 281:58.32 glusterfs
Wed Jul 31 13:33:42 IST 2019
29027 root      20   0 3401712   2.3g   4512 S   0.0 14.9 281:59.32 glusterfs
Wed Jul 31 13:35:42 IST 2019
29027 root      20   0 3401712   2.3g   4512 S   0.0 15.1 282:00.55 glusterfs
Wed Jul 31 13:37:42 IST 2019
29027 root      20   0 3401712   2.4g   4512 S   0.0 15.3 282:01.68 glusterfs
Wed Jul 31 13:39:43 IST 2019
29027 root      20   0 3467248   2.4g   4512 S   0.0 15.4 282:02.72 glusterfs
Wed Jul 31 13:41:43 IST 2019
29027 root      20   0 3467248   2.4g   4512 S   0.0 15.5 282:03.88 glusterfs
Wed Jul 31 13:43:43 IST 2019
29027 root      20   0 3467248   2.4g   4512 S   6.2 15.6 282:04.98 glusterfs
Wed Jul 31 13:45:43 IST 2019
29027 root      20   0 3532784   2.4g   4512 S   6.2 15.8 282:06.08 glusterfs
Wed Jul 31 13:47:43 IST 2019
29027 root      20   0 3532784   2.5g   4512 S   0.0 15.9 282:07.15 glusterfs
Wed Jul 31 13:49:44 IST 2019
29027 root      20   0 3532784   2.5g   4512 S   0.0 16.0 282:08.22 glusterfs
Wed Jul 31 13:51:44 IST 2019
29027 root      20   0 3598320   2.5g   4512 S   0.0 16.2 282:09.30 glusterfs
Wed Jul 31 13:53:44 IST 2019
29027 root      20   0 3598320   2.5g   4512 S   0.0 16.3 282:10.36 glusterfs
Wed Jul 31 13:55:44 IST 2019
29027 root      20   0 3598320   2.5g   4512 S   0.0 16.4 282:11.42 glusterfs
Wed Jul 31 13:57:44 IST 2019
29027 root      20   0 3663856   2.6g   4512 S   0.0 16.6 282:12.47 glusterfs
Wed Jul 31 13:59:45 IST 2019
29027 root      20   0 3663856   2.6g   4516 S   0.0 16.7 282:13.54 glusterfs
Wed Jul 31 14:01:45 IST 2019
29027 root      20   0 3663856   2.6g   4516 S   0.0 16.8 282:14.57 glusterfs
Wed Jul 31 17:12:04 IST 2019
29027 root      20   0 5695472   4.5g   4516 S   0.0 29.1 284:13.63 glusterfs
Wed Jul 31 17:14:04 IST 2019
29027 root      20   0 5695472   4.5g   4516 S   0.0 29.3 284:14.70 glusterfs
Wed Jul 31 17:16:05 IST 2019
29027 root      20   0 5695472   4.5g   4516 S   0.0 29.4 284:15.76 glusterfs
Wed Jul 31 17:18:05 IST 2019
29027 root      20   0 5761008   4.6g   4516 S   0.0 29.5 284:16.84 glusterfs
Wed Jul 31 17:20:05 IST 2019
29027 root      20   0 5761008   4.6g   4516 S   0.0 29.6 284:17.86 glusterfs
Wed Jul 31 17:22:05 IST 2019
29027 root      20   0 5761008   4.6g   4516 S   0.0 29.8 284:18.94 glusterfs
Wed Jul 31 17:24:05 IST 2019
29027 root      20   0 5826544   4.6g   4516 S   0.0 29.9 284:20.01 glusterfs
Wed Jul 31 17:26:06 IST 2019
29027 root      20   0 5826544   4.6g   4516 S   0.0 30.0 284:21.10 glusterfs
Wed Jul 31 17:28:06 IST 2019
29027 root      20   0 5826544   4.7g   4516 S   0.0 30.2 284:22.19 glusterfs
Wed Jul 31 17:30:06 IST 2019
29027 root      20   0 5892080   4.7g   4516 S   0.0 30.3 284:23.24 glusterfs
Wed Jul 31 17:32:06 IST 2019
29027 root      20   0 5892080   4.7g   4516 S   0.0 30.4 284:24.28 glusterfs
Wed Jul 31 17:34:06 IST 2019
29027 root      20   0 5892080   4.7g   4516 S   0.0 30.5 284:25.32 glusterfs
Thu Aug  1 04:25:12 IST 2019
29027 root      20   0   12.1g  11.1g   4516 S   0.0 72.0 290:19.44 glusterfs
Thu Aug  1 04:27:13 IST 2019
29027 root      20   0   12.1g  11.2g   4516 S   0.0 72.1 290:20.47 glusterfs
Thu Aug  1 04:29:13 IST 2019
29027 root      20   0   12.1g  11.2g   4516 S   0.0 72.3 290:21.57 glusterfs
Thu Aug  1 04:31:13 IST 2019
29027 root      20   0   12.1g  11.2g   4516 S   0.0 72.5 290:22.79 glusterfs
Thu Aug  1 04:33:13 IST 2019
29027 root      20   0   12.2g  11.2g   4516 S   0.0 72.6 290:23.94 glusterfs
Thu Aug  1 04:35:13 IST 2019
29027 root      20   0   12.2g  11.3g   4516 S   0.0 72.8 290:25.12 glusterfs
Thu Aug  1 04:37:14 IST 2019
29027 root      20   0   12.2g  11.3g   4516 S   0.0 72.9 290:26.30 glusterfs
Thu Aug  1 04:39:14 IST 2019
29027 root      20   0   12.2g  11.3g   4516 S   0.0 73.1 290:27.46 glusterfs
Thu Aug  1 04:41:14 IST 2019
29027 root      20   0   12.2g  11.3g   4516 S   0.0 73.3 290:28.64 glusterfs
Thu Aug  1 04:43:14 IST 2019
29027 root      20   0   12.2g  11.4g   4516 S   0.0 73.4 290:29.68 glusterfs
Thu Aug  1 04:45:15 IST 2019
29027 root      20   0   12.3g  11.4g   4516 S   0.0 73.5 290:30.78 glusterfs
Thu Aug  1 04:47:15 IST 2019
29027 root      20   0   12.3g  11.4g   4516 S   0.0 73.7 290:31.90 glusterfs
Thu Aug  1 04:49:15 IST 2019
29027 root      20   0   12.4g  11.4g   4516 S   0.0 73.8 290:33.11 glusterfs
Thu Aug  1 04:51:15 IST 2019
29027 root      20   0   12.4g  11.5g   4516 S   6.7 74.0 290:34.29 glusterfs
Thu Aug  1 04:53:15 IST 2019
29027 root      20   0   12.4g  11.5g   4516 S   6.2 74.1 290:35.32 glusterfs
Thu Aug  1 04:55:16 IST 2019
29027 root      20   0   12.4g  11.5g   4516 S   0.0 74.2 290:36.34 glusterfs
Thu Aug  1 04:57:16 IST 2019
29027 root      20   0   12.4g  11.5g   4516 S   0.0 74.3 290:37.40 glusterfs
Thu Aug  1 04:59:16 IST 2019
29027 root      20   0   12.4g  11.5g   4516 S   0.0 74.4 290:38.42 glusterfs
Thu Aug  1 05:01:16 IST 2019
29027 root      20   0   12.4g  11.5g   4516 S   0.0 74.5 290:39.44 glusterfs
Thu Aug  1 05:03:16 IST 2019
29027 root      20   0   12.5g  11.6g   4516 S   0.0 74.6 290:40.50 glusterfs
Thu Aug  1 05:05:17 IST 2019
29027 root      20   0   12.5g  11.6g   4516 S   0.0 74.8 290:41.53 glusterfs
Thu Aug  1 05:07:17 IST 2019
29027 root      20   0   12.5g  11.6g   4516 S   0.0 74.9 290:42.57 glusterfs
Thu Aug  1 05:09:17 IST 2019
29027 root      20   0   12.5g  11.6g   4516 S   0.0 75.0 290:43.60 glusterfs
Thu Aug  1 05:11:17 IST 2019
29027 root      20   0   12.6g  11.6g   4516 S   0.0 75.1 290:44.98 glusterfs
Thu Aug  1 05:13:17 IST 2019
29027 root      20   0   12.6g  11.7g   4516 S   6.2 75.3 290:46.28 glusterfs
Thu Aug  1 05:15:18 IST 2019
29027 root      20   0   12.6g  11.7g   4516 S   0.0 75.4 290:47.30 glusterfs
Thu Aug  1 05:17:18 IST 2019
29027 root      20   0   12.6g  11.7g   4516 S   0.0 75.6 290:48.36 glusterfs
Thu Aug  1 05:19:18 IST 2019
29027 root      20   0   12.6g  11.7g   4516 S   0.0 75.7 290:49.39 glusterfs
Thu Aug  1 05:21:18 IST 2019
29027 root      20   0   12.6g  11.7g   4516 S   0.0 75.8 290:50.43 glusterfs
Thu Aug  1 05:23:18 IST 2019
29027 root      20   0   12.7g  11.8g   4516 S   0.0 75.9 290:51.48 glusterfs
Thu Aug  1 05:25:19 IST 2019
29027 root      20   0   12.7g  11.8g   4516 S   0.0 76.1 290:52.51 glusterfs
Thu Aug  1 05:27:19 IST 2019
29027 root      20   0   12.7g  11.8g   4516 S   0.0 76.2 290:53.55 glusterfs
Thu Aug  1 05:29:19 IST 2019
29027 root      20   0   12.7g  11.8g   4516 S   0.0 76.3 290:54.56 glusterfs
Thu Aug  1 05:31:19 IST 2019
29027 root      20   0   12.7g  11.8g   4516 S   6.2 76.4 290:55.61 glusterfs
Thu Aug  1 05:33:20 IST 2019
29027 root      20   0   12.7g  11.9g   4516 S   0.0 76.6 290:56.65 glusterfs
Thu Aug  1 05:35:20 IST 2019
29027 root      20   0   12.8g  11.9g   4516 S   0.0 76.7 290:57.73 glusterfs
Thu Aug  1 05:37:20 IST 2019
29027 root      20   0   12.8g  11.9g   4516 S   0.0 76.8 290:58.79 glusterfs
Thu Aug  1 05:39:20 IST 2019
29027 root      20   0   12.8g  11.9g   4516 S   0.0 77.0 290:59.87 glusterfs
Thu Aug  1 05:41:20 IST 2019
29027 root      20   0   12.9g  11.9g   4516 S   0.0 77.1 291:00.93 glusterfs
Thu Aug  1 05:43:21 IST 2019
29027 root      20   0   12.9g  12.0g   4516 S   0.0 77.2 291:01.96 glusterfs
Thu Aug  1 05:45:21 IST 2019
29027 root      20   0   12.9g  12.0g   4516 S   0.0 77.4 291:02.98 glusterfs
Thu Aug  1 05:47:21 IST 2019
29027 root      20   0   12.9g  12.0g   4516 S   0.0 77.5 291:04.04 glusterfs
Thu Aug  1 05:49:21 IST 2019
29027 root      20   0   12.9g  12.0g   4516 S   0.0 77.6 291:05.07 glusterfs
Thu Aug  1 05:51:21 IST 2019
29027 root      20   0   12.9g  12.0g   4516 S   0.0 77.8 291:06.18 glusterfs
Thu Aug  1 05:53:22 IST 2019
29027 root      20   0   13.0g  12.1g   4516 S   0.0 77.9 291:07.24 glusterfs
Thu Aug  1 05:55:22 IST 2019
29027 root      20   0   13.0g  12.1g   4516 S   0.0 78.0 291:08.31 glusterfs
Thu Aug  1 05:57:22 IST 2019
29027 root      20   0   13.0g  12.1g   4516 S   0.0 78.2 291:09.35 glusterfs
Thu Aug  1 05:59:22 IST 2019
29027 root      20   0   13.1g  12.1g   4516 S   0.0 78.3 291:10.43 glusterfs
Thu Aug  1 06:01:23 IST 2019
29027 root      20   0   13.1g  12.1g   4516 S   0.0 78.4 291:11.50 glusterfs
Thu Aug  1 06:03:23 IST 2019
29027 root      20   0   13.1g  12.2g   4516 S   6.2 78.5 291:12.61 glusterfs
Thu Aug  1 06:05:23 IST 2019
29027 root      20   0   13.1g  12.2g   4516 S   0.0 78.7 291:13.71 glusterfs
Thu Aug  1 06:07:23 IST 2019
29027 root      20   0   13.1g  12.2g   4516 S   0.0 78.8 291:14.77 glusterfs
Thu Aug  1 06:09:23 IST 2019
29027 root      20   0   13.1g  12.2g   4516 S   0.0 78.9 291:15.86 glusterfs
Thu Aug  1 06:11:24 IST 2019
29027 root      20   0   13.1g  12.2g   4516 S   0.0 79.1 291:16.96 glusterfs
Thu Aug  1 06:13:24 IST 2019
29027 root      20   0   13.2g  12.3g   4516 S   0.0 79.2 291:18.08 glusterfs
Thu Aug  1 06:15:24 IST 2019
29027 root      20   0   13.2g  12.3g   4516 S   0.0 79.3 291:19.16 glusterfs
Thu Aug  1 06:17:24 IST 2019
29027 root      20   0   13.2g  12.3g   4516 S   0.0 79.5 291:20.26 glusterfs
Thu Aug  1 06:19:24 IST 2019
29027 root      20   0   13.2g  12.3g   4516 S   0.0 79.6 291:21.36 glusterfs
Thu Aug  1 06:21:25 IST 2019
29027 root      20   0   13.2g  12.3g   4516 S   0.0 79.7 291:22.43 glusterfs
Thu Aug  1 06:23:25 IST 2019
29027 root      20   0   13.2g  12.4g   4516 S   0.0 79.9 291:23.50 glusterfs
Thu Aug  1 06:25:25 IST 2019
29027 root      20   0   13.3g  12.4g   4516 S   0.0 80.0 291:24.55 glusterfs
Thu Aug  1 06:27:25 IST 2019
29027 root      20   0   13.3g  12.4g   4516 S   0.0 80.1 291:25.64 glusterfs
Thu Aug  1 06:29:25 IST 2019
29027 root      20   0   13.3g  12.4g   4516 S   0.0 80.3 291:26.72 glusterfs
Thu Aug  1 06:31:26 IST 2019
29027 root      20   0   13.4g  12.4g   4516 S   0.0 80.4 291:27.82 glusterfs
Thu Aug  1 06:33:26 IST 2019
29027 root      20   0   13.4g  12.5g   4516 S   0.0 80.5 291:28.92 glusterfs
Thu Aug  1 06:35:26 IST 2019
29027 root      20   0   13.4g  12.5g   4516 S   0.0 80.7 291:30.03 glusterfs
Thu Aug  1 06:37:26 IST 2019
29027 root      20   0   13.4g  12.5g   4516 S   0.0 80.8 291:31.12 glusterfs
Thu Aug  1 06:39:26 IST 2019
29027 root      20   0   13.4g  12.5g   4516 S   0.0 80.9 291:32.24 glusterfs
Thu Aug  1 06:41:27 IST 2019
29027 root      20   0   13.4g  12.5g   4516 S   0.0 81.0 291:33.30 glusterfs
Thu Aug  1 06:43:27 IST 2019
29027 root      20   0   13.5g  12.6g   4516 S   0.0 81.2 291:34.37 glusterfs
Thu Aug  1 06:45:27 IST 2019
29027 root      20   0   13.5g  12.6g   4516 S   0.0 81.3 291:35.39 glusterfs
Thu Aug  1 06:47:27 IST 2019
29027 root      20   0   13.5g  12.6g   4516 S   0.0 81.4 291:36.51 glusterfs
Thu Aug  1 06:49:27 IST 2019
29027 root      20   0   13.6g  12.6g   4516 S   0.0 81.5 291:37.65 glusterfs
Thu Aug  1 06:51:28 IST 2019
29027 root      20   0   13.6g  12.6g   4516 S   0.0 81.7 291:38.93 glusterfs
Thu Aug  1 06:53:28 IST 2019
29027 root      20   0   13.6g  12.7g   4516 S   0.0 81.8 291:40.15 glusterfs
Thu Aug  1 06:55:28 IST 2019
29027 root      20   0   13.6g  12.7g   4516 S   0.0 82.0 291:41.42 glusterfs
Thu Aug  1 06:57:28 IST 2019
29027 root      20   0   13.6g  12.7g   4516 S   0.0 82.1 291:42.70 glusterfs
Thu Aug  1 06:59:28 IST 2019
29027 root      20   0   13.6g  12.7g   4516 S   6.2 82.3 291:43.94 glusterfs
Thu Aug  1 07:01:29 IST 2019
29027 root      20   0   13.7g  12.8g   4516 S   0.0 82.4 291:45.08 glusterfs
Thu Aug  1 07:03:29 IST 2019
29027 root      20   0   13.7g  12.8g   4516 S   0.0 82.5 291:46.15 glusterfs
Thu Aug  1 07:05:29 IST 2019
29027 root      20   0   13.7g  12.8g   4516 S   0.0 82.6 291:47.25 glusterfs
Thu Aug  1 07:07:29 IST 2019
29027 root      20   0   13.7g  12.8g   4516 S   0.0 82.8 291:48.34 glusterfs
Thu Aug  1 07:09:30 IST 2019
29027 root      20   0   13.7g  12.8g   4516 S   0.0 82.9 291:49.43 glusterfs
Thu Aug  1 07:11:30 IST 2019
29027 root      20   0   13.7g  12.9g   4516 S   0.0 83.0 291:50.48 glusterfs
Thu Aug  1 07:13:30 IST 2019
29027 root      20   0   13.8g  12.9g   4516 S   0.0 83.2 291:51.49 glusterfs
Thu Aug  1 07:15:30 IST 2019
29027 root      20   0   13.8g  12.9g   4516 S   0.0 83.3 291:52.56 glusterfs
Thu Aug  1 07:17:30 IST 2019
29027 root      20   0   13.8g  12.9g   4516 S   0.0 83.4 291:53.59 glusterfs
Thu Aug  1 07:19:31 IST 2019
29027 root      20   0   13.9g  12.9g   4516 S   0.0 83.6 291:54.67 glusterfs
Thu Aug  1 07:21:31 IST 2019
29027 root      20   0   13.9g  13.0g   4516 S   0.0 83.7 291:55.77 glusterfs
Thu Aug  1 07:23:31 IST 2019
29027 root      20   0   13.9g  13.0g   4516 S   0.0 83.8 291:56.85 glusterfs
Thu Aug  1 07:25:31 IST 2019
29027 root      20   0   13.9g  13.0g   4516 S   0.0 83.9 291:57.94 glusterfs
Thu Aug  1 07:27:31 IST 2019
29027 root      20   0   13.9g  13.0g   4516 S   0.0 84.1 291:59.10 glusterfs
Thu Aug  1 07:29:32 IST 2019
29027 root      20   0   13.9g  13.0g   4516 S   0.0 84.2 292:00.20 glusterfs
Thu Aug  1 07:31:32 IST 2019
29027 root      20   0   14.0g  13.1g   4516 S   6.2 84.4 292:01.34 glusterfs
Thu Aug  1 07:33:32 IST 2019
29027 root      20   0   14.0g  13.1g   4516 S   0.0 84.5 292:02.42 glusterfs
Thu Aug  1 07:35:32 IST 2019
29027 root      20   0   14.0g  13.1g   4516 S   0.0 84.6 292:03.53 glusterfs
Thu Aug  1 07:37:32 IST 2019
29027 root      20   0   14.1g  13.1g   4516 S   0.0 84.8 292:04.62 glusterfs
Thu Aug  1 07:39:33 IST 2019
29027 root      20   0   14.1g  13.1g   4516 S   0.0 84.9 292:05.71 glusterfs
Thu Aug  1 07:41:33 IST 2019
29027 root      20   0   14.1g  13.2g   4516 S   0.0 85.0 292:06.82 glusterfs
Thu Aug  1 07:43:33 IST 2019
29027 root      20   0   14.1g  13.2g   4516 S   0.0 85.2 292:07.91 glusterfs
Thu Aug  1 07:45:34 IST 2019
29027 root      20   0   14.1g  13.2g   4516 S   0.0 85.3 292:09.01 glusterfs
Thu Aug  1 07:47:34 IST 2019
29027 root      20   0   14.1g  13.2g   4516 S   5.6 85.4 292:10.09 glusterfs
Thu Aug  1 07:49:34 IST 2019
29027 root      20   0   14.2g  13.2g   4516 S   0.0 85.5 292:11.15 glusterfs
Thu Aug  1 07:51:34 IST 2019
29027 root      20   0   14.2g  13.3g   4516 S   0.0 85.6 292:12.21 glusterfs
Thu Aug  1 07:53:35 IST 2019
29027 root      20   0   14.2g  13.3g   4516 S   0.0 85.7 292:13.29 glusterfs
Thu Aug  1 07:55:35 IST 2019
29027 root      20   0   14.2g  13.3g   4516 S   0.0 85.8 292:14.34 glusterfs
Thu Aug  1 07:57:35 IST 2019
29027 root      20   0   14.2g  13.3g   4516 S   0.0 85.9 292:15.46 glusterfs
Thu Aug  1 07:59:35 IST 2019
29027 root      20   0   14.2g  13.3g   4516 S   0.0 86.0 292:16.53 glusterfs
Thu Aug  1 08:01:35 IST 2019
29027 root      20   0   14.2g  13.3g   4516 S   0.0 86.1 292:17.61 glusterfs
Thu Aug  1 08:03:36 IST 2019
29027 root      20   0   14.2g  13.3g   4516 S   0.0 86.2 292:18.67 glusterfs
Thu Aug  1 08:05:36 IST 2019
29027 root      20   0   14.2g  13.4g   4516 S   6.2 86.3 292:19.79 glusterfs
Thu Aug  1 08:07:36 IST 2019
29027 root      20   0   14.3g  13.4g   4516 S   0.0 86.4 292:20.92 glusterfs
Thu Aug  1 08:09:36 IST 2019
29027 root      20   0   14.3g  13.4g   4516 S   0.0 86.5 292:21.96 glusterfs
Thu Aug  1 08:11:36 IST 2019
29027 root      20   0   14.3g  13.4g   4516 S   6.7 86.6 292:23.05 glusterfs
Thu Aug  1 08:13:37 IST 2019
29027 root      20   0   14.4g  13.4g   4516 S   0.0 86.7 292:24.12 glusterfs
Thu Aug  1 08:15:37 IST 2019
29027 root      20   0   14.4g  13.4g   4516 S   6.2 86.8 292:25.21 glusterfs
Thu Aug  1 08:17:37 IST 2019
29027 root      20   0   14.4g  13.5g   4516 S   0.0 86.9 292:26.39 glusterfs
Thu Aug  1 08:19:37 IST 2019
29027 root      20   0   14.4g  13.5g   4516 S   0.0 87.1 292:27.90 glusterfs
Thu Aug  1 08:21:38 IST 2019
29027 root      20   0   14.4g  13.5g   4516 S   0.0 87.3 292:29.52 glusterfs
Thu Aug  1 08:23:38 IST 2019
29027 root      20   0   14.4g  13.5g   4516 S   0.0 87.4 292:30.92 glusterfs
Thu Aug  1 08:25:38 IST 2019
29027 root      20   0   14.5g  13.6g   4516 S   0.0 87.6 292:32.21 glusterfs
Thu Aug  1 08:27:38 IST 2019
29027 root      20   0   14.5g  13.6g   4516 S   0.0 87.7 292:33.26 glusterfs
Thu Aug  1 08:29:38 IST 2019
29027 root      20   0   14.5g  13.6g   4516 S   0.0 87.8 292:34.35 glusterfs
Thu Aug  1 08:31:39 IST 2019
29027 root      20   0   14.5g  13.6g   4516 S   6.2 87.9 292:35.44 glusterfs
Thu Aug  1 08:33:39 IST 2019
29027 root      20   0   14.6g  13.6g   4516 S   0.0 88.1 292:36.58 glusterfs
Thu Aug  1 08:35:39 IST 2019
29027 root      20   0   14.6g  13.7g   4516 S   0.0 88.2 292:37.64 glusterfs
Thu Aug  1 08:37:39 IST 2019
29027 root      20   0   14.6g  13.7g   4516 S   0.0 88.3 292:38.75 glusterfs
Thu Aug  1 08:39:39 IST 2019
29027 root      20   0   14.6g  13.7g   4516 S   0.0 88.4 292:39.85 glusterfs
Thu Aug  1 08:41:40 IST 2019
29027 root      20   0   14.6g  13.7g   4516 S   0.0 88.6 292:41.02 glusterfs
Thu Aug  1 08:43:40 IST 2019
29027 root      20   0   14.6g  13.7g   4516 S   0.0 88.7 292:42.04 glusterfs
Thu Aug  1 08:45:40 IST 2019
29027 root      20   0   14.7g  13.8g   4516 S   0.0 88.8 292:43.16 glusterfs
Thu Aug  1 08:47:40 IST 2019
29027 root      20   0   14.7g  13.8g   4516 S   0.0 88.9 292:44.24 glusterfs
Thu Aug  1 08:49:40 IST 2019
29027 root      20   0   14.7g  13.8g   4516 S   0.0 89.1 292:45.32 glusterfs
Thu Aug  1 08:51:41 IST 2019
29027 root      20   0   14.7g  13.8g   4516 S   0.0 89.2 292:46.45 glusterfs
Thu Aug  1 08:53:41 IST 2019
29027 root      20   0   14.7g  13.8g   4516 S   0.0 89.3 292:47.55 glusterfs
Thu Aug  1 08:55:41 IST 2019
29027 root      20   0   14.7g  13.9g   4516 S   0.0 89.5 292:48.66 glusterfs
Thu Aug  1 08:57:41 IST 2019
29027 root      20   0   14.8g  13.9g   4368 S   0.0 89.6 292:49.82 glusterfs
Thu Aug  1 08:59:41 IST 2019
29027 root      20   0   14.8g  13.9g   3296 S   0.0 89.7 292:50.96 glusterfs
Thu Aug  1 09:01:42 IST 2019
29027 root      20   0   14.8g  13.9g   3296 S   0.0 89.9 292:52.04 glusterfs
Thu Aug  1 09:03:42 IST 2019
29027 root      20   0   14.9g  13.9g   3296 S   0.0 90.0 292:53.21 glusterfs
Thu Aug  1 09:05:42 IST 2019
29027 root      20   0   14.9g  14.0g   2980 S   0.0 90.1 292:54.32 glusterfs
Thu Aug  1 09:07:42 IST 2019
29027 root      20   0   14.9g  14.0g   2980 S   0.0 90.2 292:55.46 glusterfs
Thu Aug  1 09:09:43 IST 2019
29027 root      20   0   14.9g  14.0g   2980 S   0.0 90.4 292:56.50 glusterfs
Thu Aug  1 09:11:43 IST 2019
29027 root      20   0   14.9g  14.0g   2980 S   0.0 90.5 292:57.55 glusterfs
Thu Aug  1 09:13:43 IST 2019
29027 root      20   0   14.9g  14.0g   2980 S   0.0 90.6 292:58.62 glusterfs
Thu Aug  1 09:15:43 IST 2019
29027 root      20   0   15.0g  14.1g   2872 S   0.0 90.8 292:59.71 glusterfs
Thu Aug  1 09:17:43 IST 2019
29027 root      20   0   15.0g  14.1g   2392 S   0.0 90.9 293:00.78 glusterfs
Thu Aug  1 09:19:44 IST 2019
29027 root      20   0   15.0g  14.1g   2392 S   0.0 91.0 293:01.79 glusterfs
Thu Aug  1 09:21:44 IST 2019
29027 root      20   0   15.0g  14.1g   1624 S   0.0 91.1 293:02.90 glusterfs
Thu Aug  1 09:23:46 IST 2019
29027 root      20   0   15.1g  14.1g   1624 S   0.0 91.2 293:03.99 glusterfs
Thu Aug  1 09:25:46 IST 2019
29027 root      20   0   15.1g  14.1g   1616 S   0.0 91.3 293:05.08 glusterfs
Thu Aug  1 09:27:46 IST 2019
29027 root      20   0   15.1g  14.1g   1616 S   0.0 90.8 293:06.18 glusterfs
Thu Aug  1 09:29:47 IST 2019
29027 root      20   0   15.1g  14.1g   1616 S   0.0 91.0 293:07.25 glusterfs
Thu Aug  1 09:31:47 IST 2019
29027 root      20   0   15.1g  14.1g   1616 S   0.0 91.1 293:08.32 glusterfs
Thu Aug  1 09:33:47 IST 2019
29027 root      20   0   15.1g  14.1g   1616 S   0.0 91.2 293:09.43 glusterfs
Thu Aug  1 09:35:47 IST 2019
29027 root      20   0   15.1g  14.1g   1616 S   0.0 91.3 293:10.54 glusterfs
Thu Aug  1 09:37:47 IST 2019
29027 root      20   0   15.2g  14.2g   1616 S   0.0 91.4 293:11.62 glusterfs
Thu Aug  1 09:39:48 IST 2019
29027 root      20   0   15.2g  14.0g   1616 S   0.0 90.6 293:12.81 glusterfs
Thu Aug  1 09:41:48 IST 2019
29027 root      20   0   15.2g  14.0g   1616 S   0.0 90.7 293:13.90 glusterfs
Thu Aug  1 09:43:48 IST 2019
29027 root      20   0   15.2g  14.1g   1616 S   0.0 90.8 293:14.92 glusterfs
Thu Aug  1 09:45:48 IST 2019
29027 root      20   0   15.2g  14.1g   1616 S   0.0 91.0 293:15.97 glusterfs
Thu Aug  1 09:47:49 IST 2019
29027 root      20   0   15.2g  14.1g   1616 S   0.0 91.1 293:17.02 glusterfs
Thu Aug  1 09:49:49 IST 2019
29027 root      20   0   15.3g  14.1g   1616 S   5.3 91.2 293:18.11 glusterfs
Thu Aug  1 09:51:49 IST 2019
29027 root      20   0   15.3g  14.1g   1616 S   0.0 91.3 293:19.13 glusterfs
Thu Aug  1 09:53:49 IST 2019
29027 root      20   0   15.3g  14.2g   1616 S   0.0 91.4 293:20.24 glusterfs
Thu Aug  1 09:55:50 IST 2019
29027 root      20   0   15.3g  14.2g   1616 S   0.0 91.6 293:21.28 glusterfs
Thu Aug  1 09:57:50 IST 2019
29027 root      20   0   15.4g  14.1g   1616 S   0.0 91.2 293:22.35 glusterfs
Thu Aug  1 09:59:50 IST 2019
29027 root      20   0   15.4g  14.1g   1616 S   0.0 91.3 293:23.41 glusterfs
Thu Aug  1 10:01:50 IST 2019
29027 root      20   0   15.4g  14.2g   1616 S   0.0 91.4 293:24.46 glusterfs
Thu Aug  1 10:03:50 IST 2019
29027 root      20   0   15.4g  14.2g   1616 S   0.0 91.6 293:25.48 glusterfs
Thu Aug  1 10:05:51 IST 2019
29027 root      20   0   15.4g  14.2g   1616 S   0.0 91.7 293:26.50 glusterfs
Thu Aug  1 10:07:51 IST 2019
29027 root      20   0   15.4g  14.2g   1616 S   0.0 91.8 293:27.59 glusterfs
Thu Aug  1 10:09:51 IST 2019
29027 root      20   0   15.5g  14.2g   1616 S   0.0 91.9 293:28.65 glusterfs
Thu Aug  1 10:11:51 IST 2019
29027 root      20   0   15.5g  14.2g   1616 S   0.0 91.4 293:29.79 glusterfs
Thu Aug  1 10:13:52 IST 2019
29027 root      20   0   15.5g  14.2g   1616 S   0.0 91.6 293:30.90 glusterfs
Thu Aug  1 10:15:52 IST 2019
29027 root      20   0   15.6g  14.2g   1612 S   0.0 91.7 293:32.03 glusterfs
Thu Aug  1 10:17:53 IST 2019
29027 root      20   0   15.6g  14.2g   1612 S   0.0 91.8 293:33.15 glusterfs
Thu Aug  1 10:19:53 IST 2019
29027 root      20   0   15.6g  14.2g   1612 S   0.0 91.8 293:34.22 glusterfs
Thu Aug  1 10:21:53 IST 2019
29027 root      20   0   15.6g  14.2g   1612 S   0.0 91.9 293:35.32 glusterfs
Thu Aug  1 10:23:54 IST 2019
29027 root      20   0   15.6g  14.2g   1612 S   0.0 92.0 293:36.38 glusterfs
Thu Aug  1 10:25:54 IST 2019
29027 root      20   0   15.6g  14.3g   1612 S   0.0 92.1 293:37.45 glusterfs
Thu Aug  1 10:27:54 IST 2019
29027 root      20   0   15.6g  14.3g   1612 S   0.0 92.3 293:38.55 glusterfs
Thu Aug  1 10:29:55 IST 2019
29027 root      20   0   15.7g  14.1g   1612 S   0.0 91.2 293:39.65 glusterfs
Thu Aug  1 10:31:55 IST 2019
29027 root      20   0   15.7g  14.1g   1612 S   0.0 91.3 293:40.70 glusterfs
Thu Aug  1 10:33:55 IST 2019
29027 root      20   0   15.7g  14.2g   1612 S   0.0 91.5 293:41.80 glusterfs
Thu Aug  1 10:35:55 IST 2019
29027 root      20   0   15.7g  14.2g   1612 S   0.0 91.6 293:42.89 glusterfs
Thu Aug  1 10:37:56 IST 2019
29027 root      20   0   15.7g  14.2g   1528 S   6.2 91.7 293:44.03 glusterfs
Thu Aug  1 10:39:56 IST 2019
29027 root      20   0   15.7g  14.2g   1528 S   0.0 91.9 293:45.12 glusterfs
Thu Aug  1 10:41:56 IST 2019
29027 root      20   0   15.8g  14.2g   1528 S   5.3 92.0 293:46.21 glusterfs
Thu Aug  1 10:43:56 IST 2019
29027 root      20   0   15.8g  14.3g   1528 S   0.0 92.1 293:47.33 glusterfs
Thu Aug  1 10:45:57 IST 2019
29027 root      20   0   15.8g  14.3g   1524 S   0.0 92.2 293:48.42 glusterfs
Thu Aug  1 10:47:57 IST 2019
29027 root      20   0   15.8g  14.1g   1524 S   0.0 91.2 293:49.53 glusterfs
Thu Aug  1 10:49:57 IST 2019
29027 root      20   0   15.9g  14.1g   1524 S   0.0 91.3 293:50.59 glusterfs
Thu Aug  1 10:51:57 IST 2019
29027 root      20   0   15.9g  14.2g   1524 S   0.0 91.4 293:51.61 glusterfs
Thu Aug  1 10:53:58 IST 2019
29027 root      20   0   15.9g  14.2g   1524 S   0.0 91.5 293:52.69 glusterfs
Thu Aug  1 10:55:58 IST 2019
29027 root      20   0   15.9g  14.2g   1524 S   6.2 91.7 293:53.78 glusterfs
Thu Aug  1 10:57:58 IST 2019
29027 root      20   0   15.9g  14.2g   1524 S   0.0 91.8 293:54.94 glusterfs
Thu Aug  1 10:59:58 IST 2019
29027 root      20   0   15.9g  14.2g   1524 S   0.0 91.9 293:56.15 glusterfs
Thu Aug  1 11:01:59 IST 2019
29027 root      20   0   16.0g  14.3g   1524 S   6.2 92.1 293:57.44 glusterfs
Thu Aug  1 11:03:59 IST 2019
29027 root      20   0   16.0g  14.3g   1524 S   0.0 92.2 293:58.67 glusterfs
Thu Aug  1 11:05:59 IST 2019
29027 root      20   0   16.0g  14.3g   1524 S   0.0 92.3 293:59.84 glusterfs
Thu Aug  1 11:07:59 IST 2019
29027 root      20   0   16.1g  14.3g   1524 S   0.0 92.5 294:01.12 glusterfs
Thu Aug  1 11:10:00 IST 2019
29027 root      20   0   16.1g  14.3g   1524 S   0.0 92.6 294:02.32 glusterfs
Thu Aug  1 11:12:00 IST 2019
29027 root      20   0   16.1g  14.3g   1524 S   0.0 92.3 294:03.74 glusterfs
Thu Aug  1 11:14:00 IST 2019
29027 root      20   0   16.1g  14.3g   1524 S   0.0 92.5 294:05.12 glusterfs
Thu Aug  1 11:16:00 IST 2019
29027 root      20   0   16.1g  14.3g   1524 S   0.0 92.6 294:06.47 glusterfs
Thu Aug  1 11:18:01 IST 2019
29027 root      20   0   16.2g  14.3g   1524 S   0.0 92.6 294:07.61 glusterfs
Thu Aug  1 11:20:01 IST 2019
29027 root      20   0   16.2g  14.4g   1524 S   0.0 92.7 294:08.69 glusterfs
Thu Aug  1 11:22:01 IST 2019
29027 root      20   0   16.2g  14.4g   1524 S   0.0 92.8 294:09.72 glusterfs
Thu Aug  1 11:24:01 IST 2019
29027 root      20   0   16.2g  14.3g   1524 S   6.2 92.7 294:10.74 glusterfs
Thu Aug  1 11:26:02 IST 2019
29027 root      20   0   16.2g  14.4g   1524 S   0.0 92.8 294:11.80 glusterfs
Thu Aug  1 11:28:02 IST 2019
29027 root      20   0   16.2g  14.4g   1524 S   0.0 92.9 294:12.83 glusterfs
Thu Aug  1 11:30:02 IST 2019
29027 root      20   0   16.2g  14.4g   1524 S   0.0 92.8 294:13.91 glusterfs
Thu Aug  1 11:32:02 IST 2019
29027 root      20   0   16.3g  14.4g   1524 S   0.0 92.9 294:14.95 glusterfs
Thu Aug  1 11:34:03 IST 2019
29027 root      20   0   16.3g  14.3g    956 S   0.0 92.6 294:16.12 glusterfs
Thu Aug  1 11:36:03 IST 2019
29027 root      20   0   16.3g  14.4g    956 S   0.0 92.7 294:17.18 glusterfs
Thu Aug  1 11:38:03 IST 2019
29027 root      20   0   16.4g  14.4g    956 S   0.0 92.8 294:18.26 glusterfs
Thu Aug  1 11:40:04 IST 2019
29027 root      20   0   16.4g  14.4g    964 S   0.0 93.0 294:19.34 glusterfs
Thu Aug  1 11:42:04 IST 2019
29027 root      20   0   16.4g  14.3g    936 S   0.0 92.2 294:20.43 glusterfs
Thu Aug  1 11:44:05 IST 2019
29027 root      20   0   16.4g  14.3g    976 S   0.0 92.3 294:21.45 glusterfs
Thu Aug  1 11:46:05 IST 2019
29027 root      20   0   16.4g  14.3g    976 S   0.0 92.1 294:22.52 glusterfs
Thu Aug  1 11:48:05 IST 2019
29027 root      20   0   16.4g  14.3g    976 S   0.0 92.2 294:23.61 glusterfs
Thu Aug  1 11:50:05 IST 2019
29027 root      20   0   16.4g  14.3g    976 S   0.0 92.3 294:24.70 glusterfs
Thu Aug  1 11:52:06 IST 2019
29027 root      20   0   16.5g  14.3g    976 S   0.0 92.4 294:25.81 glusterfs
Thu Aug  1 11:54:06 IST 2019
29027 root      20   0   16.5g  14.3g    976 S   0.0 92.6 294:26.86 glusterfs
Thu Aug  1 11:56:06 IST 2019
29027 root      20   0   16.5g  14.3g    980 S   6.2 92.7 294:27.95 glusterfs
Thu Aug  1 11:58:06 IST 2019
29027 root      20   0   16.6g  14.4g    980 S   0.0 92.7 294:29.04 glusterfs
Thu Aug  1 12:00:06 IST 2019
29027 root      20   0   16.6g  14.4g    980 S   0.0 92.8 294:30.08 glusterfs
Thu Aug  1 12:02:07 IST 2019
29027 root      20   0   16.6g  14.4g    976 S   0.0 92.8 294:31.23 glusterfs
Thu Aug  1 12:04:07 IST 2019
29027 root      20   0   16.6g  14.4g    976 S   0.0 92.9 294:32.34 glusterfs
Thu Aug  1 12:06:07 IST 2019
29027 root      20   0   16.6g  14.4g    976 S   0.0 93.0 294:33.40 glusterfs
Thu Aug  1 12:08:08 IST 2019
29027 root      20   0   16.6g  14.4g    964 S   6.7 93.0 294:34.49 glusterfs
Thu Aug  1 12:10:08 IST 2019
29027 root      20   0   16.6g  14.4g    984 S   0.0 93.0 294:35.61 glusterfs
Thu Aug  1 12:12:09 IST 2019
29027 root      20   0   16.7g  14.4g    980 S   6.2 93.0 294:36.67 glusterfs
Thu Aug  1 12:14:09 IST 2019
29027 root      20   0   16.7g  14.3g    980 S   0.0 92.1 294:37.72 glusterfs
Thu Aug  1 12:16:10 IST 2019
29027 root      20   0   16.7g  14.2g    216 S   0.0 91.5 294:38.84 glusterfs
Thu Aug  1 12:18:10 IST 2019
29027 root      20   0   16.7g  14.2g    220 S   6.2 91.6 294:40.00 glusterfs
Thu Aug  1 12:20:10 IST 2019
29027 root      20   0   16.7g  14.2g    748 S   0.0 91.8 294:41.37 glusterfs
Thu Aug  1 12:22:10 IST 2019
29027 root      20   0   16.7g  14.2g    844 S   0.0 91.9 294:42.55 glusterfs
Thu Aug  1 12:24:10 IST 2019
29027 root      20   0   16.8g  14.3g    844 S   0.0 92.1 294:43.81 glusterfs
Thu Aug  1 12:26:11 IST 2019
29027 root      20   0   16.8g  14.3g    912 S   0.0 92.2 294:45.00 glusterfs
Thu Aug  1 12:28:11 IST 2019
29027 root      20   0   16.8g  14.3g    912 S   0.0 92.3 294:46.14 glusterfs
Thu Aug  1 12:30:11 IST 2019
29027 root      20   0   16.9g  14.3g    912 S   0.0 92.5 294:47.21 glusterfs
Thu Aug  1 12:32:11 IST 2019
29027 root      20   0   16.9g  14.2g    912 S   0.0 92.0 294:48.31 glusterfs
Thu Aug  1 12:34:12 IST 2019
29027 root      20   0   16.9g  14.3g    912 S   0.0 92.2 294:49.42 glusterfs
Thu Aug  1 12:36:12 IST 2019
29027 root      20   0   16.9g  14.3g    912 S   0.0 92.3 294:50.45 glusterfs
Thu Aug  1 12:38:12 IST 2019
29027 root      20   0   16.9g  14.3g    908 S   0.0 92.4 294:51.53 glusterfs
Thu Aug  1 12:40:13 IST 2019
29027 root      20   0   16.9g  14.3g    908 S   0.0 92.5 294:52.57 glusterfs
Thu Aug  1 12:42:13 IST 2019
29027 root      20   0   17.0g  14.3g    892 S   6.7 92.7 294:53.62 glusterfs
Thu Aug  1 12:44:13 IST 2019
29027 root      20   0   17.0g  14.4g    892 S   0.0 92.8 294:54.62 glusterfs
Thu Aug  1 12:46:13 IST 2019
29027 root      20   0   17.0g  14.4g   1468 S   0.0 92.9 294:56.48 glusterfs
Thu Aug  1 12:48:14 IST 2019
29027 root      20   0   17.0g  14.2g      0 S   0.0 91.6 294:57.61 glusterfs
Thu Aug  1 12:50:14 IST 2019
29027 root      20   0   17.1g  14.2g      0 S   0.0 91.7 294:58.60 glusterfs
Thu Aug  1 12:52:14 IST 2019
29027 root      20   0   17.1g  14.2g      0 S   6.2 91.9 294:59.64 glusterfs
Thu Aug  1 12:54:15 IST 2019
29027 root      20   0   17.1g  14.2g      0 S   0.0 92.0 295:00.70 glusterfs
Thu Aug  1 12:56:15 IST 2019
29027 root      20   0   17.1g  14.3g      0 S   0.0 92.1 295:01.78 glusterfs
Thu Aug  1 12:58:15 IST 2019
29027 root      20   0   17.1g  14.3g      0 S   0.0 92.2 295:02.84 glusterfs
Thu Aug  1 13:00:15 IST 2019
29027 root      20   0   17.1g  14.3g      0 S   0.0 92.3 295:03.90 glusterfs
Thu Aug  1 13:02:15 IST 2019
29027 root      20   0   17.1g  14.3g      0 S   0.0 92.4 295:04.99 glusterfs
Thu Aug  1 13:04:16 IST 2019
29027 root      20   0   17.2g  14.3g      0 S   0.0 92.5 295:06.03 glusterfs
Thu Aug  1 13:06:16 IST 2019
29027 root      20   0   17.2g  14.3g      0 S   6.2 92.7 295:07.14 glusterfs
Thu Aug  1 13:08:16 IST 2019
29027 root      20   0   17.2g  14.4g      0 S   0.0 92.8 295:08.23 glusterfs
Thu Aug  1 13:10:16 IST 2019
29027 root      20   0   17.2g  14.4g      0 S   0.0 92.9 295:09.29 glusterfs
Thu Aug  1 13:12:17 IST 2019
29027 root      20   0   17.2g  14.4g      0 S   0.0 93.0 295:10.43 glusterfs
Thu Aug  1 13:14:17 IST 2019
29027 root      20   0   17.2g  14.4g      0 S   0.0 93.2 295:11.57 glusterfs
Thu Aug  1 13:16:17 IST 2019
29027 root      20   0   17.3g  14.4g      0 S   0.0 93.3 295:12.91 glusterfs
Thu Aug  1 13:18:17 IST 2019
29027 root      20   0   17.3g  14.5g      0 S   0.0 93.5 295:14.37 glusterfs
Thu Aug  1 13:20:18 IST 2019
29027 root      20   0   17.4g  14.4g      0 S   0.0 93.3 295:15.80 glusterfs
Thu Aug  1 13:22:18 IST 2019
29027 root      20   0   17.4g  14.5g      0 S   0.0 93.4 295:17.15 glusterfs
Thu Aug  1 13:24:18 IST 2019
29027 root      20   0   17.4g  14.4g      0 S   5.9 92.7 295:18.44 glusterfs
Thu Aug  1 13:26:19 IST 2019
29027 root      20   0   17.4g  14.4g      0 S   0.0 92.8 295:19.61 glusterfs
Thu Aug  1 13:28:19 IST 2019
29027 root      20   0   17.4g  14.4g      0 S   0.0 93.0 295:20.78 glusterfs
Thu Aug  1 13:30:19 IST 2019
29027 root      20   0   17.4g  14.4g      0 S   0.0 93.1 295:21.96 glusterfs
Thu Aug  1 13:32:20 IST 2019
29027 root      20   0   17.5g  14.4g      0 S   0.0 92.8 295:23.16 glusterfs
Thu Aug  1 13:34:20 IST 2019
29027 root      20   0   17.5g  14.4g      0 S   0.0 92.9 295:24.36 glusterfs
Thu Aug  1 13:36:20 IST 2019
29027 root      20   0   17.5g  14.4g      0 S   0.0 93.1 295:25.54 glusterfs
Thu Aug  1 13:38:20 IST 2019
29027 root      20   0   18.9g  14.6g    116 S   0.0 94.1 296:43.52 glusterfs
Thu Aug  1 16:04:40 IST 2019
29027 root      20   0   18.9g  14.5g    116 S   0.0 93.4 296:44.58 glusterfs
Thu Aug  1 16:06:40 IST 2019
29027 root      20   0   18.9g  14.5g    116 S   0.0 93.6 296:45.58 glusterfs
Thu Aug  1 16:08:41 IST 2019
29027 root      20   0   18.9g  14.5g    116 S   0.0 93.7 296:46.64 glusterfs
Thu Aug  1 16:10:41 IST 2019
29027 root      20   0   18.9g  14.5g    116 S   0.0 93.8 296:47.74 glusterfs
Thu Aug  1 16:12:41 IST 2019
29027 root      20   0   19.0g  14.5g    116 S   0.0 93.9 296:48.82 glusterfs
Thu Aug  1 16:14:41 IST 2019
29027 root      20   0   19.0g  14.6g    116 S   0.0 94.0 296:49.87 glusterfs
Thu Aug  1 16:16:42 IST 2019
29027 root      20   0   19.0g  14.4g      0 S   0.0 93.2 296:50.98 glusterfs
Thu Aug  1 16:18:42 IST 2019
29027 root      20   0   19.1g  14.5g      0 S   0.0 93.3 296:52.03 glusterfs
Thu Aug  1 16:20:42 IST 2019
29027 root      20   0   19.1g  14.5g      0 S   6.2 93.5 296:53.09 glusterfs
Thu Aug  1 16:22:43 IST 2019
29027 root      20   0   19.1g  14.5g      0 S   0.0 93.6 296:54.12 glusterfs
Thu Aug  1 16:24:43 IST 2019
29027 root      20   0   19.1g  14.5g      0 S   6.2 93.7 296:55.17 glusterfs
Thu Aug  1 16:26:43 IST 2019
29027 root      20   0   19.1g  14.5g      0 S   0.0 93.8 296:56.21 glusterfs
Thu Aug  1 16:28:43 IST 2019
29027 root      20   0   19.1g  14.6g      0 S   0.0 94.0 296:57.27 glusterfs
Thu Aug  1 16:30:43 IST 2019
29027 root      20   0   19.2g  14.6g      0 S   0.0 94.1 296:58.37 glusterfs
Thu Aug  1 16:32:44 IST 2019
29027 root      20   0   19.2g  14.6g      0 S   0.0 94.2 296:59.46 glusterfs
Thu Aug  1 16:34:44 IST 2019
29027 root      20   0   19.2g  14.6g      0 S   0.0 94.2 297:00.54 glusterfs
Thu Aug  1 16:36:44 IST 2019
29027 root      20   0   19.2g  14.6g      0 S   0.0 94.2 297:01.65 glusterfs
Thu Aug  1 16:38:45 IST 2019
29027 root      20   0   19.2g  14.6g      0 S   0.0 94.4 297:02.73 glusterfs
Thu Aug  1 16:40:45 IST 2019
29027 root      20   0   19.2g  14.6g      0 S   0.0 94.5 297:03.83 glusterfs
Thu Aug  1 16:42:45 IST 2019
29027 root      20   0   19.2g  14.4g      0 S   0.0 93.3 297:04.90 glusterfs
Thu Aug  1 16:44:45 IST 2019
29027 root      20   0   19.3g  14.5g      0 S   0.0 93.5 297:05.99 glusterfs
Thu Aug  1 16:46:45 IST 2019
29027 root      20   0   19.3g  14.5g      0 S   0.0 93.6 297:07.09 glusterfs
Thu Aug  1 16:48:46 IST 2019
29027 root      20   0   19.3g  14.5g      0 S   0.0 93.7 297:08.18 glusterfs
Thu Aug  1 16:50:46 IST 2019
29027 root      20   0   19.4g  14.5g      0 S   0.0 93.9 297:09.28 glusterfs
Thu Aug  1 16:52:46 IST 2019
29027 root      20   0   19.4g  14.5g      0 S   0.0 94.0 297:10.35 glusterfs
Thu Aug  1 16:54:46 IST 2019
29027 root      20   0   19.4g  14.6g      0 S   0.0 94.1 297:11.47 glusterfs
Thu Aug  1 16:56:47 IST 2019
29027 root      20   0   19.4g  14.6g      0 S   6.2 94.2 297:12.52 glusterfs
Thu Aug  1 16:58:47 IST 2019
29027 root      20   0   19.4g  14.6g      0 S   0.0 94.2 297:13.60 glusterfs
Thu Aug  1 17:00:47 IST 2019
29027 root      20   0   19.4g  14.6g      0 S   0.0 94.4 297:14.67 glusterfs
Thu Aug  1 17:02:47 IST 2019
29027 root      20   0   19.5g  14.6g      0 S   0.0 94.3 297:15.81 glusterfs
Thu Aug  1 17:04:47 IST 2019
29027 root      20   0   19.5g  14.6g      0 S   0.0 94.5 297:16.90 glusterfs
Thu Aug  1 17:06:48 IST 2019
29027 root      20   0   19.5g  14.6g      0 S   0.0 94.4 297:18.01 glusterfs
Thu Aug  1 17:08:48 IST 2019
29027 root      20   0   19.6g  14.6g      0 S   0.0 94.4 297:19.08 glusterfs
Thu Aug  1 17:10:48 IST 2019
29027 root      20   0   19.6g  14.6g      0 S   0.0 94.3 297:20.15 glusterfs
Thu Aug  1 17:12:49 IST 2019
29027 root      20   0   19.6g  14.6g      0 S   0.0 94.4 297:21.19 glusterfs
Thu Aug  1 17:14:49 IST 2019
29027 root      20   0   19.6g  14.6g      0 S   6.2 94.4 297:22.19 glusterfs
Thu Aug  1 17:16:49 IST 2019
29027 root      20   0   19.6g  14.6g      0 S   0.0 94.3 297:23.25 glusterfs
Thu Aug  1 17:18:49 IST 2019
29027 root      20   0   19.6g  14.6g      0 S   0.0 94.4 297:24.32 glusterfs
Thu Aug  1 17:20:50 IST 2019
29027 root      20   0   19.6g  14.6g      0 S   0.0 94.0 297:25.40 glusterfs
Thu Aug  1 17:22:50 IST 2019
29027 root      20   0   19.7g  14.6g      0 S   0.0 94.2 297:26.43 glusterfs
Thu Aug  1 17:24:50 IST 2019
29027 root      20   0   19.7g  14.6g      0 S   0.0 94.1 297:27.46 glusterfs
Thu Aug  1 17:26:50 IST 2019
29027 root      20   0   19.7g  14.6g      0 S   0.0 94.3 297:28.53 glusterfs
Thu Aug  1 17:28:50 IST 2019
29027 root      20   0   19.7g  14.6g      0 S   0.0 94.1 297:29.60 glusterfs
Thu Aug  1 17:30:51 IST 2019
29027 root      20   0   19.7g  14.6g      0 S   0.0 94.2 297:30.60 glusterfs
Thu Aug  1 17:32:51 IST 2019
29027 root      20   0   19.7g  14.6g      0 S   6.2 94.3 297:31.66 glusterfs
Thu Aug  1 17:34:51 IST 2019
29027 root      20   0   19.8g  14.6g      0 S   0.0 94.4 297:32.70 glusterfs
Thu Aug  1 17:36:51 IST 2019
29027 root      20   0   19.8g  14.6g      0 S   0.0 94.4 297:33.80 glusterfs
Thu Aug  1 17:38:52 IST 2019
29027 root      20   0   19.8g  14.6g      0 S   0.0 94.5 297:34.75 glusterfs
Thu Aug  1 17:40:52 IST 2019
29027 root      20   0   22.2g  14.5g    752 S   0.0 93.7 300:50.14 glusterfs
Thu Aug  1 22:55:49 IST 2019
29027 root      20   0   22.2g  14.5g    752 S   0.0 93.8 300:51.15 glusterfs
Thu Aug  1 22:57:49 IST 2019
29027 root      20   0   22.3g  14.4g    752 S   0.0 93.1 300:52.21 glusterfs
Thu Aug  1 22:59:49 IST 2019
29027 root      20   0   22.3g  14.4g    752 S   6.2 93.2 300:53.25 glusterfs
Thu Aug  1 23:01:49 IST 2019
29027 root      20   0   22.3g  14.4g    752 S   6.2 93.3 300:54.42 glusterfs
Thu Aug  1 23:03:49 IST 2019
29027 root      20   0   22.4g  14.5g    752 S   0.0 93.4 300:55.48 glusterfs
Thu Aug  1 23:05:50 IST 2019
29027 root      20   0   22.4g  14.5g    752 S   6.2 93.4 300:56.52 glusterfs
Thu Aug  1 23:07:50 IST 2019
29027 root      20   0   22.4g  14.5g    752 S   0.0 93.5 300:57.55 glusterfs
Thu Aug  1 23:09:50 IST 2019
29027 root      20   0   22.4g  14.5g    752 S   0.0 93.6 300:58.59 glusterfs
Thu Aug  1 23:11:50 IST 2019
29027 root      20   0   22.4g  14.5g    752 S   0.0 93.8 300:59.75 glusterfs
Thu Aug  1 23:13:50 IST 2019
29027 root      20   0   22.4g  14.5g    752 S   0.0 93.7 301:00.77 glusterfs
Thu Aug  1 23:15:51 IST 2019
29027 root      20   0   22.4g  14.5g    752 S   0.0 93.7 301:01.85 glusterfs
Thu Aug  1 23:17:51 IST 2019
29027 root      20   0   22.5g  14.5g    752 S   6.2 93.9 301:02.91 glusterfs
Thu Aug  1 23:19:51 IST 2019
29027 root      20   0   22.5g  14.3g    752 S   0.0 92.5 301:03.97 glusterfs
Thu Aug  1 23:21:51 IST 2019
29027 root      20   0   22.5g  14.3g    752 S   0.0 92.6 301:05.15 glusterfs
Thu Aug  1 23:23:51 IST 2019
29027 root      20   0   22.6g  14.4g    752 S   0.0 92.7 301:06.20 glusterfs
Thu Aug  1 23:25:52 IST 2019
29027 root      20   0   22.6g  14.4g    752 S   0.0 92.8 301:07.23 glusterfs
Thu Aug  1 23:27:52 IST 2019
29027 root      20   0   22.6g  14.4g    752 S   6.2 92.9 301:08.27 glusterfs
Thu Aug  1 23:29:52 IST 2019
29027 root      20   0   22.6g  14.4g    752 S   0.0 93.0 301:09.25 glusterfs
Thu Aug  1 23:31:52 IST 2019
29027 root      20   0   22.6g  14.4g    752 S   0.0 93.2 301:10.45 glusterfs
Thu Aug  1 23:33:52 IST 2019
29027 root      20   0   22.6g  14.4g    752 S   0.0 93.3 301:11.57 glusterfs
Thu Aug  1 23:35:53 IST 2019
29027 root      20   0   22.7g  14.5g    752 S   0.0 93.4 301:12.67 glusterfs
Thu Aug  1 23:37:53 IST 2019
29027 root      20   0   22.7g  14.5g    752 S   0.0 93.6 301:13.78 glusterfs
Thu Aug  1 23:39:53 IST 2019
29027 root      20   0   22.7g  14.5g    776 S   0.0 93.7 301:14.86 glusterfs
Thu Aug  1 23:41:53 IST 2019
29027 root      20   0   22.7g  14.5g    776 S   0.0 93.8 301:16.23 glusterfs
Thu Aug  1 23:43:54 IST 2019
29027 root      20   0   22.7g  14.5g    776 S   0.0 94.0 301:17.53 glusterfs
Thu Aug  1 23:45:54 IST 2019
29027 root      20   0   22.7g  14.4g    840 S   0.0 93.2 301:18.58 glusterfs
Thu Aug  1 23:47:54 IST 2019
29027 root      20   0   22.8g  14.5g    840 S   0.0 93.3 301:19.64 glusterfs
Thu Aug  1 23:49:54 IST 2019
29027 root      20   0   22.8g  14.5g    840 S   0.0 93.5 301:20.72 glusterfs
Thu Aug  1 23:51:54 IST 2019
29027 root      20   0   22.8g  14.5g    840 S   0.0 93.5 301:21.94 glusterfs
Thu Aug  1 23:53:55 IST 2019
29027 root      20   0   22.8g  14.5g    840 S   6.2 93.6 301:23.00 glusterfs
Thu Aug  1 23:55:55 IST 2019
29027 root      20   0   22.9g  14.5g    840 S   0.0 93.5 301:24.05 glusterfs
Thu Aug  1 23:57:55 IST 2019
29027 root      20   0   22.9g  14.5g    840 S   0.0 93.7 301:25.14 glusterfs
Thu Aug  1 23:59:56 IST 2019
29027 root      20   0   22.9g  14.5g    840 S   0.0 93.6 301:26.19 glusterfs
Fri Aug  2 00:01:56 IST 2019
29027 root      20   0   22.9g  14.5g    872 S   0.0 93.6 301:27.42 glusterfs
Fri Aug  2 00:03:56 IST 2019
29027 root      20   0   22.9g  14.5g    880 S   0.0 93.8 301:28.59 glusterfs
Fri Aug  2 00:05:56 IST 2019
29027 root      20   0   22.8g  14.5g    508 S   0.0 93.7 301:29.74 glusterfs
Fri Aug  2 00:07:57 IST 2019
29027 root      20   0   22.9g  14.5g    508 S   0.0 93.9 301:30.88 glusterfs
Fri Aug  2 00:09:58 IST 2019
29027 root      20   0   22.9g  14.5g      0 S   0.0 94.0 301:32.03 glusterfs
Fri Aug  2 00:12:07 IST 2019
29027 root      20   0   22.9g  14.6g      0 S   0.1 94.1 301:33.51 glusterfs
Fri Aug  2 00:15:23 IST 2019
29027 root      20   0   22.9g  14.6g      0 S   0.2 94.2 301:38.53 glusterfs
Fri Aug  2 00:29:03 IST 2019
29027 root      20   0       0      0      0 Z  80.0  0.0 301:42.23 glusterfs
Fri Aug  2 00:31:31 IST 2019
Fri Aug  2 00:33:31 IST 2019
Fri Aug  2 00:35:31 IST 2019
Fri Aug  2 00:37:31 IST 2019

Comment 4 Nag Pavan Chilakam 2019-08-02 07:07:09 UTC
Created attachment 1599510 [details]
top-output.txt

Comment 6 Nag Pavan Chilakam 2019-08-02 07:21:15 UTC
sosreports and logs and dumps @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1736830/

Comment 7 Nag Pavan Chilakam 2019-08-02 07:25:55 UTC
root@rhs-gp-srv1 ~]# gluster v get all all
Option                                  Value                                   
------                                  -----                                   
cluster.server-quorum-ratio             51                                      
cluster.enable-shared-storage           disable                                 
cluster.op-version                      70000                                   
cluster.max-op-version                  70000                                   
cluster.brick-multiplex                 enable                                  
cluster.max-bricks-per-process          250                                     
glusterd.vol_count_per_thread           100                                     
cluster.daemon-log-level                INFO                                    
[root@rhs-gp-srv1 ~]# 


[root@rhs-gp-srv1 ~]# gluster  v info
 
Volume Name: ecv
Type: Distributed-Disperse
Volume ID: 6c9ff0c6-0cc4-4ff2-88a6-d411821335a1
Status: Started
Snapshot Count: 0
Number of Bricks: 21 x (4 + 2) = 126
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/ecv-sv1
Brick4: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1
Brick5: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1
Brick6: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/ecv-sv1
Brick7: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2
Brick8: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2
Brick9: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/ecv-sv2
Brick10: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2
Brick11: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2
Brick12: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/ecv-sv2
Brick13: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3
Brick14: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3
Brick15: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/ecv-sv3
Brick16: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3
Brick17: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3
Brick18: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/ecv-sv3
Brick19: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4
Brick20: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4
Brick21: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/ecv-sv4
Brick22: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4
Brick23: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4
Brick24: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/ecv-sv4
Brick25: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5
Brick26: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5
Brick27: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/ecv-sv5
Brick28: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5
Brick29: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5
Brick30: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/ecv-sv5
Brick31: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6
Brick32: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6
Brick33: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/ecv-sv6
Brick34: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6
Brick35: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6
Brick36: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/ecv-sv6
Brick37: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7
Brick38: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7
Brick39: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/ecv-sv7
Brick40: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7
Brick41: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7
Brick42: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/ecv-sv7
Brick43: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8
Brick44: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8
Brick45: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/ecv-sv8
Brick46: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8
Brick47: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8
Brick48: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/ecv-sv8
Brick49: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9
Brick50: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9
Brick51: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/ecv-sv9
Brick52: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9
Brick53: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9
Brick54: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/ecv-sv9
Brick55: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10
Brick56: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10
Brick57: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/ecv-sv10
Brick58: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10
Brick59: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10
Brick60: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/ecv-sv10
Brick61: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11
Brick62: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11
Brick63: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/ecv-sv11
Brick64: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11
Brick65: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11
Brick66: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/ecv-sv11
Brick67: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12
Brick68: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12
Brick69: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/ecv-sv12
Brick70: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12
Brick71: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12
Brick72: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/ecv-sv12
Brick73: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13
Brick74: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13
Brick75: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/ecv-sv13
Brick76: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13
Brick77: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13
Brick78: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick26/ecv-sv13
Brick79: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14
Brick80: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14
Brick81: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick27/ecv-sv14
Brick82: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14
Brick83: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14
Brick84: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick28/ecv-sv14
Brick85: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15
Brick86: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15
Brick87: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick29/ecv-sv15
Brick88: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15
Brick89: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15
Brick90: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick30/ecv-sv15
Brick91: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16
Brick92: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16
Brick93: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick31/ecv-sv16
Brick94: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16
Brick95: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16
Brick96: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick32/ecv-sv16
Brick97: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17
Brick98: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17
Brick99: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick33/ecv-sv17
Brick100: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17
Brick101: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17
Brick102: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick34/ecv-sv17
Brick103: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18
Brick104: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18
Brick105: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick35/ecv-sv18
Brick106: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18
Brick107: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18
Brick108: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick36/ecv-sv18
Brick109: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19
Brick110: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19
Brick111: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick37/ecv-sv19
Brick112: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19
Brick113: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19
Brick114: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick38/ecv-sv19
Brick115: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20
Brick116: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20
Brick117: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick39/ecv-sv20
Brick118: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20
Brick119: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20
Brick120: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick40/ecv-sv20
Brick121: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21
Brick122: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21
Brick123: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick41/ecv-sv21
Brick124: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21
Brick125: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21
Brick126: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick42/ecv-sv21
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
disperse.shd-max-threads: 8
client.event-threads: 8
server.event-threads: 8
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
cluster.brick-multiplex: enable
 
Volume Name: repvol
Type: Replicate
Volume ID: 3e3e865d-4cae-4ccd-9811-3df39b8b7a45
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick43/repvol
Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick43/repvol
Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick43/repvol
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
cluster.brick-multiplex: enable



[root@rhs-gp-srv1 ~]# gluster v status
Status of volume: ecv
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49153     0          Y       17665
Self-heal Daemon on localhost               N/A       N/A        Y       29174
Self-heal Daemon on rhs-gp-srv4.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       18016
Self-heal Daemon on rhs-gp-srv2.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       22399
 
Task Status of Volume ecv
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: repvol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49152     0          Y       28408
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49152     0          Y       21681
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49152     0          Y       17288
Self-heal Daemon on localhost               N/A       N/A        Y       29174
Self-heal Daemon on rhs-gp-srv2.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       22399
Self-heal Daemon on rhs-gp-srv4.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       18016
 
Task Status of Volume repvol
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@rhs-gp-srv1 ~]# 
[root@rhs-gp-srv1 ~]# gluster v status
Status of volume: ecv
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick1/ecv-sv1                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick2/ecv-sv1                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick3/ecv-sv2                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick4/ecv-sv2                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick5/ecv-sv3                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick6/ecv-sv3                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick7/ecv-sv4                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick8/ecv-sv4                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick9/ecv-sv5                       49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick10/ecv-sv5                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick11/ecv-sv6                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick12/ecv-sv6                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick13/ecv-sv7                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick14/ecv-sv7                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick15/ecv-sv8                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick16/ecv-sv8                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick17/ecv-sv9                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick18/ecv-sv9                      49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick19/ecv-sv10                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick20/ecv-sv10                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick21/ecv-sv11                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick22/ecv-sv11                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick23/ecv-sv12                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick24/ecv-sv12                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick25/ecv-sv13                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick26/ecv-sv13                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick27/ecv-sv14                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick28/ecv-sv14                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick29/ecv-sv15                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick30/ecv-sv15                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick31/ecv-sv16                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick32/ecv-sv16                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick33/ecv-sv17                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick34/ecv-sv17                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick35/ecv-sv18                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick36/ecv-sv18                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick37/ecv-sv19                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick38/ecv-sv19                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick39/ecv-sv20                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick40/ecv-sv20                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick41/ecv-sv21                     49153     0          Y       17665
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49153     0          Y       28822
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49153     0          Y       22064
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick42/ecv-sv21                     49153     0          Y       17665
Self-heal Daemon on localhost               N/A       N/A        Y       29174
Self-heal Daemon on rhs-gp-srv4.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       18016
Self-heal Daemon on rhs-gp-srv2.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       22399
 
Task Status of Volume ecv
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: repvol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49152     0          Y       28408
Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49152     0          Y       21681
Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/g
luster/brick43/repvol                       49152     0          Y       17288
Self-heal Daemon on localhost               N/A       N/A        Y       29174
Self-heal Daemon on rhs-gp-srv4.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       18016
Self-heal Daemon on rhs-gp-srv2.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       22399
 
Task Status of Volume repvol
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@rhs-gp-srv1 ~]#

Comment 8 Amar Tumballi 2019-08-05 03:04:09 UTC
Looks like this is the same issue we saw with interrupt leak.. Csaba can you confirm?

Comment 13 Nag Pavan Chilakam 2019-08-10 04:52:36 UTC
have retried on 6.0.11
I ran the test steps as mentioned in the description.
Hit a hang on a client session where lookups were happening after about 6-8 hrs of test
Issued an interrupt and even after 12 hrs I dont see any increase in resident memory, it was constant at about 1.2GB
Hence not seeing the leak anymore as mentioned in description, once interrupt was issued(ctrl+c) on a hung session.
Hence moving to verified

Comment 17 errata-xmlrpc 2019-10-30 12:22:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249