Description of problem: ======================== after hitting the issue captured in BZ#1731826 and details mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1731826#c0 One of the fuse clients of the ec-volume has hung on a "ls" I retried by opening another terminal and issuing "ls" again on same path, and yes the lookup hangs. I retried with a new mount point, and don't see the issue. Version-Release number of selected component (if applicable): ====================================== 6.0.8 Steps to Reproduce: =================== 1.created a 3 node cluster , enabled brickmux 2. created a 1x3 volume "repvol"and 21x(4+2) ecvolume "ecv" {each server hosts 2 bricks of each ec set) 3. set below options to ecvol server.event-threads: 8 client.event-threads: 8 disperse.shd-max-threads: 24 4. mounted both ecv and repvol on 6 different clients 5. top o/p being continously captured for each client in repvol 6. IOs triggered on ecv in below pattern a). linux untar from 2 clients for 50times b). crefi from 2 clients "#for j in {1..20};do for i in {create,chmod,chown,chgrp,symlink,truncate,rename,hardlink}; do crefi --multi -n 5 -b 20 -d 10 --max=1K --min=50 --random -T 2 -t text --fop=$i /mnt/cvlt-ecv/IOs/crefi/$HOSTNAME/ ; sleep 10 ; done;rm -rf /mnt/cvlt-ecv/IOs/crefi/$HOSTNAME/*;done" c). lookups (find *|xargs stat) from all 6 clients 7. after 2.5 days, both a) and b) completed successfully 8. issued a du -sh on the directory hosting linux untar directories from client1 9. stopped lookups on couple of clients 10. reissued crefi from the 2 client , but this time without rm -rf 11. in about 15 min, some of the bricks went offline as below Actual results: ============= at around this time, when I issued ls on one of the clients (rhs-client2), the ls is hung
sosreports and logs same as http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1731826/ logs for client where ls is hung is available @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1731826/clients/rhs-client2.lab.eng.blr.redhat.com-hung-client/
at this point, below are IOs that are happening: client1 -> rhs-client44.lab.eng.blr.redhat.com {find *|xargs stat and du -sh * on one of the directories} client2 -> rhs-client26.lab.eng.blr.redhat.com -->linux untar client3 -> rhs-client45 -> no IOs client4 -> rhs-client22 -> no IOs client5 -> rhs-client12 -> find *|xargs stat ---->SEEMS LIKE HERE IT IS HUNG client6 -> rhs-client2 -> ls is hung the issue which was reported was seen here
ok, the fuse mount got OOM killed finally 40000 [Mon Jul 22 12:14:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:14:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:14:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:14:27 2019] Call Trace: [Mon Jul 22 12:14:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:14:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:14:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:14:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:14:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:14:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:14:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:14:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:14:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:14:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:14:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:14:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:14:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:14:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:14:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:14:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:14:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:14:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:14:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:14:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:14:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:14:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:14:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:16:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:16:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:16:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:16:27 2019] Call Trace: [Mon Jul 22 12:16:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:16:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:16:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:16:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:16:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:16:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:16:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:16:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:16:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:16:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:16:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:16:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:16:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:16:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:16:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:16:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:16:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:16:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:16:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:16:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:16:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:16:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:16:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:18:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:18:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:18:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:18:27 2019] Call Trace: [Mon Jul 22 12:18:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:18:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:18:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:18:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:18:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:18:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:18:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:18:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:18:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:18:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:18:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:18:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:18:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:18:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:18:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:18:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:18:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:18:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:18:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:18:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:18:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:18:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:18:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:20:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:20:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:20:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:20:27 2019] Call Trace: [Mon Jul 22 12:20:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:20:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:20:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:20:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:20:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:20:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:20:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:20:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:20:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:20:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:20:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:20:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:20:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:20:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:20:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:20:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:20:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:20:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:20:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:20:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:20:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:20:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:20:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:22:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:22:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:22:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:22:27 2019] Call Trace: [Mon Jul 22 12:22:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:22:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:22:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:22:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:22:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:22:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:22:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:22:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:22:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:22:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:22:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:22:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:22:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:22:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:22:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:22:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:22:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:22:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:22:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:22:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:22:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:22:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:22:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:24:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:24:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:24:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:24:27 2019] Call Trace: [Mon Jul 22 12:24:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:24:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:24:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:24:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:24:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:24:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:24:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:24:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:24:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:24:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:24:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:24:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:24:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:24:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:24:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:24:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:24:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:24:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:24:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:24:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:24:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:24:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:24:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:26:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:26:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:26:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:26:27 2019] Call Trace: [Mon Jul 22 12:26:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:26:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:26:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:26:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:26:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:26:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:26:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:26:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:26:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:26:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:26:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:26:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:26:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:26:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:26:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:26:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:26:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:26:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:26:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:26:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:26:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:26:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:26:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:28:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:28:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:28:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:28:27 2019] Call Trace: [Mon Jul 22 12:28:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:28:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:28:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:28:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:28:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:28:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:28:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:28:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:28:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:28:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:28:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:28:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:28:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:28:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:28:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:28:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:28:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:28:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:28:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:28:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:28:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:28:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:28:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:30:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:30:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:30:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:30:27 2019] Call Trace: [Mon Jul 22 12:30:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:30:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:30:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:30:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:30:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:30:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:30:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:30:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:30:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:30:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:30:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:30:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:30:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:30:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:30:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:30:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:30:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:30:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:30:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:30:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:30:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:30:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:30:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 12:32:27 2019] INFO: task ls:4479 blocked for more than 120 seconds. [Mon Jul 22 12:32:27 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Jul 22 12:32:27 2019] ls D ffff99fbb5d85230 0 4479 10260 0x00000084 [Mon Jul 22 12:32:27 2019] Call Trace: [Mon Jul 22 12:32:27 2019] [<ffffffff9837f1c9>] schedule+0x29/0x70 [Mon Jul 22 12:32:27 2019] [<ffffffffc08724d5>] __fuse_request_send+0xf5/0x2e0 [fuse] [Mon Jul 22 12:32:27 2019] [<ffffffff97cc61c0>] ? wake_up_atomic_t+0x30/0x30 [Mon Jul 22 12:32:27 2019] [<ffffffffc08726d2>] fuse_request_send+0x12/0x20 [fuse] [Mon Jul 22 12:32:27 2019] [<ffffffffc0877d5a>] fuse_do_getattr+0x10a/0x330 [fuse] [Mon Jul 22 12:32:27 2019] [<ffffffffc0878135>] fuse_permission+0x1b5/0x2b0 [fuse] [Mon Jul 22 12:32:27 2019] [<ffffffff97e54471>] __inode_permission+0x71/0xd0 [Mon Jul 22 12:32:27 2019] [<ffffffff97e544e8>] inode_permission+0x18/0x50 [Mon Jul 22 12:32:27 2019] [<ffffffff97e5854e>] link_path_walk+0x27e/0x8b0 [Mon Jul 22 12:32:27 2019] [<ffffffff97e23826>] ? kmem_cache_alloc_trace+0x1d6/0x200 [Mon Jul 22 12:32:27 2019] [<ffffffff97f0945c>] ? selinux_file_alloc_security+0x3c/0x60 [Mon Jul 22 12:32:27 2019] [<ffffffff97e59c3e>] path_openat+0xae/0x5a0 [Mon Jul 22 12:32:27 2019] [<ffffffff9806989b>] ? n_tty_ioctl_helper+0x2b/0x120 [Mon Jul 22 12:32:27 2019] [<ffffffff98064f55>] ? n_tty_ioctl+0x55/0x100 [Mon Jul 22 12:32:27 2019] [<ffffffff98069c73>] ? tty_ldisc_ref_wait+0x23/0x50 [Mon Jul 22 12:32:27 2019] [<ffffffff98069d29>] ? tty_ldisc_deref+0x19/0x20 [Mon Jul 22 12:32:27 2019] [<ffffffff97e5b5cd>] do_filp_open+0x4d/0xb0 [Mon Jul 22 12:32:27 2019] [<ffffffff97e69087>] ? __alloc_fd+0x47/0x170 [Mon Jul 22 12:32:27 2019] [<ffffffff97e47524>] do_sys_open+0x124/0x220 [Mon Jul 22 12:32:27 2019] [<ffffffff9838be15>] ? system_call_after_swapgs+0xa2/0x146 [Mon Jul 22 12:32:27 2019] [<ffffffff97e47654>] SyS_openat+0x14/0x20 [Mon Jul 22 12:32:27 2019] [<ffffffff9838bede>] system_call_fastpath+0x25/0x2a [Mon Jul 22 12:32:27 2019] [<ffffffff9838be21>] ? system_call_after_swapgs+0xae/0x146 [Mon Jul 22 17:13:17 2019] nr_pdflush_threads exported in /proc is scheduled for removal [Mon Jul 22 17:13:18 2019] MACsec IEEE 802.1AE [Mon Jul 22 17:14:36 2019] warning: `turbostat' uses 32-bit capabilities (legacy support in use) [Tue Jul 23 07:51:36 2019] glfs_timer invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [Tue Jul 23 07:51:36 2019] glfs_timer cpuset=/ mems_allowed=0-1 [Tue Jul 23 07:51:36 2019] CPU: 14 PID: 10340 Comm: glfs_timer Kdump: loaded Not tainted 3.10.0-1061.el7.x86_64 #1 [Tue Jul 23 07:51:36 2019] Hardware name: Supermicro X9DRW-3LN4F+/X9DRW-3TF+/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 1.0b 05/29/2012 [Tue Jul 23 07:51:36 2019] Call Trace: [Tue Jul 23 07:51:36 2019] [<ffffffff98379262>] dump_stack+0x19/0x1b [Tue Jul 23 07:51:36 2019] [<ffffffff98373c04>] dump_header+0x90/0x229 [Tue Jul 23 07:51:36 2019] [<ffffffff97f0825b>] ? cred_has_capability+0x6b/0x120 [Tue Jul 23 07:51:36 2019] [<ffffffff97dbfd74>] oom_kill_process+0x254/0x3e0 [Tue Jul 23 07:51:36 2019] [<ffffffff97f0833e>] ? selinux_capable+0x2e/0x40 [Tue Jul 23 07:51:36 2019] [<ffffffff97dc05c6>] out_of_memory+0x4b6/0x4f0 [Tue Jul 23 07:51:36 2019] [<ffffffff9837471c>] __alloc_pages_slowpath+0x5d6/0x724 [Tue Jul 23 07:51:36 2019] [<ffffffff97dc6b84>] __alloc_pages_nodemask+0x404/0x420 [Tue Jul 23 07:51:36 2019] [<ffffffff97e14c68>] alloc_pages_current+0x98/0x110 [Tue Jul 23 07:51:36 2019] [<ffffffff97dbbc77>] __page_cache_alloc+0x97/0xb0 [Tue Jul 23 07:51:36 2019] [<ffffffff97dbe838>] filemap_fault+0x298/0x490 [Tue Jul 23 07:51:36 2019] [<ffffffffc062b57e>] __xfs_filemap_fault+0x7e/0x1d0 [xfs] [Tue Jul 23 07:51:36 2019] [<ffffffff9829c927>] ? ip_local_out_sk+0x37/0x40 [Tue Jul 23 07:51:36 2019] [<ffffffffc062b77c>] xfs_filemap_fault+0x2c/0x30 [xfs] [Tue Jul 23 07:51:36 2019] [<ffffffff97dea29a>] __do_fault.isra.61+0x8a/0x100 [Tue Jul 23 07:51:36 2019] [<ffffffff97dea84c>] do_read_fault.isra.63+0x4c/0x1b0 [Tue Jul 23 07:51:36 2019] [<ffffffff97def2fa>] handle_pte_fault+0x22a/0xe20 [Tue Jul 23 07:51:36 2019] [<ffffffff97cd6539>] ? ttwu_do_wakeup+0x19/0xe0 [Tue Jul 23 07:51:36 2019] [<ffffffff97cd666f>] ? ttwu_do_activate+0x6f/0x80 [Tue Jul 23 07:51:36 2019] [<ffffffff97cd9df0>] ? try_to_wake_up+0x190/0x390 [Tue Jul 23 07:51:36 2019] [<ffffffff97df200d>] handle_mm_fault+0x39d/0x9b0 [Tue Jul 23 07:51:36 2019] [<ffffffff97cda0c2>] ? default_wake_function+0x12/0x20 [Tue Jul 23 07:51:36 2019] [<ffffffff98386633>] __do_page_fault+0x213/0x500 [Tue Jul 23 07:51:36 2019] [<ffffffff98386955>] do_page_fault+0x35/0x90 [Tue Jul 23 07:51:36 2019] [<ffffffff98382ab6>] ? error_swapgs+0xa7/0xbd [Tue Jul 23 07:51:36 2019] [<ffffffff98382768>] page_fault+0x28/0x30 [Tue Jul 23 07:51:36 2019] Mem-Info: [Tue Jul 23 07:51:36 2019] active_anon:3374527 inactive_anon:514225 isolated_anon:0 active_file:0 inactive_file:342 isolated_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:8192 slab_unreclaimable:21375 mapped:671 shmem:695 pagetables:13109 bounce:0 free:35067 free_pcp:839 free_cma:0 [Tue Jul 23 07:51:36 2019] Node 0 DMA free:15848kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15932kB managed:15848kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Tue Jul 23 07:51:36 2019] lowmem_reserve[]: 0 1778 7777 7777 [Tue Jul 23 07:51:36 2019] Node 0 DMA32 free:37020kB min:10112kB low:12640kB high:15168kB active_anon:1300512kB inactive_anon:433704kB active_file:0kB inactive_file:568kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052108kB managed:1821676kB mlocked:0kB dirty:0kB writeback:0kB mapped:148kB shmem:144kB slab_reclaimable:2804kB slab_unreclaimable:6984kB kernel_stack:352kB pagetables:3668kB unstable:0kB bounce:0kB free_pcp:2048kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:358 all_unreclaimable? no [Tue Jul 23 07:51:36 2019] lowmem_reserve[]: 0 0 5998 5998 [Tue Jul 23 07:51:36 2019] Node 0 Normal free:38384kB min:34080kB low:42600kB high:51120kB active_anon:5152848kB inactive_anon:736276kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6291456kB managed:6142164kB mlocked:0kB dirty:0kB writeback:0kB mapped:2356kB shmem:2268kB slab_reclaimable:10780kB slab_unreclaimable:32760kB kernel_stack:2800kB pagetables:22052kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Tue Jul 23 07:51:36 2019] lowmem_reserve[]: 0 0 0 0 [Tue Jul 23 07:51:36 2019] Node 1 Normal free:49016kB min:45828kB low:57284kB high:68740kB active_anon:7044748kB inactive_anon:886920kB active_file:0kB inactive_file:1232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8388608kB managed:8255016kB mlocked:0kB dirty:0kB writeback:0kB mapped:180kB shmem:368kB slab_reclaimable:19184kB slab_unreclaimable:45756kB kernel_stack:3088kB pagetables:26716kB unstable:0kB bounce:0kB free_pcp:1184kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1385 all_unreclaimable? no [Tue Jul 23 07:51:36 2019] lowmem_reserve[]: 0 0 0 0 [Tue Jul 23 07:51:36 2019] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15848kB [Tue Jul 23 07:51:36 2019] Node 0 DMA32: 1087*4kB (UEM) 454*8kB (UEM) 205*16kB (UEM) 163*32kB (UEM) 97*64kB (UEM) 56*128kB (UEM) 20*256kB (UEM) 5*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 37532kB [Tue Jul 23 07:51:36 2019] Node 0 Normal: 1877*4kB (UEM) 550*8kB (UEM) 337*16kB (UEM) 336*32kB (UEM) 153*64kB (UEM) 17*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 40020kB [Tue Jul 23 07:51:36 2019] Node 1 Normal: 1846*4kB (UEM) 1037*8kB (UEM) 452*16kB (UEM) 590*32kB (UEM) 155*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 51712kB [Tue Jul 23 07:51:36 2019] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Tue Jul 23 07:51:36 2019] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Tue Jul 23 07:51:36 2019] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [Tue Jul 23 07:51:36 2019] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Tue Jul 23 07:51:36 2019] 46200 total pagecache pages [Tue Jul 23 07:51:36 2019] 44918 pages in swap cache [Tue Jul 23 07:51:36 2019] Swap cache stats: add 2068339, delete 2022058, find 14089/15134 [Tue Jul 23 07:51:36 2019] Free swap = 0kB [Tue Jul 23 07:51:36 2019] Total swap = 8257532kB [Tue Jul 23 07:51:36 2019] 4187026 pages RAM [Tue Jul 23 07:51:36 2019] 0 pages HighMem/MovableOnly [Tue Jul 23 07:51:36 2019] 128350 pages reserved [Tue Jul 23 07:51:36 2019] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [Tue Jul 23 07:51:36 2019] [ 950] 0 950 9770 724 26 49 0 systemd-journal [Tue Jul 23 07:51:36 2019] [ 981] 0 981 50275 0 29 397 0 lvmetad [Tue Jul 23 07:51:36 2019] [ 990] 0 990 12156 1 28 742 -1000 systemd-udevd [Tue Jul 23 07:51:36 2019] [ 1469] 0 1469 13882 24 26 88 -1000 auditd [Tue Jul 23 07:51:36 2019] [ 1493] 999 1493 153252 92 61 1951 0 polkitd [Tue Jul 23 07:51:36 2019] [ 1496] 0 1496 6595 38 18 36 0 systemd-logind [Tue Jul 23 07:51:36 2019] [ 1497] 0 1497 5444 80 16 40 0 irqbalance [Tue Jul 23 07:51:36 2019] [ 1498] 81 1498 16616 92 32 97 -900 dbus-daemon [Tue Jul 23 07:51:36 2019] [ 1502] 998 1502 30101 29 32 93 0 chronyd [Tue Jul 23 07:51:36 2019] [ 1522] 0 1522 31573 29 19 131 0 crond [Tue Jul 23 07:51:36 2019] [ 1550] 0 1550 90654 234 98 6408 0 firewalld [Tue Jul 23 07:51:36 2019] [ 1551] 0 1551 27527 1 10 33 0 agetty [Tue Jul 23 07:51:36 2019] [ 1555] 0 1555 138115 266 87 353 0 NetworkManager [Tue Jul 23 07:51:36 2019] [ 1706] 0 1706 25724 1 52 515 0 dhclient [Tue Jul 23 07:51:36 2019] [ 1957] 0 1957 146648 648 103 3259 0 tuned [Tue Jul 23 07:51:36 2019] [ 1958] 0 1958 28230 1 57 257 -1000 sshd [Tue Jul 23 07:51:36 2019] [ 1965] 0 1965 54105 559 43 255 0 rsyslogd [Tue Jul 23 07:51:36 2019] [ 1993] 0 1993 26993 31 10 7 0 rhnsd [Tue Jul 23 07:51:36 2019] [ 1997] 0 1997 28928 23 13 16 0 rhsmcertd [Tue Jul 23 07:51:36 2019] [ 2556] 0 2556 22424 18 42 241 0 master [Tue Jul 23 07:51:36 2019] [ 2575] 89 2575 22467 14 47 241 0 qmgr [Tue Jul 23 07:51:36 2019] [10256] 0 10256 39195 0 80 341 0 sshd [Tue Jul 23 07:51:36 2019] [10260] 0 10260 28891 1 14 120 0 bash [Tue Jul 23 07:51:36 2019] [10339] 0 10339 6044154 3837063 11430 1974575 0 glusterfs [Tue Jul 23 07:51:36 2019] [10419] 0 10419 158970 1832 67 1016 0 glusterfs [Tue Jul 23 07:51:36 2019] [10453] 0 10453 31976 64 18 88 0 screen [Tue Jul 23 07:51:36 2019] [10454] 0 10454 33581 135 23 4672 0 bash [Tue Jul 23 07:51:36 2019] [10838] 0 10838 31976 1 18 152 0 screen [Tue Jul 23 07:51:36 2019] [10839] 0 10839 28894 1 13 120 0 bash [Tue Jul 23 07:51:36 2019] [13097] 0 13097 30067 1 14 92 0 find [Tue Jul 23 07:51:36 2019] [13098] 0 13098 27064 1 9 68 0 xargs [Tue Jul 23 07:51:36 2019] [15096] 0 15096 29782 1 15 79 0 stat [Tue Jul 23 07:51:36 2019] [ 4479] 0 4479 30784 0 17 45 0 ls [Tue Jul 23 07:51:36 2019] [ 5307] 0 5307 39195 0 79 337 0 sshd [Tue Jul 23 07:51:36 2019] [ 5311] 0 5311 28891 1 13 115 0 bash [Tue Jul 23 07:51:36 2019] [ 5336] 0 5336 30784 0 17 45 0 ls [Tue Jul 23 07:51:36 2019] [ 5645] 0 5645 39195 0 81 340 0 sshd [Tue Jul 23 07:51:36 2019] [ 5649] 0 5649 28891 1 13 124 0 bash [Tue Jul 23 07:51:36 2019] [ 5722] 0 5722 276387 1365 95 10313 0 glusterfs [Tue Jul 23 07:51:36 2019] [ 9294] 0 9294 93623 0 145 4766 -900 rhsmd [Tue Jul 23 07:51:36 2019] [19707] 89 19707 22450 253 45 0 0 pickup [Tue Jul 23 07:51:36 2019] [23996] 0 23996 28296 49 12 0 0 egrep [Tue Jul 23 07:51:36 2019] [23997] 0 23997 26989 26 10 0 0 tee [Tue Jul 23 07:51:36 2019] Out of memory: Kill process 10339 (glusterfs) score 922 or sacrifice child [Tue Jul 23 07:51:36 2019] Killed process 10339 (glusterfs), UID 0, total-vm:24176616kB, anon-rss:15348252kB, file-rss:0kB, shmem-rss:0kB [Tue Jul 23 09:27:55 2019] perf: interrupt took too long (6208 > 6192), lowering kernel.perf_event_max_sample_rate to 32000
Below we can see memory footprint is increasing since the hang, and till it got oomkilled Mon Jul 22 12:37:35 IST 2019 10339 root 20 0 2746344 1.5g 4544 S 18.8 9.5 442:51.83 glusterfs Mon Jul 22 12:39:35 IST 2019 10339 root 20 0 2746344 1.5g 4544 S 0.0 9.7 442:54.14 glusterfs Mon Jul 22 12:41:35 IST 2019 10339 root 20 0 2746344 1.5g 4544 S 0.0 9.8 442:56.48 glusterfs Mon Jul 22 12:43:35 IST 2019 10339 root 20 0 2811880 1.5g 4544 S 0.0 9.9 442:58.81 glusterfs Mon Jul 22 12:45:35 IST 2019 10339 root 20 0 2811880 1.6g 4544 S 0.0 10.1 443:01.14 glusterfs Mon Jul 22 12:47:36 IST 2019 10339 root 20 0 2811880 1.6g 4544 S 0.0 10.2 443:03.52 glusterfs Mon Jul 22 12:49:36 IST 2019 10339 root 20 0 2877416 1.6g 4544 S 0.0 10.3 443:05.92 glusterfs Mon Jul 22 12:51:36 IST 2019 10339 root 20 0 2877416 1.6g 4544 S 0.0 10.5 443:08.17 glusterfs Mon Jul 22 12:53:36 IST 2019 10339 root 20 0 2877416 1.6g 4544 S 0.0 10.6 443:10.49 glusterfs Mon Jul 22 12:55:36 IST 2019 10339 root 20 0 2942952 1.7g 4544 S 0.0 10.7 443:12.77 glusterfs Mon Jul 22 12:57:37 IST 2019 10339 root 20 0 2942952 1.7g 4544 S 0.0 10.9 443:15.12 glusterfs Mon Jul 22 12:59:37 IST 2019 10339 root 20 0 2942952 1.7g 4548 S 0.0 11.0 443:17.50 glusterfs Mon Jul 22 13:01:37 IST 2019 10339 root 20 0 3008488 1.7g 4548 S 0.0 11.1 443:19.88 glusterfs Mon Jul 22 13:03:37 IST 2019 10339 root 20 0 3008488 1.7g 4548 S 0.0 11.2 443:22.18 glusterfs Mon Jul 22 13:05:37 IST 2019 10339 root 20 0 3008488 1.8g 4548 S 6.2 11.4 443:24.51 glusterfs Mon Jul 22 13:07:38 IST 2019 10339 root 20 0 3074024 1.8g 4548 S 0.0 11.5 443:26.77 glusterfs Mon Jul 22 13:09:38 IST 2019 10339 root 20 0 3074024 1.8g 4548 S 0.0 11.6 443:29.01 glusterfs Mon Jul 22 13:11:38 IST 2019 10339 root 20 0 3074024 1.8g 4548 S 0.0 11.8 443:31.34 glusterfs Mon Jul 22 13:13:38 IST 2019 10339 root 20 0 3074024 1.8g 4548 S 0.0 11.9 443:33.67 glusterfs Mon Jul 22 13:15:39 IST 2019 10339 root 20 0 3139560 1.9g 4548 S 0.0 12.0 443:36.01 glusterfs Mon Jul 22 13:17:39 IST 2019 10339 root 20 0 3139560 1.9g 4548 S 0.0 12.2 443:38.35 glusterfs Mon Jul 22 13:19:39 IST 2019 10339 root 20 0 3139560 1.9g 4548 S 0.0 12.3 443:40.70 glusterfs Mon Jul 22 13:21:39 IST 2019 10339 root 20 0 3205096 1.9g 4548 S 0.0 12.4 443:42.99 glusterfs Mon Jul 22 13:23:39 IST 2019 10339 root 20 0 3205096 1.9g 4548 S 0.0 12.6 443:45.22 glusterfs Mon Jul 22 13:25:40 IST 2019 10339 root 20 0 3205096 2.0g 4548 S 0.0 12.7 443:47.45 glusterfs Mon Jul 22 13:27:40 IST 2019 10339 root 20 0 3270632 2.0g 4548 S 6.2 12.8 443:49.69 glusterfs Mon Jul 22 13:29:40 IST 2019 10339 root 20 0 3270632 2.0g 4548 S 0.0 12.9 443:51.97 glusterfs Mon Jul 22 13:31:40 IST 2019 10339 root 20 0 3270632 2.0g 4548 S 0.0 13.1 443:54.28 glusterfs Mon Jul 22 13:33:40 IST 2019 10339 root 20 0 3336168 2.0g 4548 S 0.0 13.2 443:56.51 glusterfs Mon Jul 22 13:35:41 IST 2019 10339 root 20 0 3336168 2.1g 4548 S 0.0 13.3 443:58.77 glusterfs Mon Jul 22 13:37:41 IST 2019 10339 root 20 0 3336168 2.1g 4548 S 0.0 13.4 444:01.03 glusterfs Mon Jul 22 13:39:41 IST 2019 10339 root 20 0 3401704 2.1g 4548 S 0.0 13.6 444:03.32 glusterfs Mon Jul 22 13:41:41 IST 2019 10339 root 20 0 3401704 2.1g 4548 S 0.0 13.7 444:05.64 glusterfs Mon Jul 22 13:43:41 IST 2019 10339 root 20 0 3401704 2.1g 4548 S 0.0 13.8 444:07.96 glusterfs Mon Jul 22 13:45:42 IST 2019 10339 root 20 0 3467240 2.2g 4548 S 0.0 14.0 444:10.26 glusterfs Mon Jul 22 13:47:42 IST 2019 10339 root 20 0 3467240 2.2g 4548 S 0.0 14.1 444:12.54 glusterfs Mon Jul 22 13:49:42 IST 2019 10339 root 20 0 3467240 2.2g 4548 S 0.0 14.2 444:14.85 glusterfs Mon Jul 22 13:51:42 IST 2019 10339 root 20 0 3532776 2.2g 4548 S 0.0 14.4 444:17.18 glusterfs Mon Jul 22 13:53:43 IST 2019 10339 root 20 0 3532776 2.2g 4548 S 0.0 14.5 444:19.56 glusterfs Mon Jul 22 13:55:43 IST 2019 10339 root 20 0 3532776 2.3g 4548 S 0.0 14.6 444:21.95 glusterfs Mon Jul 22 13:57:43 IST 2019 10339 root 20 0 3598312 2.3g 4548 S 0.0 14.8 444:24.39 glusterfs Mon Jul 22 13:59:43 IST 2019 10339 root 20 0 3598312 2.3g 4548 S 0.0 14.9 444:26.78 glusterfs Mon Jul 22 14:01:43 IST 2019 10339 root 20 0 3598312 2.3g 4548 S 18.8 15.0 444:29.05 glusterfs Mon Jul 22 14:03:44 IST 2019 10339 root 20 0 3663848 2.3g 4548 S 0.0 15.2 444:31.38 glusterfs Mon Jul 22 14:05:44 IST 2019 10339 root 20 0 3663848 2.4g 4548 S 6.2 15.3 444:33.71 glusterfs Mon Jul 22 14:07:44 IST 2019 10339 root 20 0 3663848 2.4g 4548 S 12.5 15.4 444:36.03 glusterfs Mon Jul 22 14:09:44 IST 2019 10339 root 20 0 3729384 2.4g 4548 S 0.0 15.6 444:38.34 glusterfs Mon Jul 22 14:11:44 IST 2019 10339 root 20 0 3729384 2.4g 4548 S 0.0 15.7 444:40.69 glusterfs Mon Jul 22 14:13:45 IST 2019 10339 root 20 0 3729384 2.5g 4548 S 12.5 15.9 444:43.02 glusterfs Mon Jul 22 14:15:45 IST 2019 10339 root 20 0 3794920 2.5g 4548 S 6.2 16.0 444:45.34 glusterfs Mon Jul 22 14:17:45 IST 2019 10339 root 20 0 3794920 2.5g 4548 S 6.2 16.1 444:47.65 glusterfs Mon Jul 22 14:19:45 IST 2019 10339 root 20 0 3794920 2.5g 4548 S 0.0 16.3 444:49.97 glusterfs Mon Jul 22 14:21:45 IST 2019 10339 root 20 0 3860456 2.5g 4548 S 0.0 16.4 444:52.25 glusterfs Mon Jul 22 14:23:46 IST 2019 10339 root 20 0 3860456 2.6g 4548 S 0.0 16.5 444:54.57 glusterfs Mon Jul 22 14:25:46 IST 2019 10339 root 20 0 3860456 2.6g 4548 S 0.0 16.7 444:56.94 glusterfs Mon Jul 22 14:27:46 IST 2019 10339 root 20 0 3925992 2.6g 4548 S 0.0 16.8 444:59.22 glusterfs Mon Jul 22 14:29:46 IST 2019 10339 root 20 0 3925992 2.6g 4548 S 0.0 16.9 445:01.53 glusterfs Mon Jul 22 14:31:47 IST 2019 10339 root 20 0 3925992 2.6g 4548 S 0.0 17.1 445:03.84 glusterfs Mon Jul 22 14:33:47 IST 2019 10339 root 20 0 3991528 2.7g 4548 S 0.0 17.2 445:06.15 glusterfs Mon Jul 22 14:35:47 IST 2019 10339 root 20 0 3991528 2.7g 4548 S 0.0 17.3 445:08.50 glusterfs Mon Jul 22 14:37:47 IST 2019 10339 root 20 0 3991528 2.7g 4548 S 6.2 17.5 445:10.85 glusterfs Mon Jul 22 14:39:48 IST 2019 10339 root 20 0 4057064 2.7g 4548 S 0.0 17.6 445:13.16 glusterfs Mon Jul 22 14:41:48 IST 2019 10339 root 20 0 4057064 2.7g 4548 S 6.2 17.7 445:15.49 glusterfs Mon Jul 22 14:43:48 IST 2019 10339 root 20 0 4057064 2.8g 4548 S 0.0 17.9 445:17.79 glusterfs Mon Jul 22 14:45:48 IST 2019 10339 root 20 0 4122600 2.8g 4548 S 0.0 18.0 445:20.07 glusterfs Mon Jul 22 14:47:48 IST 2019 10339 root 20 0 4122600 2.8g 4548 S 0.0 18.1 445:22.38 glusterfs Mon Jul 22 14:49:49 IST 2019 10339 root 20 0 4122600 2.8g 4548 S 0.0 18.2 445:24.70 glusterfs Mon Jul 22 14:51:49 IST 2019 10339 root 20 0 4122600 2.8g 4548 S 6.2 18.4 445:27.07 glusterfs Mon Jul 22 14:53:49 IST 2019 ## LOOP 2019 #### 10339 root 20 0 4188136 2.9g 4548 S 0.0 18.5 445:29.38 glusterfs Mon Jul 22 14:55:49 IST 2019 10339 root 20 0 4188136 2.9g 4548 S 0.0 18.6 445:31.74 glusterfs Mon Jul 22 14:57:50 IST 2019 10339 root 20 0 4188136 2.9g 4548 S 6.2 18.8 445:34.04 glusterfs Mon Jul 22 14:59:50 IST 2019 10339 root 20 0 4253672 2.9g 4548 S 0.0 18.9 445:36.33 glusterfs Mon Jul 22 15:01:50 IST 2019 10339 root 20 0 4253672 2.9g 4548 S 0.0 19.0 445:38.68 glusterfs Mon Jul 22 15:03:50 IST 2019 10339 root 20 0 4253672 3.0g 4548 S 0.0 19.1 445:41.26 glusterfs Mon Jul 22 15:05:50 IST 2019 KiB Mem : 16234704 total, 12019336 free, 3830828 used, 384540 buff/cache 10339 root 20 0 4319208 3.0g 4548 S 0.0 19.3 445:43.56 glusterfs Mon Jul 22 15:07:51 IST 2019 10339 root 20 0 4319208 3.0g 4548 S 0.0 19.4 445:45.85 glusterfs Mon Jul 22 15:09:51 IST 2019 10339 root 20 0 4319208 3.0g 4548 S 6.2 19.5 445:48.17 glusterfs Mon Jul 22 15:11:51 IST 2019 10339 root 20 0 4384744 3.0g 4548 S 0.0 19.7 445:50.57 glusterfs Mon Jul 22 15:13:51 IST 2019 10339 root 20 0 4384744 3.1g 4548 S 6.2 19.8 445:52.96 glusterfs Mon Jul 22 15:15:52 IST 2019 10339 root 20 0 4384744 3.1g 4548 S 0.0 19.9 445:55.30 glusterfs Mon Jul 22 15:17:52 IST 2019 10339 root 20 0 4450280 3.1g 4548 S 0.0 20.1 445:57.60 glusterfs Mon Jul 22 15:19:52 IST 2019 10339 root 20 0 4450280 3.1g 4548 S 0.0 20.2 445:59.87 glusterfs Mon Jul 22 15:21:52 IST 2019 10339 root 20 0 4450280 3.1g 4548 S 0.0 20.3 446:02.13 glusterfs Mon Jul 22 15:23:52 IST 2019 10339 root 20 0 4515816 3.2g 4548 S 0.0 20.4 446:04.57 glusterfs Mon Jul 22 15:25:53 IST 2019 10339 root 20 0 4515816 3.2g 4548 S 0.0 20.6 446:06.95 glusterfs Mon Jul 22 15:27:53 IST 2019 10339 root 20 0 4515816 3.2g 4548 S 6.2 20.7 446:09.31 glusterfs Mon Jul 22 15:29:53 IST 2019 10339 root 20 0 4581352 3.2g 4548 S 0.0 20.8 446:11.67 glusterfs Mon Jul 22 15:31:53 IST 2019 10339 root 20 0 4581352 3.2g 4548 S 0.0 21.0 446:13.98 glusterfs Mon Jul 22 15:33:54 IST 2019 10339 root 20 0 4581352 3.3g 4548 S 6.7 21.1 446:16.33 glusterfs Mon Jul 22 15:35:54 IST 2019 10339 root 20 0 4646888 3.3g 4548 S 0.0 21.2 446:18.71 glusterfs Mon Jul 22 15:37:54 IST 2019 10339 root 20 0 4646888 3.3g 4548 S 0.0 21.3 446:21.08 glusterfs Mon Jul 22 15:39:54 IST 2019 10339 root 20 0 4646888 3.3g 4548 S 0.0 21.5 446:23.40 glusterfs Mon Jul 22 15:41:54 IST 2019 10339 root 20 0 4646888 3.3g 4548 S 0.0 21.6 446:25.64 glusterfs Mon Jul 22 15:43:55 IST 2019 10339 root 20 0 4712424 3.4g 4548 S 0.0 21.7 446:27.93 glusterfs Mon Jul 22 15:45:55 IST 2019 10339 root 20 0 4712424 3.4g 4548 S 6.7 21.9 446:30.24 glusterfs Mon Jul 22 15:47:55 IST 2019 10339 root 20 0 4712424 3.4g 4548 S 0.0 22.0 446:32.52 glusterfs Mon Jul 22 15:49:55 IST 2019 10339 root 20 0 4777960 3.4g 4548 S 0.0 22.1 446:34.80 glusterfs Mon Jul 22 15:51:55 IST 2019 10339 root 20 0 4777960 3.4g 4548 S 0.0 22.3 446:37.10 glusterfs Mon Jul 22 15:53:56 IST 2019 10339 root 20 0 4777960 3.5g 4548 S 0.0 22.4 446:39.45 glusterfs Mon Jul 22 15:55:56 IST 2019 10339 root 20 0 4843496 3.5g 4548 S 0.0 22.5 446:41.79 glusterfs Mon Jul 22 15:57:56 IST 2019 10339 root 20 0 4843496 3.5g 4548 S 0.0 22.7 446:44.13 glusterfs Mon Jul 22 15:59:56 IST 2019 10339 root 20 0 4843496 3.5g 4548 S 0.0 22.8 446:46.49 glusterfs Mon Jul 22 16:01:57 IST 2019 10339 root 20 0 4909032 3.5g 4548 S 0.0 22.9 446:48.87 glusterfs Mon Jul 22 16:03:57 IST 2019 10339 root 20 0 4909032 3.6g 4548 S 6.2 23.0 446:51.19 glusterfs Mon Jul 22 16:05:57 IST 2019 10339 root 20 0 4909032 3.6g 4548 S 0.0 23.2 446:53.52 glusterfs Mon Jul 22 16:07:57 IST 2019 10339 root 20 0 4974568 3.6g 4548 S 0.0 23.4 446:55.87 glusterfs Mon Jul 22 16:09:57 IST 2019 10339 root 20 0 5040108 3.7g 4548 S 11.1 23.6 446:58.29 glusterfs Mon Jul 22 16:11:58 IST 2019 10339 root 20 0 5040104 3.7g 4548 S 0.0 23.9 447:00.85 glusterfs Mon Jul 22 16:13:58 IST 2019 10339 root 20 0 5105640 3.7g 4548 S 0.0 24.1 447:03.36 glusterfs Mon Jul 22 16:15:58 IST 2019 10339 root 20 0 5105640 3.8g 4548 S 0.0 24.4 447:05.88 glusterfs Mon Jul 22 16:17:58 IST 2019 10339 root 20 0 5171176 3.8g 4548 S 6.2 24.7 447:08.08 glusterfs Mon Jul 22 16:19:59 IST 2019 10339 root 20 0 5236712 3.9g 4548 S 12.5 25.0 447:10.70 glusterfs Mon Jul 22 16:21:59 IST 2019 10339 root 20 0 5236712 3.9g 4552 t 0.0 25.1 447:11.77 glusterfs Mon Jul 22 16:23:59 IST 2019 10339 root 20 0 5302248 3.9g 4552 S 0.0 25.4 447:14.08 glusterfs Mon Jul 22 16:25:59 IST 2019 10339 root 20 0 5302248 3.9g 4552 t 0.0 25.5 447:14.94 glusterfs Mon Jul 22 16:28:00 IST 2019 10339 root 20 0 5302248 3.9g 4552 t 0.0 25.5 447:14.94 glusterfs Mon Jul 22 16:30:00 IST 2019 10339 root 20 0 5367784 4.0g 4552 S 0.0 25.7 447:16.44 glusterfs Mon Jul 22 16:32:00 IST 2019 10339 root 20 0 5367784 4.0g 4552 S 0.0 25.9 447:18.94 glusterfs Mon Jul 22 16:34:00 IST 2019 10339 root 20 0 5433320 4.1g 4552 S 0.0 26.2 447:21.45 glusterfs Mon Jul 22 16:36:00 IST 2019 10339 root 20 0 5498856 4.1g 4552 S 0.0 26.5 447:23.99 glusterfs Mon Jul 22 16:38:02 IST 2019 10339 root 20 0 5498856 4.1g 4552 S 0.0 26.8 447:26.45 glusterfs Mon Jul 22 16:40:02 IST 2019 10339 root 20 0 5564392 4.2g 4552 S 0.0 27.0 447:28.92 glusterfs Mon Jul 22 16:42:02 IST 2019 10339 root 20 0 5629928 4.2g 4552 S 0.0 27.3 447:31.40 glusterfs Mon Jul 22 16:44:02 IST 2019 10339 root 20 0 5629928 4.3g 4600 S 0.0 27.5 447:42.32 glusterfs Mon Jul 22 16:46:03 IST 2019 10339 root 20 0 5695464 4.3g 4600 S 0.0 27.8 447:44.88 glusterfs Mon Jul 22 16:48:03 IST 2019 10339 root 20 0 5761000 4.3g 4600 S 12.5 28.1 447:47.42 glusterfs Mon Jul 22 16:50:03 IST 2019 10339 root 20 0 5761000 4.4g 4600 S 6.2 28.3 447:49.95 glusterfs Mon Jul 22 16:52:03 IST 2019 10339 root 20 0 5826536 4.4g 4600 S 18.8 28.6 447:52.51 glusterfs Mon Jul 22 16:54:04 IST 2019 10339 root 20 0 5892072 4.5g 4600 S 12.5 28.9 447:55.05 glusterfs Mon Jul 22 16:56:04 IST 2019 10339 root 20 0 5892072 4.5g 4600 S 0.0 29.2 447:57.56 glusterfs Mon Jul 22 16:58:04 IST 2019 10339 root 20 0 5957608 4.6g 4600 S 0.0 29.5 448:00.07 glusterfs Mon Jul 22 17:00:04 IST 2019 10339 root 20 0 6023144 4.6g 4600 S 0.0 29.7 448:02.60 glusterfs Mon Jul 22 17:02:04 IST 2019 10339 root 20 0 6023144 4.6g 4600 S 0.0 30.0 448:05.12 glusterfs Mon Jul 22 17:04:05 IST 2019 10339 root 20 0 6088680 4.7g 4600 S 0.0 30.3 448:07.59 glusterfs Mon Jul 22 17:06:05 IST 2019 10339 root 20 0 6154216 4.7g 4600 S 6.2 30.6 448:10.08 glusterfs Mon Jul 22 17:08:05 IST 2019 10339 root 20 0 6154216 4.8g 4600 S 0.0 30.8 448:12.57 glusterfs Mon Jul 22 17:10:05 IST 2019 10339 root 20 0 6219752 4.8g 4600 S 0.0 31.1 448:15.10 glusterfs Mon Jul 22 17:12:06 IST 2019 10339 root 20 0 6285288 4.9g 4600 S 6.2 31.4 448:17.71 glusterfs Mon Jul 22 17:14:06 IST 2019 10339 root 20 0 6350824 4.9g 4612 S 10.5 31.7 448:28.91 glusterfs Mon Jul 22 17:16:06 IST 2019 10339 root 20 0 6350824 5.0g 4612 S 6.2 32.0 448:31.49 glusterfs Mon Jul 22 17:18:06 IST 2019 10339 root 20 0 6416360 5.0g 4612 S 0.0 32.3 448:34.01 glusterfs Mon Jul 22 17:20:06 IST 2019 10339 root 20 0 6481896 5.0g 4612 S 6.2 32.6 448:36.55 glusterfs Mon Jul 22 17:22:07 IST 2019 10339 root 20 0 6481896 5.1g 4612 S 0.0 32.9 448:39.03 glusterfs Mon Jul 22 17:24:07 IST 2019 10339 root 20 0 6547432 5.1g 4612 S 0.0 33.1 448:41.50 glusterfs Mon Jul 22 17:26:07 IST 2019 10339 root 20 0 6612968 5.2g 4612 S 0.0 33.4 448:43.96 glusterfs Mon Jul 22 17:28:07 IST 2019 10339 root 20 0 6612968 5.2g 4612 S 0.0 33.7 448:46.39 glusterfs Mon Jul 22 17:30:08 IST 2019 10339 root 20 0 6678504 5.3g 4612 S 0.0 33.9 448:48.84 glusterfs Mon Jul 22 17:32:08 IST 2019 10339 root 20 0 6744040 5.3g 4612 S 18.8 34.2 448:51.36 glusterfs Mon Jul 22 17:34:08 IST 2019 10339 root 20 0 6744040 5.3g 4612 S 6.2 34.5 448:53.83 glusterfs Mon Jul 22 17:36:08 IST 2019 10339 root 20 0 6809576 5.4g 4612 S 0.0 34.7 448:56.32 glusterfs Mon Jul 22 17:38:09 IST 2019 10339 root 20 0 6875112 5.4g 4612 S 6.2 35.0 448:58.80 glusterfs Mon Jul 22 17:40:09 IST 2019 10339 root 20 0 6875112 5.5g 4612 S 6.2 35.3 449:01.29 glusterfs Mon Jul 22 17:42:09 IST 2019 10339 root 20 0 6940648 5.5g 4612 S 0.0 35.6 449:03.77 glusterfs Mon Jul 22 17:44:09 IST 2019 10339 root 20 0 7006184 5.5g 4612 S 0.0 35.8 449:06.24 glusterfs Mon Jul 22 17:46:10 IST 2019 10339 root 20 0 7006184 5.6g 4612 S 0.0 36.1 449:08.74 glusterfs Mon Jul 22 17:48:10 IST 2019 10339 root 20 0 7071720 5.6g 4612 S 6.2 36.4 449:11.27 glusterfs Mon Jul 22 17:50:10 IST 2019 10339 root 20 0 7137256 5.7g 4612 S 0.0 36.7 449:13.76 glusterfs Mon Jul 22 17:52:10 IST 2019 10339 root 20 0 7137256 5.7g 4612 S 0.0 36.9 449:16.27 glusterfs Mon Jul 22 17:54:10 IST 2019 10339 root 20 0 7202792 5.8g 4612 S 6.7 37.2 449:18.79 glusterfs Mon Jul 22 17:56:11 IST 2019 10339 root 20 0 7268328 5.8g 4612 S 6.2 37.5 449:21.29 glusterfs Mon Jul 22 17:58:11 IST 2019 10339 root 20 0 7333864 5.8g 4612 S 0.0 37.8 449:23.84 glusterfs Mon Jul 22 18:00:11 IST 2019 10339 root 20 0 7333864 5.9g 4612 S 0.0 38.0 449:26.35 glusterfs Mon Jul 22 18:02:11 IST 2019 10339 root 20 0 7399400 5.9g 4612 S 0.0 38.3 449:28.83 glusterfs Mon Jul 22 18:04:12 IST 2019 10339 root 20 0 7464936 6.0g 4612 S 0.0 38.6 449:31.31 glusterfs Mon Jul 22 18:06:12 IST 2019 10339 root 20 0 7464936 6.0g 4612 t 0.0 38.6 449:31.33 glusterfs Mon Jul 22 18:08:12 IST 2019 10339 root 20 0 7464936 6.0g 4612 t 0.0 38.6 449:31.33 glusterfs Mon Jul 22 18:10:12 IST 2019 10339 root 20 0 7464936 6.0g 4612 t 0.0 38.6 449:31.33 glusterfs Mon Jul 22 18:12:12 IST 2019 10339 root 20 0 7464936 6.0g 4612 t 0.0 38.6 449:31.33 glusterfs Mon Jul 22 18:14:13 IST 2019 10339 root 20 0 7464936 6.0g 4612 t 0.0 38.6 449:31.33 glusterfs Mon Jul 22 18:16:13 IST 2019 10339 root 20 0 7464936 6.0g 4612 S 6.2 38.7 449:32.41 glusterfs Mon Jul 22 18:18:13 IST 2019 10339 root 20 0 7464936 6.0g 4612 S 18.8 39.0 449:34.99 glusterfs Mon Jul 22 18:20:13 IST 2019 10339 root 20 0 7530472 6.1g 4612 S 0.0 39.2 449:37.50 glusterfs Mon Jul 22 18:22:14 IST 2019 10339 root 20 0 7596008 6.1g 4612 S 0.0 39.5 449:40.03 glusterfs Mon Jul 22 18:24:14 IST 2019 10339 root 20 0 7661544 6.2g 4612 S 0.0 39.8 449:42.55 glusterfs Mon Jul 22 18:26:14 IST 2019 10339 root 20 0 7661544 6.2g 4612 S 0.0 40.0 449:45.08 glusterfs Mon Jul 22 18:28:14 IST 2019 10339 root 20 0 7727080 6.2g 4612 S 0.0 40.3 449:47.62 glusterfs Mon Jul 22 18:30:15 IST 2019 10339 root 20 0 7792616 6.3g 4612 S 0.0 40.6 449:50.16 glusterfs Mon Jul 22 18:32:15 IST 2019 10339 root 20 0 7792616 6.3g 4612 S 6.2 40.9 449:52.70 glusterfs Mon Jul 22 18:34:15 IST 2019 10339 root 20 0 7858152 6.4g 4612 S 0.0 41.2 449:55.25 glusterfs Mon Jul 22 18:36:15 IST 2019 10339 root 20 0 7923688 6.4g 4612 S 0.0 41.5 449:57.80 glusterfs Mon Jul 22 18:38:15 IST 2019 10339 root 20 0 7923688 6.5g 4612 S 0.0 41.7 450:00.31 glusterfs Mon Jul 22 18:40:16 IST 2019 10339 root 20 0 7989224 6.5g 4612 S 0.0 42.0 450:02.92 glusterfs Mon Jul 22 18:42:16 IST 2019 10339 root 20 0 8054760 6.5g 4612 S 0.0 42.3 450:05.47 glusterfs Mon Jul 22 18:44:16 IST 2019 10339 root 20 0 8054760 6.6g 4612 S 0.0 42.6 450:07.97 glusterfs Mon Jul 22 18:46:16 IST 2019 10339 root 20 0 8120296 6.6g 4612 S 0.0 42.8 450:10.48 glusterfs Mon Jul 22 18:48:17 IST 2019 10339 root 20 0 8185832 6.7g 4612 S 0.0 43.1 450:13.06 glusterfs Mon Jul 22 18:50:17 IST 2019 10339 root 20 0 8185832 6.7g 4612 S 0.0 43.4 450:15.62 glusterfs Mon Jul 22 18:52:17 IST 2019 10339 root 20 0 8251368 6.8g 4612 S 0.0 43.7 450:18.22 glusterfs Mon Jul 22 18:54:17 IST 2019 10339 root 20 0 8316904 6.8g 4612 S 0.0 44.0 450:20.79 glusterfs Mon Jul 22 18:56:17 IST 2019 10339 root 20 0 8382440 6.9g 4612 S 0.0 44.3 450:23.31 glusterfs Mon Jul 22 18:58:18 IST 2019 10339 root 20 0 8382440 6.9g 4612 S 0.0 44.5 450:25.84 glusterfs Mon Jul 22 19:00:18 IST 2019 10339 root 20 0 8447976 6.9g 4612 S 0.0 44.8 450:28.40 glusterfs Mon Jul 22 19:02:18 IST 2019 10339 root 20 0 8513512 7.0g 4612 S 11.8 45.1 450:30.98 glusterfs Mon Jul 22 19:04:18 IST 2019 10339 root 20 0 8513512 7.0g 4612 S 0.0 45.4 450:33.56 glusterfs Mon Jul 22 19:06:18 IST 2019 10339 root 20 0 8579048 7.1g 4612 S 0.0 45.7 450:36.18 glusterfs Mon Jul 22 19:08:19 IST 2019 10339 root 20 0 8644584 7.1g 4612 S 0.0 45.9 450:38.79 glusterfs Mon Jul 22 19:10:19 IST 2019 10339 root 20 0 8644584 7.2g 4612 S 6.2 46.2 450:41.39 glusterfs Mon Jul 22 19:12:19 IST 2019 10339 root 20 0 8710120 7.2g 4612 S 0.0 46.5 450:43.52 glusterfs Mon Jul 22 19:14:19 IST 2019 10339 root 20 0 8775660 7.2g 4612 S 6.2 46.7 450:46.07 glusterfs Mon Jul 22 19:16:19 IST 2019 10339 root 20 0 8775656 7.3g 4612 S 6.2 47.0 450:48.64 glusterfs Mon Jul 22 19:18:20 IST 2019 10339 root 20 0 8841192 7.3g 4612 S 0.0 47.3 450:51.21 glusterfs Mon Jul 22 19:20:20 IST 2019 10339 root 20 0 8906728 7.4g 4612 S 0.0 47.5 450:53.78 glusterfs Mon Jul 22 19:22:20 IST 2019 10339 root 20 0 8906728 7.4g 4612 S 0.0 47.8 450:56.40 glusterfs Mon Jul 22 19:24:20 IST 2019 10339 root 20 0 8972264 7.4g 4612 S 0.0 48.1 450:58.98 glusterfs Mon Jul 22 19:26:21 IST 2019 10339 root 20 0 9037800 7.5g 4612 S 0.0 48.3 451:01.56 glusterfs Mon Jul 22 19:28:21 IST 2019 10339 root 20 0 9037800 7.5g 4612 S 6.2 48.6 451:04.05 glusterfs Mon Jul 22 19:30:21 IST 2019 10339 root 20 0 9103336 7.6g 4612 S 0.0 48.9 451:06.52 glusterfs Mon Jul 22 19:32:21 IST 2019 10339 root 20 0 9168872 7.6g 4612 S 0.0 49.1 451:08.98 glusterfs Mon Jul 22 19:34:21 IST 2019 10339 root 20 0 9168872 7.6g 4612 S 0.0 49.4 451:11.49 glusterfs Mon Jul 22 19:36:22 IST 2019 10339 root 20 0 9234408 7.7g 4612 S 0.0 49.7 451:14.03 glusterfs Mon Jul 22 19:38:22 IST 2019 10339 root 20 0 9299944 7.7g 4612 S 0.0 50.0 451:16.57 glusterfs Mon Jul 22 19:40:22 IST 2019 10339 root 20 0 9299944 7.8g 4612 S 6.2 50.2 451:19.13 glusterfs Mon Jul 22 19:42:22 IST 2019 10339 root 20 0 9365480 7.8g 4612 S 12.5 50.5 451:21.69 glusterfs Mon Jul 22 19:44:23 IST 2019 10339 root 20 0 9431016 7.9g 4612 S 0.0 50.8 451:24.14 glusterfs Mon Jul 22 19:46:23 IST 2019 10339 root 20 0 9431016 7.9g 4612 S 0.0 51.0 451:26.63 glusterfs Mon Jul 22 19:48:23 IST 2019 10339 root 20 0 9496552 7.9g 4612 S 0.0 51.3 451:29.12 glusterfs Mon Jul 22 19:50:23 IST 2019 10339 root 20 0 9562088 8.0g 4612 S 6.2 51.6 451:31.67 glusterfs Mon Jul 22 19:52:23 IST 2019 10339 root 20 0 9562088 8.0g 4612 S 0.0 51.9 451:34.20 glusterfs Mon Jul 22 19:54:24 IST 2019 10339 root 20 0 9627624 8.1g 4612 S 0.0 52.1 451:36.71 glusterfs Mon Jul 22 19:56:24 IST 2019 10339 root 20 0 9693160 8.1g 4612 S 0.0 52.4 451:39.26 glusterfs Mon Jul 22 19:58:24 IST 2019 10339 root 20 0 9693160 8.2g 4612 S 0.0 52.7 451:41.81 glusterfs Mon Jul 22 20:00:24 IST 2019 10339 root 20 0 9758696 8.2g 4612 S 0.0 52.9 451:44.32 glusterfs Mon Jul 22 20:02:25 IST 2019 10339 root 20 0 9824232 8.2g 4612 S 0.0 53.2 451:46.84 glusterfs Mon Jul 22 20:04:25 IST 2019 10339 root 20 0 9824232 8.3g 4612 S 12.5 53.5 451:49.36 glusterfs Mon Jul 22 20:06:25 IST 2019 10339 root 20 0 9889768 8.3g 4612 S 0.0 53.7 451:51.85 glusterfs Mon Jul 22 20:08:25 IST 2019 10339 root 20 0 9955304 8.4g 4612 S 0.0 54.0 451:54.39 glusterfs Mon Jul 22 20:10:26 IST 2019 10339 root 20 0 9955304 8.4g 4612 S 6.2 54.3 451:56.93 glusterfs Mon Jul 22 20:12:26 IST 2019 10339 root 20 0 9786.0m 8.4g 4612 S 0.0 54.5 451:59.46 glusterfs Mon Jul 22 20:14:26 IST 2019 10339 root 20 0 9850.0m 8.5g 4612 S 6.2 54.8 452:02.01 glusterfs Mon Jul 22 20:16:26 IST 2019 10339 root 20 0 9850.0m 8.5g 4612 S 12.5 55.1 452:04.55 glusterfs Mon Jul 22 20:18:27 IST 2019 10339 root 20 0 9914.0m 8.6g 4612 S 0.0 55.3 452:07.09 glusterfs Mon Jul 22 20:20:27 IST 2019 10339 root 20 0 9978.0m 8.6g 4612 S 6.2 55.6 452:09.61 glusterfs Mon Jul 22 20:22:27 IST 2019 10339 root 20 0 9978.0m 8.7g 4612 S 0.0 55.9 452:12.13 glusterfs Mon Jul 22 20:24:27 IST 2019 10339 root 20 0 9.8g 8.7g 4612 S 0.0 56.1 452:14.70 glusterfs Mon Jul 22 20:26:28 IST 2019 10339 root 20 0 9.9g 8.7g 4612 S 0.0 56.4 452:17.26 glusterfs Mon Jul 22 20:28:28 IST 2019 10339 root 20 0 9.9g 8.8g 4612 S 0.0 56.7 452:19.83 glusterfs Mon Jul 22 20:30:28 IST 2019 10339 root 20 0 9.9g 8.8g 4612 S 0.0 56.9 452:22.43 glusterfs Mon Jul 22 20:32:28 IST 2019 10339 root 20 0 10.0g 8.9g 4612 S 0.0 57.2 452:24.96 glusterfs Mon Jul 22 20:34:28 IST 2019 10339 root 20 0 10.0g 8.9g 4612 S 0.0 57.4 452:27.46 glusterfs Mon Jul 22 20:36:29 IST 2019 10339 root 20 0 10.1g 8.9g 4612 S 0.0 57.7 452:29.89 glusterfs Mon Jul 22 20:38:29 IST 2019 10339 root 20 0 10.1g 9.0g 4612 S 0.0 57.9 452:32.35 glusterfs Mon Jul 22 20:40:29 IST 2019 10339 root 20 0 10.1g 9.0g 4612 S 0.0 58.2 452:34.80 glusterfs Mon Jul 22 20:42:29 IST 2019 10339 root 20 0 10.2g 9.1g 4612 S 0.0 58.5 452:37.28 glusterfs Mon Jul 22 20:44:30 IST 2019 10339 root 20 0 10.2g 9.1g 4612 S 0.0 58.7 452:39.77 glusterfs Mon Jul 22 20:46:30 IST 2019 10339 root 20 0 10.2g 9.1g 4612 S 0.0 59.0 452:42.28 glusterfs Mon Jul 22 20:48:30 IST 2019 10339 root 20 0 10.3g 9.2g 4612 S 0.0 59.2 452:44.78 glusterfs Mon Jul 22 20:50:30 IST 2019 10339 root 20 0 10.3g 9.2g 4612 S 0.0 59.5 452:47.27 glusterfs Mon Jul 22 20:52:31 IST 2019 10339 root 20 0 10.4g 9.3g 4612 S 0.0 59.7 452:49.80 glusterfs Mon Jul 22 20:54:31 IST 2019 10339 root 20 0 10.4g 9.3g 4612 S 6.2 60.0 452:52.42 glusterfs Mon Jul 22 20:56:31 IST 2019 10339 root 20 0 10.4g 9.3g 4612 S 0.0 60.3 452:55.17 glusterfs Mon Jul 22 20:58:31 IST 2019 10339 root 20 0 10.5g 9.4g 4612 S 0.0 60.6 452:57.87 glusterfs Mon Jul 22 21:00:31 IST 2019 10339 root 20 0 10.6g 9.4g 4612 S 0.0 60.9 453:00.57 glusterfs Mon Jul 22 21:02:32 IST 2019 10339 root 20 0 10.6g 9.5g 4612 S 0.0 61.1 453:03.30 glusterfs Mon Jul 22 21:04:32 IST 2019 10339 root 20 0 10.6g 9.5g 4612 S 6.2 61.4 453:06.02 glusterfs Mon Jul 22 21:06:32 IST 2019 10339 root 20 0 10.7g 9.5g 4612 S 0.0 61.7 453:08.68 glusterfs Mon Jul 22 21:08:32 IST 2019 10339 root 20 0 10.7g 9.6g 4612 S 0.0 61.9 453:11.31 glusterfs Mon Jul 22 21:10:32 IST 2019 10339 root 20 0 10.7g 9.6g 4612 S 0.0 62.2 453:13.97 glusterfs Mon Jul 22 21:12:33 IST 2019 10339 root 20 0 10.8g 9.7g 4612 S 0.0 62.5 453:16.71 glusterfs Mon Jul 22 21:14:33 IST 2019 10339 root 20 0 10.8g 9.7g 4612 S 0.0 62.8 453:19.54 glusterfs Mon Jul 22 21:16:33 IST 2019 10339 root 20 0 10.9g 9.8g 4612 S 0.0 63.0 453:22.25 glusterfs Mon Jul 22 21:18:33 IST 2019 10339 root 20 0 10.9g 9.8g 4612 S 0.0 63.3 453:24.90 glusterfs Mon Jul 22 21:20:34 IST 2019 10339 root 20 0 10.9g 9.8g 4612 S 0.0 63.6 453:27.71 glusterfs Mon Jul 22 21:22:34 IST 2019 10339 root 20 0 11.0g 9.9g 4612 S 13.3 63.8 453:30.37 glusterfs Mon Jul 22 21:24:34 IST 2019 10339 root 20 0 11.1g 9.9g 4612 S 0.0 64.1 453:32.97 glusterfs Mon Jul 22 21:26:34 IST 2019 10339 root 20 0 11.1g 10.0g 4612 S 12.5 64.3 453:35.42 glusterfs Mon Jul 22 21:28:34 IST 2019 10339 root 20 0 11.1g 10.0g 4612 S 12.5 64.6 453:37.88 glusterfs Mon Jul 22 21:30:35 IST 2019 10339 root 20 0 11.2g 10.0g 4612 S 0.0 64.8 453:40.30 glusterfs Mon Jul 22 21:32:35 IST 2019 10339 root 20 0 11.2g 10.1g 4612 S 6.2 65.0 453:42.73 glusterfs Mon Jul 22 21:34:35 IST 2019 10339 root 20 0 11.2g 10.1g 4612 S 0.0 65.3 453:45.17 glusterfs Mon Jul 22 21:36:35 IST 2019 10339 root 20 0 11.2g 10.1g 4612 S 0.0 65.5 453:47.64 glusterfs Mon Jul 22 21:38:36 IST 2019 10339 root 20 0 11.3g 10.2g 4612 S 0.0 65.8 453:50.12 glusterfs Mon Jul 22 21:40:36 IST 2019 10339 root 20 0 11.4g 10.2g 4612 S 0.0 66.0 453:52.75 glusterfs Mon Jul 22 21:42:36 IST 2019 10339 root 20 0 11.4g 10.3g 4612 S 0.0 66.3 453:55.51 glusterfs Mon Jul 22 21:44:36 IST 2019 10339 root 20 0 11.4g 10.3g 4612 S 6.2 66.6 453:58.31 glusterfs Mon Jul 22 21:46:36 IST 2019 10339 root 20 0 11.5g 10.4g 4612 S 0.0 66.9 454:00.94 glusterfs Mon Jul 22 21:48:37 IST 2019 10339 root 20 0 11.5g 10.4g 4612 S 0.0 67.1 454:03.53 glusterfs Mon Jul 22 21:50:37 IST 2019 10339 root 20 0 11.6g 10.4g 4612 S 0.0 67.3 454:06.22 glusterfs Mon Jul 22 21:52:37 IST 2019 10339 root 20 0 11.6g 10.5g 4612 S 0.0 67.6 454:08.95 glusterfs Mon Jul 22 21:54:37 IST 2019 10339 root 20 0 11.6g 10.5g 4612 S 6.2 67.9 454:11.73 glusterfs Mon Jul 22 21:56:37 IST 2019 10339 root 20 0 11.7g 10.6g 4612 S 12.5 68.1 454:14.39 glusterfs Mon Jul 22 21:58:38 IST 2019 10339 root 20 0 11.7g 10.6g 4612 S 0.0 68.4 454:16.95 glusterfs Mon Jul 22 22:00:38 IST 2019 10339 root 20 0 11.7g 10.6g 4612 S 0.0 68.7 454:19.71 glusterfs Mon Jul 22 22:02:38 IST 2019 10339 root 20 0 11.8g 10.7g 4612 S 0.0 68.9 454:22.28 glusterfs Mon Jul 22 22:04:38 IST 2019 10339 root 20 0 11.8g 10.7g 4612 S 6.2 69.2 454:25.06 glusterfs Mon Jul 22 22:06:38 IST 2019 10339 root 20 0 11.9g 10.7g 4612 S 12.5 69.4 454:27.71 glusterfs Mon Jul 22 22:08:39 IST 2019 10339 root 20 0 11.9g 10.8g 4612 S 0.0 69.7 454:30.30 glusterfs Mon Jul 22 22:10:39 IST 2019 10339 root 20 0 11.9g 10.8g 4612 S 0.0 69.9 454:32.98 glusterfs Mon Jul 22 22:12:39 IST 2019 10339 root 20 0 12.0g 10.9g 4612 S 0.0 70.2 454:35.71 glusterfs Mon Jul 22 22:14:39 IST 2019 10339 root 20 0 12.1g 10.9g 4612 S 0.0 70.5 454:38.52 glusterfs Mon Jul 22 22:16:40 IST 2019 10339 root 20 0 12.1g 11.0g 4612 S 0.0 70.8 454:41.17 glusterfs Mon Jul 22 22:18:40 IST 2019 10339 root 20 0 12.1g 11.0g 4612 S 6.2 71.0 454:43.95 glusterfs Mon Jul 22 22:20:40 IST 2019 10339 root 20 0 12.2g 11.0g 4612 S 0.0 71.3 454:46.78 glusterfs Mon Jul 22 22:22:40 IST 2019 10339 root 20 0 12.2g 11.1g 4612 S 0.0 71.5 454:49.30 glusterfs Mon Jul 22 22:24:41 IST 2019 10339 root 20 0 12.2g 11.1g 4612 S 0.0 71.8 454:51.85 glusterfs Mon Jul 22 22:26:41 IST 2019 10339 root 20 0 12.2g 11.2g 4612 S 0.0 72.0 454:54.63 glusterfs Mon Jul 22 22:28:41 IST 2019 10339 root 20 0 12.3g 11.2g 4612 S 0.0 72.3 454:57.36 glusterfs Mon Jul 22 22:30:41 IST 2019 10339 root 20 0 12.4g 11.2g 4612 S 0.0 72.6 455:00.16 glusterfs Mon Jul 22 22:32:41 IST 2019 10339 root 20 0 12.4g 11.3g 4612 S 0.0 72.9 455:02.83 glusterfs Mon Jul 22 22:34:42 IST 2019 10339 root 20 0 12.4g 11.3g 4612 S 0.0 73.1 455:05.59 glusterfs Mon Jul 22 22:36:42 IST 2019 10339 root 20 0 12.5g 11.4g 4612 S 0.0 73.4 455:08.31 glusterfs Mon Jul 22 22:38:42 IST 2019 10339 root 20 0 12.5g 11.4g 4612 S 20.0 73.6 455:10.96 glusterfs Mon Jul 22 22:40:42 IST 2019 10339 root 20 0 12.6g 11.4g 4612 S 6.2 73.9 455:13.66 glusterfs Mon Jul 22 22:42:42 IST 2019 10339 root 20 0 12.6g 11.5g 4612 S 0.0 74.2 455:16.50 glusterfs Mon Jul 22 22:44:43 IST 2019 10339 root 20 0 12.6g 11.5g 4612 S 6.2 74.5 455:19.16 glusterfs Mon Jul 22 22:46:43 IST 2019 10339 root 20 0 12.7g 11.6g 4612 S 0.0 74.7 455:21.83 glusterfs Mon Jul 22 22:48:43 IST 2019 10339 root 20 0 12.7g 11.6g 4612 S 0.0 74.9 455:24.37 glusterfs Mon Jul 22 22:50:43 IST 2019 10339 root 20 0 12.7g 11.6g 4612 S 0.0 75.2 455:27.08 glusterfs Mon Jul 22 22:52:44 IST 2019 10339 root 20 0 12.8g 11.7g 4612 S 0.0 75.4 455:29.67 glusterfs Mon Jul 22 22:54:44 IST 2019 10339 root 20 0 12.9g 11.7g 4612 S 0.0 75.7 455:32.36 glusterfs Mon Jul 22 22:56:44 IST 2019 10339 root 20 0 12.9g 11.8g 4612 S 0.0 76.0 455:34.97 glusterfs Mon Jul 22 22:58:44 IST 2019 10339 root 20 0 12.9g 11.8g 4612 S 0.0 76.2 455:37.70 glusterfs Mon Jul 22 23:00:45 IST 2019 10339 root 20 0 12.9g 11.8g 4612 S 12.5 76.5 455:40.35 glusterfs Mon Jul 22 23:02:45 IST 2019 10339 root 20 0 13.0g 11.9g 4612 S 18.8 76.7 455:43.11 glusterfs Mon Jul 22 23:04:45 IST 2019 10339 root 20 0 13.1g 11.9g 4612 S 0.0 77.1 455:45.95 glusterfs Mon Jul 22 23:06:45 IST 2019 10339 root 20 0 13.1g 12.0g 4612 S 0.0 77.3 455:48.69 glusterfs Mon Jul 22 23:08:46 IST 2019 10339 root 20 0 13.1g 12.0g 4612 S 0.0 77.6 455:51.45 glusterfs Mon Jul 22 23:10:46 IST 2019 10339 root 20 0 13.2g 12.1g 4612 S 0.0 77.9 455:54.19 glusterfs Mon Jul 22 23:12:46 IST 2019 10339 root 20 0 13.2g 12.1g 4612 S 0.0 78.2 455:56.88 glusterfs Mon Jul 22 23:14:46 IST 2019 10339 root 20 0 13.2g 12.1g 4612 S 6.2 78.4 455:59.53 glusterfs Mon Jul 22 23:16:46 IST 2019 10339 root 20 0 13.3g 12.2g 4612 S 0.0 78.7 456:02.06 glusterfs Mon Jul 22 23:18:47 IST 2019 10339 root 20 0 13.4g 12.2g 4612 S 0.0 78.9 456:04.56 glusterfs Mon Jul 22 23:20:47 IST 2019 10339 root 20 0 13.4g 12.3g 4612 S 0.0 79.2 456:07.05 glusterfs Mon Jul 22 23:22:47 IST 2019 10339 root 20 0 13.4g 12.3g 4612 S 0.0 79.5 456:09.49 glusterfs Mon Jul 22 23:24:47 IST 2019 10339 root 20 0 13.4g 12.3g 4612 S 0.0 79.7 456:11.97 glusterfs Mon Jul 22 23:26:48 IST 2019 10339 root 20 0 13.5g 12.4g 4612 S 0.0 80.0 456:14.55 glusterfs Mon Jul 22 23:28:48 IST 2019 10339 root 20 0 13.6g 12.4g 4612 S 6.2 80.2 456:17.29 glusterfs Mon Jul 22 23:30:48 IST 2019 10339 root 20 0 13.6g 12.5g 4612 S 0.0 80.5 456:20.03 glusterfs Mon Jul 22 23:32:48 IST 2019 10339 root 20 0 13.6g 12.5g 4612 S 6.2 80.8 456:22.99 glusterfs Mon Jul 22 23:34:48 IST 2019 10339 root 20 0 13.7g 12.6g 4612 S 6.2 81.1 456:25.81 glusterfs Mon Jul 22 23:36:49 IST 2019 10339 root 20 0 13.7g 12.6g 4612 S 0.0 81.4 456:28.63 glusterfs Mon Jul 22 23:38:49 IST 2019 10339 root 20 0 13.7g 12.7g 4612 S 12.5 81.7 456:31.43 glusterfs Mon Jul 22 23:40:49 IST 2019 10339 root 20 0 13.8g 12.7g 4612 S 0.0 82.0 456:34.26 glusterfs Mon Jul 22 23:42:49 IST 2019 10339 root 20 0 13.9g 12.7g 4612 S 0.0 82.3 456:37.20 glusterfs Mon Jul 22 23:44:50 IST 2019 10339 root 20 0 13.9g 12.8g 4612 S 0.0 82.6 456:39.80 glusterfs Mon Jul 22 23:46:50 IST 2019 10339 root 20 0 13.9g 12.8g 4612 S 6.2 82.8 456:42.38 glusterfs Mon Jul 22 23:48:50 IST 2019 10339 root 20 0 14.0g 12.9g 4612 S 0.0 83.0 456:45.00 glusterfs Mon Jul 22 23:50:50 IST 2019 10339 root 20 0 14.0g 12.9g 4612 S 0.0 83.3 456:47.75 glusterfs Mon Jul 22 23:52:50 IST 2019 10339 root 20 0 14.1g 12.9g 4612 S 0.0 83.6 456:50.45 glusterfs Mon Jul 22 23:54:51 IST 2019 10339 root 20 0 14.1g 13.0g 4612 S 0.0 83.9 456:53.21 glusterfs Mon Jul 22 23:56:51 IST 2019 10339 root 20 0 14.1g 13.0g 4612 S 12.5 84.1 456:55.88 glusterfs Mon Jul 22 23:58:51 IST 2019 10339 root 20 0 14.2g 13.1g 4612 S 0.0 84.4 456:58.58 glusterfs Tue Jul 23 00:00:51 IST 2019 10339 root 20 0 14.2g 13.1g 4612 S 0.0 84.7 457:01.41 glusterfs Tue Jul 23 00:02:52 IST 2019 10339 root 20 0 14.2g 13.2g 4612 S 0.0 85.0 457:04.04 glusterfs Tue Jul 23 00:04:52 IST 2019 10339 root 20 0 14.3g 13.2g 4612 S 0.0 85.2 457:06.77 glusterfs Tue Jul 23 00:06:52 IST 2019 10339 root 20 0 14.4g 13.2g 4612 S 0.0 85.5 457:09.38 glusterfs Tue Jul 23 00:08:52 IST 2019 10339 root 20 0 14.4g 13.3g 4612 S 12.5 85.7 457:11.86 glusterfs Tue Jul 23 00:10:52 IST 2019 10339 root 20 0 14.4g 13.3g 4612 S 0.0 85.9 457:14.29 glusterfs Tue Jul 23 00:12:53 IST 2019 10339 root 20 0 14.4g 13.3g 4612 S 12.5 86.2 457:16.72 glusterfs Tue Jul 23 00:14:53 IST 2019 10339 root 20 0 14.5g 13.4g 4612 S 6.7 86.4 457:19.15 glusterfs Tue Jul 23 00:16:53 IST 2019 10339 root 20 0 14.6g 13.4g 4612 S 6.2 86.7 457:21.60 glusterfs Tue Jul 23 00:18:53 IST 2019 10339 root 20 0 14.6g 13.5g 4612 S 0.0 86.9 457:24.05 glusterfs Tue Jul 23 00:20:54 IST 2019 10339 root 20 0 14.6g 13.5g 4612 S 0.0 87.2 457:26.63 glusterfs Tue Jul 23 00:22:54 IST 2019 10339 root 20 0 14.7g 13.5g 4612 S 0.0 87.4 457:29.27 glusterfs Tue Jul 23 00:24:54 IST 2019 10339 root 20 0 14.7g 13.6g 4612 S 0.0 87.6 457:31.89 glusterfs Tue Jul 23 00:26:55 IST 2019 10339 root 20 0 14.7g 13.6g 4608 S 6.2 87.8 457:34.65 glusterfs Tue Jul 23 00:28:55 IST 2019 10339 root 20 0 14.7g 13.6g 4608 S 6.2 87.9 457:37.28 glusterfs Tue Jul 23 00:30:55 IST 2019 10339 root 20 0 14.7g 13.6g 4608 S 0.0 88.1 457:39.75 glusterfs Tue Jul 23 00:32:55 IST 2019 10339 root 20 0 14.8g 13.7g 4608 S 0.0 88.3 457:42.23 glusterfs Tue Jul 23 00:34:56 IST 2019 10339 root 20 0 14.8g 13.7g 4608 S 0.0 88.6 457:44.73 glusterfs Tue Jul 23 00:36:56 IST 2019 10339 root 20 0 14.9g 13.8g 4608 S 0.0 88.9 457:47.41 glusterfs Tue Jul 23 00:38:56 IST 2019 10339 root 20 0 14.9g 13.8g 4608 S 0.0 89.1 457:50.08 glusterfs Tue Jul 23 00:40:56 IST 2019 10339 root 20 0 14.9g 13.8g 4580 S 0.0 89.4 457:52.78 glusterfs Tue Jul 23 00:42:56 IST 2019 10339 root 20 0 15.0g 13.9g 4028 S 0.0 89.6 457:55.27 glusterfs Tue Jul 23 00:44:57 IST 2019 10339 root 20 0 15.1g 13.8g 3568 S 0.0 89.4 457:57.83 glusterfs Tue Jul 23 00:46:57 IST 2019 10339 root 20 0 15.1g 13.9g 3568 S 0.0 89.7 458:00.32 glusterfs Tue Jul 23 00:48:57 IST 2019 10339 root 20 0 15.1g 13.9g 3568 S 0.0 89.9 458:02.85 glusterfs Tue Jul 23 00:50:58 IST 2019 10339 root 20 0 15.1g 14.0g 3568 S 6.2 90.2 458:05.36 glusterfs Tue Jul 23 00:52:58 IST 2019 10339 root 20 0 15.2g 14.0g 3568 S 6.2 90.5 458:07.87 glusterfs Tue Jul 23 00:54:58 IST 2019 10339 root 20 0 15.2g 14.0g 3568 S 0.0 90.7 458:10.36 glusterfs Tue Jul 23 00:56:58 IST 2019 10339 root 20 0 15.2g 14.0g 3568 S 0.0 90.3 458:12.90 glusterfs Tue Jul 23 00:58:58 IST 2019 10339 root 20 0 15.3g 14.0g 3568 S 0.0 90.5 458:15.38 glusterfs Tue Jul 23 01:00:59 IST 2019 10339 root 20 0 15.4g 14.1g 3568 S 0.0 90.8 458:17.89 glusterfs Tue Jul 23 01:02:59 IST 2019 10339 root 20 0 15.4g 14.1g 3568 S 0.0 91.1 458:20.37 glusterfs Tue Jul 23 01:04:59 IST 2019 10339 root 20 0 15.4g 14.1g 2032 S 0.0 91.3 458:22.89 glusterfs Tue Jul 23 01:07:00 IST 2019 10339 root 20 0 15.5g 14.2g 2032 S 0.0 91.6 458:25.38 glusterfs Tue Jul 23 01:09:00 IST 2019 10339 root 20 0 15.5g 14.1g 2032 S 6.2 91.1 458:27.93 glusterfs Tue Jul 23 01:11:00 IST 2019 10339 root 20 0 15.6g 14.1g 2032 S 0.0 91.4 458:30.40 glusterfs Tue Jul 23 01:13:00 IST 2019 10339 root 20 0 15.6g 14.2g 1868 S 0.0 91.6 458:32.94 glusterfs Tue Jul 23 01:15:01 IST 2019 10339 root 20 0 15.6g 14.2g 1868 S 0.0 91.8 458:35.46 glusterfs Tue Jul 23 01:17:01 IST 2019 10339 root 20 0 15.7g 14.2g 1868 S 0.0 92.0 458:37.94 glusterfs Tue Jul 23 01:19:01 IST 2019 10339 root 20 0 15.7g 14.2g 1868 S 12.5 91.6 458:40.49 glusterfs Tue Jul 23 01:21:01 IST 2019 10339 root 20 0 15.7g 14.2g 1868 S 0.0 91.7 458:43.00 glusterfs Tue Jul 23 01:23:02 IST 2019 10339 root 20 0 15.8g 14.2g 1868 S 0.0 92.0 458:45.48 glusterfs Tue Jul 23 01:25:02 IST 2019 10339 root 20 0 15.8g 14.1g 1868 S 18.8 90.8 458:48.09 glusterfs Tue Jul 23 01:27:02 IST 2019 10339 root 20 0 15.9g 14.1g 1868 S 0.0 91.1 458:50.55 glusterfs Tue Jul 23 01:29:02 IST 2019 10339 root 20 0 15.9g 14.1g 1868 S 0.0 91.4 458:53.06 glusterfs Tue Jul 23 01:31:03 IST 2019 10339 root 20 0 15.9g 14.2g 1868 S 6.2 91.6 458:55.55 glusterfs Tue Jul 23 01:33:03 IST 2019 10339 root 20 0 16.0g 14.1g 1868 S 0.0 91.3 458:58.10 glusterfs Tue Jul 23 01:35:03 IST 2019 10339 root 20 0 16.0g 14.2g 1868 S 0.0 91.5 459:00.64 glusterfs Tue Jul 23 01:37:04 IST 2019 10339 root 20 0 16.1g 14.2g 1868 S 12.5 91.4 459:03.23 glusterfs Tue Jul 23 01:39:04 IST 2019 10339 root 20 0 16.1g 14.2g 1868 S 0.0 91.6 459:05.69 glusterfs Tue Jul 23 01:41:04 IST 2019 10339 root 20 0 16.1g 14.2g 1868 S 0.0 91.8 459:08.28 glusterfs Tue Jul 23 01:43:04 IST 2019 10339 root 20 0 16.2g 14.2g 1868 S 0.0 91.4 459:10.97 glusterfs Tue Jul 23 01:45:05 IST 2019 10339 root 20 0 16.2g 14.2g 1868 S 0.0 91.5 459:13.64 glusterfs Tue Jul 23 01:47:05 IST 2019 10339 root 20 0 16.2g 14.2g 1868 S 0.0 91.7 459:16.21 glusterfs Tue Jul 23 01:49:05 IST 2019 10339 root 20 0 16.2g 14.1g 1868 S 0.0 91.4 459:18.88 glusterfs Tue Jul 23 01:51:06 IST 2019 10339 root 20 0 16.3g 14.2g 1868 S 0.0 91.6 459:21.52 glusterfs Tue Jul 23 01:53:06 IST 2019 10339 root 20 0 16.4g 14.1g 1868 S 0.0 91.3 459:24.38 glusterfs Tue Jul 23 01:55:06 IST 2019 10339 root 20 0 16.4g 14.2g 1868 S 0.0 91.6 459:26.99 glusterfs Tue Jul 23 01:57:06 IST 2019 10339 root 20 0 16.4g 14.2g 1868 S 0.0 91.8 459:29.70 glusterfs Tue Jul 23 01:59:07 IST 2019 10339 root 20 0 16.4g 14.2g 1868 S 0.0 91.5 459:32.31 glusterfs Tue Jul 23 02:01:07 IST 2019 10339 root 20 0 16.5g 14.2g 1868 S 6.2 91.7 459:34.77 glusterfs Tue Jul 23 02:03:07 IST 2019 10339 root 20 0 16.6g 14.2g 1868 S 0.0 91.9 459:37.45 glusterfs Tue Jul 23 02:05:07 IST 2019 10339 root 20 0 16.6g 14.3g 1868 S 0.0 92.2 459:40.26 glusterfs Tue Jul 23 02:07:08 IST 2019 10339 root 20 0 16.6g 14.1g 1868 S 6.2 91.4 459:42.97 glusterfs Tue Jul 23 02:09:08 IST 2019 10339 root 20 0 16.6g 14.2g 1868 S 6.2 91.6 459:45.62 glusterfs Tue Jul 23 02:11:08 IST 2019 10339 root 20 0 16.7g 14.2g 1868 S 0.0 91.9 459:48.30 glusterfs Tue Jul 23 02:13:08 IST 2019 10339 root 20 0 16.7g 14.3g 1868 S 0.0 92.1 459:50.95 glusterfs Tue Jul 23 02:15:09 IST 2019 10339 root 20 0 16.7g 14.2g 1868 S 0.0 91.5 459:53.69 glusterfs Tue Jul 23 02:17:09 IST 2019 10339 root 20 0 16.8g 14.2g 1868 S 0.0 91.8 459:56.28 glusterfs Tue Jul 23 02:19:09 IST 2019 10339 root 20 0 16.9g 14.2g 1868 S 0.0 91.5 459:58.90 glusterfs Tue Jul 23 02:21:09 IST 2019 10339 root 20 0 16.9g 14.2g 1868 S 0.0 91.8 460:01.55 glusterfs Tue Jul 23 02:23:10 IST 2019 10339 root 20 0 16.9g 14.3g 1868 S 0.0 92.0 460:04.18 glusterfs Tue Jul 23 02:25:10 IST 2019 10339 root 20 0 17.0g 14.3g 1868 S 0.0 92.3 460:06.86 glusterfs Tue Jul 23 02:27:10 IST 2019 10339 root 20 0 17.0g 14.2g 1868 S 0.0 92.0 460:09.57 glusterfs Tue Jul 23 02:29:10 IST 2019 10339 root 20 0 17.1g 14.3g 1868 S 0.0 92.2 460:12.03 glusterfs Tue Jul 23 02:31:11 IST 2019 10339 root 20 0 17.1g 14.3g 1868 S 0.0 92.5 460:14.49 glusterfs Tue Jul 23 02:33:11 IST 2019 10339 root 20 0 17.1g 14.2g 1868 S 0.0 91.6 460:17.00 glusterfs Tue Jul 23 02:35:11 IST 2019 10339 root 20 0 17.2g 14.2g 1868 S 0.0 91.8 460:19.49 glusterfs Tue Jul 23 02:37:11 IST 2019 10339 root 20 0 17.2g 14.2g 1868 S 0.0 91.8 460:21.97 glusterfs Tue Jul 23 02:39:12 IST 2019 10339 root 20 0 17.2g 14.3g 1868 S 0.0 92.1 460:24.44 glusterfs Tue Jul 23 02:41:12 IST 2019 10339 root 20 0 17.3g 14.3g 1868 S 0.0 92.3 460:26.99 glusterfs Tue Jul 23 02:43:12 IST 2019 10339 root 20 0 17.3g 14.3g 1868 S 0.0 92.4 460:29.49 glusterfs Tue Jul 23 02:45:13 IST 2019 10339 root 20 0 17.4g 14.3g 1868 S 0.0 92.5 460:31.95 glusterfs Tue Jul 23 02:47:13 IST 2019 10339 root 20 0 17.4g 14.2g 1868 S 6.2 92.0 460:34.44 glusterfs Tue Jul 23 02:49:13 IST 2019 10339 root 20 0 17.4g 14.3g 1868 S 12.5 92.3 460:36.95 glusterfs Tue Jul 23 02:51:13 IST 2019 10339 root 20 0 17.5g 14.3g 1868 S 0.0 92.6 460:39.57 glusterfs Tue Jul 23 02:53:14 IST 2019 10339 root 20 0 17.6g 14.3g 1868 S 0.0 92.6 460:42.09 glusterfs Tue Jul 23 02:55:14 IST 2019 10339 root 20 0 17.6g 14.4g 1868 S 0.0 92.8 460:44.62 glusterfs Tue Jul 23 02:57:14 IST 2019 10339 root 20 0 17.6g 14.3g 1868 S 0.0 92.4 460:47.16 glusterfs Tue Jul 23 02:59:14 IST 2019 10339 root 20 0 17.7g 14.3g 1868 S 0.0 92.4 460:49.70 glusterfs Tue Jul 23 03:01:15 IST 2019 10339 root 20 0 17.7g 14.2g 1868 S 0.0 92.0 460:52.24 glusterfs Tue Jul 23 03:03:15 IST 2019 10339 root 20 0 17.7g 14.3g 1868 S 0.0 92.3 460:54.80 glusterfs Tue Jul 23 03:05:15 IST 2019 10339 root 20 0 17.8g 14.2g 1868 S 6.2 92.0 460:57.37 glusterfs Tue Jul 23 03:07:16 IST 2019 10339 root 20 0 17.8g 14.2g 1868 S 0.0 91.5 460:59.84 glusterfs Tue Jul 23 03:09:16 IST 2019 10339 root 20 0 17.9g 14.2g 1868 S 6.2 91.7 461:02.32 glusterfs Tue Jul 23 03:11:16 IST 2019 10339 root 20 0 17.9g 14.2g 1868 S 0.0 92.0 461:04.77 glusterfs Tue Jul 23 03:13:16 IST 2019 10339 root 20 0 17.9g 14.3g 1228 S 5.9 92.3 461:07.34 glusterfs Tue Jul 23 03:15:17 IST 2019 10339 root 20 0 18.0g 14.3g 1216 S 0.0 92.5 461:09.83 glusterfs Tue Jul 23 03:17:18 IST 2019 10339 root 20 0 18.1g 14.3g 1296 S 0.0 92.5 461:12.31 glusterfs Tue Jul 23 03:19:18 IST 2019 10339 root 20 0 18.1g 14.3g 1296 S 0.0 92.3 461:14.79 glusterfs Tue Jul 23 03:21:18 IST 2019 10339 root 20 0 18.1g 14.3g 1296 S 0.0 92.6 461:17.28 glusterfs Tue Jul 23 03:23:18 IST 2019 10339 root 20 0 18.2g 14.3g 1296 S 0.0 92.6 461:19.78 glusterfs Tue Jul 23 03:25:19 IST 2019 10339 root 20 0 18.2g 14.4g 1296 S 0.0 92.8 461:22.25 glusterfs Tue Jul 23 03:27:19 IST 2019 10339 root 20 0 18.2g 14.3g 1296 S 0.0 92.6 461:24.75 glusterfs Tue Jul 23 03:29:19 IST 2019 10339 root 20 0 18.2g 14.4g 1296 S 0.0 92.9 461:27.21 glusterfs Tue Jul 23 03:31:19 IST 2019 10339 root 20 0 18.3g 14.3g 1296 S 0.0 92.0 461:29.71 glusterfs Tue Jul 23 03:33:20 IST 2019 10339 root 20 0 18.4g 14.3g 1296 S 0.0 92.3 461:32.20 glusterfs Tue Jul 23 03:35:20 IST 2019 10339 root 20 0 18.4g 14.2g 1300 S 0.0 91.4 461:34.71 glusterfs Tue Jul 23 03:37:20 IST 2019 10339 root 20 0 18.4g 14.2g 1300 S 0.0 91.7 461:37.56 glusterfs Tue Jul 23 03:39:20 IST 2019 10339 root 20 0 18.5g 14.2g 1300 S 0.0 92.0 461:40.27 glusterfs Tue Jul 23 03:41:21 IST 2019 10339 root 20 0 18.5g 14.3g 1300 S 0.0 92.2 461:42.93 glusterfs Tue Jul 23 03:43:21 IST 2019 10339 root 20 0 18.6g 14.3g 1300 S 6.7 92.4 461:45.57 glusterfs Tue Jul 23 03:45:21 IST 2019 10339 root 20 0 18.6g 14.4g 1300 S 12.5 92.7 461:48.36 glusterfs Tue Jul 23 03:47:21 IST 2019 10339 root 20 0 18.6g 14.4g 1300 S 6.7 92.7 461:50.92 glusterfs Tue Jul 23 03:49:22 IST 2019 10339 root 20 0 18.7g 14.4g 1300 S 0.0 93.0 461:53.54 glusterfs Tue Jul 23 03:51:22 IST 2019 10339 root 20 0 18.7g 14.4g 1300 S 18.8 93.0 461:56.15 glusterfs Tue Jul 23 03:53:22 IST 2019 10339 root 20 0 18.7g 14.3g 1300 S 0.0 92.1 461:58.85 glusterfs Tue Jul 23 03:55:22 IST 2019 10339 root 20 0 18.8g 14.3g 1300 S 0.0 92.4 462:01.58 glusterfs Tue Jul 23 03:57:23 IST 2019 10339 root 20 0 18.8g 14.1g 1300 S 0.0 90.9 462:04.06 glusterfs Tue Jul 23 03:59:23 IST 2019 10339 root 20 0 18.9g 14.1g 1300 S 0.0 91.1 462:06.51 glusterfs Tue Jul 23 04:01:23 IST 2019 10339 root 20 0 18.9g 14.1g 1308 S 0.0 91.4 462:09.19 glusterfs Tue Jul 23 04:03:23 IST 2019 10339 root 20 0 18.9g 14.2g 1308 S 0.0 91.7 462:11.95 glusterfs Tue Jul 23 04:05:24 IST 2019 10339 root 20 0 19.0g 14.2g 1308 S 0.0 91.9 462:14.62 glusterfs Tue Jul 23 04:07:24 IST 2019 10339 root 20 0 19.0g 14.3g 1316 S 6.2 92.2 462:17.35 glusterfs Tue Jul 23 04:09:24 IST 2019 10339 root 20 0 19.1g 14.3g 1316 S 0.0 92.5 462:20.08 glusterfs Tue Jul 23 04:11:24 IST 2019 10339 root 20 0 19.1g 14.2g 1316 S 0.0 91.9 462:22.72 glusterfs Tue Jul 23 04:13:25 IST 2019 10339 root 20 0 19.1g 14.3g 1316 S 0.0 92.2 462:25.24 glusterfs Tue Jul 23 04:15:25 IST 2019 10339 root 20 0 19.2g 14.3g 1316 S 6.2 92.4 462:27.75 glusterfs Tue Jul 23 04:17:25 IST 2019 10339 root 20 0 19.2g 14.4g 1316 S 0.0 92.7 462:30.26 glusterfs Tue Jul 23 04:19:25 IST 2019 10339 root 20 0 19.2g 14.4g 1316 S 0.0 92.8 462:32.79 glusterfs Tue Jul 23 04:21:26 IST 2019 10339 root 20 0 19.3g 14.4g 1316 S 6.2 93.0 462:35.34 glusterfs Tue Jul 23 04:23:26 IST 2019 10339 root 20 0 19.4g 14.4g 1316 S 0.0 93.1 462:37.85 glusterfs Tue Jul 23 04:25:26 IST 2019 10339 root 20 0 19.4g 14.2g 1316 S 0.0 92.0 462:40.37 glusterfs Tue Jul 23 04:27:27 IST 2019 10339 root 20 0 19.4g 14.1g 1020 S 10.5 91.1 462:42.93 glusterfs Tue Jul 23 04:29:27 IST 2019 10339 root 20 0 19.5g 14.1g 1020 S 0.0 91.2 462:45.49 glusterfs Tue Jul 23 04:31:28 IST 2019 10339 root 20 0 19.5g 14.2g 1020 S 6.2 91.5 462:48.04 glusterfs Tue Jul 23 04:33:28 IST 2019 10339 root 20 0 19.6g 14.2g 1020 S 0.0 91.8 462:50.52 glusterfs Tue Jul 23 04:35:28 IST 2019 10339 root 20 0 19.6g 14.2g 1036 S 0.0 92.0 462:53.06 glusterfs Tue Jul 23 04:37:29 IST 2019 10339 root 20 0 19.6g 14.3g 1036 S 0.0 92.3 462:55.60 glusterfs Tue Jul 23 04:39:29 IST 2019 10339 root 20 0 19.7g 14.3g 1036 S 6.2 92.4 462:58.13 glusterfs Tue Jul 23 04:41:29 IST 2019 10339 root 20 0 19.7g 14.3g 1132 S 12.5 92.7 463:00.69 glusterfs Tue Jul 23 04:43:29 IST 2019 10339 root 20 0 19.7g 14.3g 1132 S 6.2 92.5 463:03.19 glusterfs Tue Jul 23 04:45:30 IST 2019 10339 root 20 0 19.8g 14.4g 1132 S 0.0 92.8 463:05.69 glusterfs Tue Jul 23 04:47:30 IST 2019 10339 root 20 0 19.9g 14.4g 1056 S 0.0 93.0 463:08.18 glusterfs Tue Jul 23 04:49:31 IST 2019 10339 root 20 0 19.9g 14.3g 1056 S 0.0 92.3 463:10.73 glusterfs Tue Jul 23 04:51:31 IST 2019 10339 root 20 0 19.9g 14.3g 1056 S 0.0 92.5 463:13.23 glusterfs Tue Jul 23 04:53:32 IST 2019 10339 root 20 0 20.0g 14.3g 1096 S 0.0 92.6 463:15.77 glusterfs Tue Jul 23 04:55:32 IST 2019 10339 root 20 0 20.0g 14.4g 1096 S 12.5 92.9 463:18.27 glusterfs Tue Jul 23 04:57:32 IST 2019 10339 root 20 0 20.1g 14.3g 1096 S 12.5 92.1 463:20.75 glusterfs Tue Jul 23 04:59:33 IST 2019 10339 root 20 0 20.1g 14.3g 1128 S 0.0 92.3 463:23.26 glusterfs Tue Jul 23 05:01:33 IST 2019 10339 root 20 0 20.1g 14.3g 1128 S 0.0 92.6 463:25.74 glusterfs Tue Jul 23 05:03:33 IST 2019 10339 root 20 0 20.2g 14.3g 1128 S 0.0 92.6 463:28.20 glusterfs Tue Jul 23 05:05:34 IST 2019 10339 root 20 0 20.2g 14.2g 1128 S 6.2 92.0 463:30.72 glusterfs Tue Jul 23 05:07:34 IST 2019 10339 root 20 0 20.2g 14.3g 1128 S 0.0 92.3 463:33.27 glusterfs Tue Jul 23 05:09:34 IST 2019 10339 root 20 0 20.3g 14.3g 1128 S 0.0 92.2 463:35.78 glusterfs Tue Jul 23 05:11:35 IST 2019 10339 root 20 0 20.4g 14.3g 1128 S 0.0 92.4 463:38.34 glusterfs Tue Jul 23 05:13:35 IST 2019 10339 root 20 0 20.4g 14.3g 1128 S 6.2 92.7 463:40.84 glusterfs Tue Jul 23 05:15:35 IST 2019 10339 root 20 0 20.4g 14.2g 1128 S 0.0 91.8 463:43.37 glusterfs Tue Jul 23 05:17:35 IST 2019 10339 root 20 0 20.4g 14.2g 1128 S 0.0 92.0 463:45.91 glusterfs Tue Jul 23 05:19:36 IST 2019 10339 root 20 0 20.5g 14.2g 1128 S 0.0 91.5 463:48.40 glusterfs Tue Jul 23 05:21:36 IST 2019 10339 root 20 0 20.6g 14.2g 1128 S 0.0 91.7 463:50.84 glusterfs Tue Jul 23 05:23:36 IST 2019 10339 root 20 0 20.6g 14.2g 1128 S 0.0 92.0 463:53.37 glusterfs Tue Jul 23 05:25:36 IST 2019 10339 root 20 0 20.6g 14.3g 1128 S 0.0 92.3 463:55.89 glusterfs Tue Jul 23 05:27:37 IST 2019 10339 root 20 0 20.7g 14.3g 1128 S 0.0 92.1 463:58.38 glusterfs Tue Jul 23 05:29:37 IST 2019 10339 root 20 0 20.7g 14.3g 1128 S 6.2 92.3 464:00.91 glusterfs Tue Jul 23 05:31:37 IST 2019 10339 root 20 0 20.7g 14.3g 1128 S 6.7 92.6 464:03.37 glusterfs Tue Jul 23 05:33:37 IST 2019 10339 root 20 0 20.8g 14.3g 1128 S 0.0 92.5 464:05.86 glusterfs Tue Jul 23 05:35:38 IST 2019 10339 root 20 0 20.8g 14.2g 1128 S 0.0 92.0 464:08.31 glusterfs Tue Jul 23 05:37:38 IST 2019 10339 root 20 0 20.9g 14.3g 1128 S 0.0 92.3 464:10.83 glusterfs Tue Jul 23 05:39:38 IST 2019 10339 root 20 0 20.9g 14.3g 1140 S 0.0 92.5 464:13.34 glusterfs Tue Jul 23 05:41:39 IST 2019 10339 root 20 0 20.9g 14.4g 1140 S 0.0 92.8 464:15.82 glusterfs Tue Jul 23 05:43:39 IST 2019 10339 root 20 0 21.0g 14.3g 1140 S 0.0 92.4 464:18.32 glusterfs Tue Jul 23 05:45:39 IST 2019 10339 root 20 0 21.0g 14.4g 1140 S 0.0 92.7 464:20.87 glusterfs Tue Jul 23 05:47:40 IST 2019 10339 root 20 0 21.1g 14.4g 1140 S 6.2 92.9 464:23.37 glusterfs Tue Jul 23 05:49:40 IST 2019 10339 root 20 0 21.1g 14.4g 1140 S 0.0 93.2 464:25.85 glusterfs Tue Jul 23 05:51:40 IST 2019 10339 root 20 0 21.1g 14.5g 1140 S 0.0 93.4 464:28.35 glusterfs Tue Jul 23 05:53:41 IST 2019 10339 root 20 0 21.2g 14.4g 1140 S 0.0 93.0 464:30.83 glusterfs Tue Jul 23 05:55:41 IST 2019 10339 root 20 0 21.2g 14.4g 1140 S 12.5 93.3 464:33.32 glusterfs Tue Jul 23 05:57:41 IST 2019 10339 root 20 0 21.2g 14.5g 1140 S 6.2 93.5 464:35.86 glusterfs Tue Jul 23 05:59:41 IST 2019 10339 root 20 0 21.3g 14.5g 1140 S 10.5 93.8 464:38.33 glusterfs Tue Jul 23 06:01:42 IST 2019 10339 root 20 0 21.3g 14.5g 1140 S 0.0 93.4 464:40.83 glusterfs Tue Jul 23 06:03:42 IST 2019 10339 root 20 0 21.4g 14.5g 1140 S 0.0 93.6 464:43.34 glusterfs Tue Jul 23 06:05:42 IST 2019 10339 root 20 0 21.4g 14.5g 1140 S 0.0 93.8 464:45.84 glusterfs Tue Jul 23 06:07:43 IST 2019 10339 root 20 0 21.4g 14.5g 1140 S 0.0 93.8 464:48.36 glusterfs Tue Jul 23 06:09:43 IST 2019 10339 root 20 0 21.5g 14.6g 1140 S 0.0 94.0 464:50.85 glusterfs Tue Jul 23 06:11:43 IST 2019 10339 root 20 0 21.5g 14.5g 1140 S 0.0 93.8 464:53.28 glusterfs Tue Jul 23 06:13:43 IST 2019 10339 root 20 0 21.6g 14.5g 1140 S 0.0 93.7 464:55.89 glusterfs Tue Jul 23 06:15:43 IST 2019 10339 root 20 0 21.6g 14.4g 1140 S 0.0 92.9 464:58.70 glusterfs Tue Jul 23 06:17:44 IST 2019 10339 root 20 0 21.6g 14.4g 1140 S 0.0 93.2 465:01.41 glusterfs Tue Jul 23 06:19:44 IST 2019 10339 root 20 0 21.7g 14.3g 1140 S 0.0 92.6 465:03.95 glusterfs Tue Jul 23 06:21:45 IST 2019 10339 root 20 0 21.7g 14.4g 1140 S 0.0 92.9 465:06.44 glusterfs Tue Jul 23 06:23:45 IST 2019 10339 root 20 0 21.7g 14.4g 1140 S 0.0 93.1 465:08.84 glusterfs Tue Jul 23 06:25:45 IST 2019 10339 root 20 0 21.8g 14.5g 1140 S 0.0 93.4 465:11.30 glusterfs Tue Jul 23 06:27:46 IST 2019 10339 root 20 0 21.8g 14.5g 1140 S 0.0 93.6 465:13.78 glusterfs Tue Jul 23 06:29:46 IST 2019 10339 root 20 0 21.9g 14.5g 1140 S 0.0 93.6 465:16.26 glusterfs Tue Jul 23 06:31:46 IST 2019 10339 root 20 0 21.9g 14.5g 1140 S 0.0 93.5 465:18.72 glusterfs Tue Jul 23 06:33:47 IST 2019 10339 root 20 0 21.9g 14.4g 1140 S 0.0 92.9 465:21.16 glusterfs Tue Jul 23 06:35:47 IST 2019 10339 root 20 0 22.0g 14.4g 1140 S 0.0 93.1 465:23.65 glusterfs Tue Jul 23 06:37:47 IST 2019 10339 root 20 0 22.0g 14.5g 1140 S 0.0 93.4 465:26.10 glusterfs Tue Jul 23 06:39:48 IST 2019 10339 root 20 0 22.1g 14.5g 1140 S 0.0 93.6 465:28.55 glusterfs Tue Jul 23 06:41:48 IST 2019 10339 root 20 0 22.1g 14.5g 1140 S 0.0 93.7 465:31.06 glusterfs Tue Jul 23 06:43:48 IST 2019 10339 root 20 0 22.1g 14.4g 1140 S 6.2 93.3 465:33.50 glusterfs Tue Jul 23 06:45:48 IST 2019 10339 root 20 0 22.2g 14.5g 1140 S 0.0 93.5 465:35.97 glusterfs Tue Jul 23 06:47:49 IST 2019 10339 root 20 0 22.2g 14.5g 1140 S 0.0 93.8 465:38.44 glusterfs Tue Jul 23 06:49:49 IST 2019 10339 root 20 0 22.2g 14.5g 1140 S 0.0 93.9 465:40.90 glusterfs Tue Jul 23 06:51:49 IST 2019 10339 root 20 0 22.3g 14.4g 1140 S 0.0 93.0 465:43.40 glusterfs Tue Jul 23 06:53:50 IST 2019 10339 root 20 0 22.4g 14.3g 1140 S 0.0 92.4 465:45.92 glusterfs Tue Jul 23 06:55:50 IST 2019 10339 root 20 0 22.4g 14.3g 1140 S 6.2 92.6 465:48.42 glusterfs Tue Jul 23 06:57:50 IST 2019 10339 root 20 0 22.4g 14.4g 1140 S 6.7 92.9 465:50.90 glusterfs Tue Jul 23 06:59:50 IST 2019 10339 root 20 0 22.5g 14.4g 1140 S 0.0 93.1 465:53.36 glusterfs Tue Jul 23 07:01:51 IST 2019 10339 root 20 0 22.5g 14.5g 1140 S 0.0 93.4 465:55.85 glusterfs Tue Jul 23 07:03:51 IST 2019 10339 root 20 0 22.6g 14.5g 1140 S 0.0 93.7 465:58.37 glusterfs Tue Jul 23 07:05:51 IST 2019 10339 root 20 0 22.6g 14.3g 1140 S 0.0 92.4 466:00.85 glusterfs Tue Jul 23 07:07:51 IST 2019 10339 root 20 0 22.6g 14.3g 1140 S 0.0 92.6 466:03.31 glusterfs Tue Jul 23 07:09:52 IST 2019 10339 root 20 0 22.7g 14.4g 1156 S 0.0 92.9 466:05.79 glusterfs Tue Jul 23 07:11:52 IST 2019 10339 root 20 0 22.7g 14.4g 1156 S 6.2 93.2 466:08.29 glusterfs Tue Jul 23 07:13:52 IST 2019 10339 root 20 0 22.7g 14.4g 1156 S 0.0 93.3 466:10.75 glusterfs Tue Jul 23 07:15:53 IST 2019 10339 root 20 0 22.8g 14.5g 1156 S 0.0 93.5 466:13.23 glusterfs Tue Jul 23 07:17:53 IST 2019 10339 root 20 0 22.8g 14.5g 1156 S 0.0 93.7 466:15.69 glusterfs Tue Jul 23 07:19:53 IST 2019 10339 root 20 0 22.9g 14.4g 1156 S 0.0 93.2 466:18.18 glusterfs Tue Jul 23 07:21:53 IST 2019 10339 root 20 0 22.9g 14.5g 1156 S 0.0 93.4 466:20.68 glusterfs Tue Jul 23 07:23:54 IST 2019 10339 root 20 0 22.9g 14.4g 1156 S 0.0 93.3 466:23.19 glusterfs Tue Jul 23 07:25:54 IST 2019 10339 root 20 0 23.0g 14.5g 1156 S 12.5 93.5 466:25.70 glusterfs Tue Jul 23 07:27:54 IST 2019 10339 root 20 0 23.1g 14.5g 1156 S 0.0 93.6 466:28.17 glusterfs Tue Jul 23 07:29:54 IST 2019 10339 root 20 0 23.1g 14.5g 1156 S 6.2 93.7 466:30.68 glusterfs Tue Jul 23 07:31:55 IST 2019 10339 root 20 0 23.1g 14.5g 1156 S 6.7 93.4 466:33.17 glusterfs Tue Jul 23 07:33:55 IST 2019 10339 root 20 0 23.2g 14.5g 1156 S 0.0 93.7 466:35.63 glusterfs Tue Jul 23 07:35:56 IST 2019 10339 root 20 0 22.9g 14.5g 1020 S 0.0 93.7 466:38.13 glusterfs Tue Jul 23 07:37:56 IST 2019 10339 root 20 0 23.0g 14.5g 1036 S 0.0 93.9 466:40.66 glusterfs Tue Jul 23 07:39:57 IST 2019 10339 root 20 0 23.0g 14.6g 1036 S 0.0 94.2 466:43.30 glusterfs Tue Jul 23 07:42:17 IST 2019 10339 root 20 0 23.1g 14.6g 0 S 0.1 94.4 466:50.63 glusterfs Tue Jul 23 07:49:11 IST 2019 10339 root 20 0 23.1g 14.6g 0 S 0.9 94.5 467:03.41 glusterfs Tue Jul 23 07:53:23 IST 2019 Tue Jul 23 07:55:23 IST 2019 Tue Jul 23 07:57:24 IST 2019 Tue Jul 23 07:59:24 IST 2019 Tue Jul 23 08:01:24 IST 2019 Tue Jul 23 08:03:24 IST 2019 Tue Jul 23 08:05:25 IST 2019 Tue Jul 23 08:07:25 IST 2019 Tue Jul 23 08:09:25 IST 2019 Tue Jul 23 08:11:25 IST 2019 Tue Jul 23 08:13:25 IST 2019 Tue Jul 23 08:15:26 IST 2019 Tue Jul 23 08:17:26 IST 2019
Few more important notes: ======================== 1) Although, as described in description c#0 this bug was raised as a possible after effect of BZ#1731826, however I am able to reproduce it independently too with steps in c#19 2) there are 2 issues observed which are as below i) the lookup hanging on the fuse mount ii) once the lookup hung is observed ie above issue, then if the user does an interrupt to the lookup, ie using ctrl+c a couple of times, then the fuse mount starts to consume resident memory and the leak finally leads to OOM kill as I discussed in bug triage meeting, it would be more easier to track both the issues seperately. Hence keeping this bug for the lookup hung , and will raise a new bug for oom kill
(In reply to nchilaka from comment #21) > Few more important notes: > ======================== > 1) Although, as described in description c#0 this bug was raised as a > possible after effect of BZ#1731826, however I am able to reproduce it > independently too with steps in c#19 > 2) there are 2 issues observed which are as below > i) the lookup hanging on the fuse mount > ii) once the lookup hung is observed ie above issue, then if the user > does an interrupt to the lookup, ie using ctrl+c a couple of times, then the > fuse mount starts to consume resident memory and the leak finally leads to > OOM kill > as I discussed in bug triage meeting, it would be more easier to track both > the issues seperately. Hence keeping this bug for the lookup hung , and will > raise a new bug for oom kill raised BZ#1736830 for the OOM kill
(gdb) p $4->locks[0] $5 = {lock = 0x7f3da4abc1d8, fop = 0x7f3d74317e18, owner_list = {next = 0x7f3d74317ed0, prev = 0x7f3d74317ed0}, wait_list = {next = 0x7f3da4abc208, prev = 0x7f3da4abc208}, update = {false, false}, dirty = { false, false}, optimistic_changelog = false, base = 0x0, size = 0, waiting_flags = 0, fl_start = 0, fl_end = 9223372036854775807} (gdb) p $4->locks[0].lock $6 = (ec_lock_t *) 0x7f3da4abc1d8 (gdb) p *$4->locks[0].lock $7 = {ctx = 0x7f3db7cbff70, timer = 0x0, owners = {next = 0x7f3da4abc1e8, prev = 0x7f3da4abc1e8}, waiting = {next = 0x7f3da4abc1f8, prev = 0x7f3da4abc1f8}, frozen = {next = 0x7f3d74317ee0, prev = 0x7f3d74317ee0}, mask = 0, good_mask = 18446744073709551615, healing = 0, refs_owners = 0, refs_pending = 0, waiting_flags = 0, acquired = false, unlock_now = false, release = true, query = true, fd = 0x0, loc = {path = 0x7f3d75084a40 "/IOs/kernel/rhs-client45.lab.eng.blr.redhat.com/dir.2/linux-5.2.7/Documentation/devicetree/bindings/rtc", name = 0x7f3d75084aa4 "rtc", inode = 0x7f3d98014768, parent = 0x7f3d99faad38, gfid = "\310\a\376|-\205K\v\215\000\b\363>\241\021i", pargfid = "\345\330}\212\242{Nr\233\064\373\030MD\361", <incomplete sequence \360>}, {type = ENTRYLK_WRLCK, flock = { l_type = 1, l_whence = 0, l_start = 0, l_len = 0, l_pid = 0, l_owner = {len = 0, data = '\000' <repeats 1023 times>}}}} (gdb) p &$4->locks[0].lock->owners $8 = (struct list_head *) 0x7f3da4abc1e8 (gdb) p &$4->locks[0].lock->waiting $9 = (struct list_head *) 0x7f3da4abc1f8 (gdb) p &$4->locks[0].lock->frozen $10 = (struct list_head *) 0x7f3da4abc208 This seems to suggest that the fop is stuck in frozen list which can only happen if lock->release is set to true. I have one hypothesis where this is a possibility and will test it out and update the bz with my findings.
I was not able to simulate the race completely so as we discussed I am providing the scratch build: http://brew-task-repos.usersys.redhat.com/repos/scratch/pkarampu/glusterfs/6.0/12.el7rhgs/x86_64 Let me know what you find with this package. You will need to umount/mount after installing these rpms.
(In reply to Pranith Kumar K from comment #30) > I was not able to simulate the race completely so as we discussed I am > providing the scratch build: > http://brew-task-repos.usersys.redhat.com/repos/scratch/pkarampu/glusterfs/6. > 0/12.el7rhgs/x86_64 Let me know what you find with this package. You will > need to umount/mount after installing these rpms. Hi Pranith, I have retested with the above bits, and didn't hit the hang again, even after 1.5 days of lookups. Seems like the fix has helped resolve the issue.
https://review.gluster.org/c/glusterfs/+/23272
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:3249