Bug 1764202

Summary: cgroup control-cpu-load.sh script not working
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nag Pavan Chilakam <nchilaka>
Component: coreAssignee: Mohit Agrawal <moagrawa>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: urgent Docs Contact:
Priority: urgent    
Version: rhgs-3.5CC: rhs-bugs, rkothiya, sheggodu, storage-qa-internal, vdas
Target Milestone: ---Keywords: Regression
Target Release: RHGS 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0-20 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1764208 (view as bug list) Environment:
Last Closed: 2019-10-30 12:23:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1696809, 1764208, 1766424, 1766425    

Description Nag Pavan Chilakam 2019-10-22 13:04:31 UTC
Description of problem:
==============================
i ran cgroup script to control cpu usage as glusterfsd was consuming more than 3000%
However it is not bring the cpu consumption under the mentioned range.

I am referring to script /usr/share/glusterfs/scripts/control-cpu-load.sh


Important note: I didn't test control-mem.sh
Kindly check the same for even control-mem.sh
Version-Release number of selected component (if applicable):
==================
6.0.15

How reproducible:
===========
always


[root@rhs-gp-srv2 ~]# top -n 1 -b|egrep 'RES|gluster'
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6722 root      20   0   19.9g   3.8g   3428 S  1250  8.1 229868:38 glusterfsd
 7421 root      20   0   13.4g   4.7g   4416 S  30.0  9.9   5002:00 glusterfs
 6173 root      20   0  596100  23596   2548 S   0.0  0.0   0:20.29 glusterd
 7410 root      20   0 1453872  16424   2220 S   0.0  0.0   6:18.91 glusterfsd
[root@rhs-gp-srv2 ~]# 
[root@rhs-gp-srv2 ~]# pwd
/root
[root@rhs-gp-srv2 ~]# cd /usr/share/glusterfs/scripts
[root@rhs-gp-srv2 scripts]# ./control-cpu-load.sh 
Enter gluster daemon pid for which you want to control CPU.
glusterfsd
Entered daemon_pid is not numeric so Rerun the script.
[root@rhs-gp-srv2 scripts]# ./control-cpu-load.sh 
Enter gluster daemon pid for which you want to control CPU.
6722
If you want to continue the script to attach 6722 with new cgroup_gluster_6722 cgroup Press (y/n)?n
no
[root@rhs-gp-srv2 scripts]# ./control-cpu-load.sh 
Enter gluster daemon pid for which you want to control CPU.
6722
If you want to continue the script to attach 6722 with new cgroup_gluster_6722 cgroup Press (y/n)?y
yes
Creating child cgroup directory 'cgroup_gluster_6722 cgroup' for glusterd.service.
Enter quota value in range [10,100]:  
25
Entered quota value is 25
Setting 25000 to cpu.cfs_quota_us for gluster_cgroup.
Tasks are attached successfully specific to 6722 to cgroup_gluster_6722.
[root@rhs-gp-srv2 scripts]# top -n 1 -b|egrep 'RES|gluster'
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6722 root      20   0   19.9g   3.8g   3392 S  2845  8.1 229895:21 glusterfsd
 7421 root      20   0   13.4g   4.9g   4416 S  30.0 10.5   5002:22 glusterfs
 6173 root      20   0  596100  23448   2400 S   0.0  0.0   0:20.29 glusterd
 7410 root      20   0 1453872  16424   2220 S   0.0  0.0   6:18.91 glusterfsd
[root@rhs-gp-srv2 scripts]# top -n 1 -b|egrep 'RES|gluster'
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6722 root      20   0   19.9g   3.8g   3392 S  2640  8.1 229896:07 glusterfsd
 7421 root      20   0   13.4g   5.0g   4416 S  25.0 10.6   5002:23 glusterfs
 6173 root      20   0  596100  23448   2400 S   0.0  0.0   0:20.29 glusterd
 7410 root      20   0 1453872  16424   2220 S   0.0  0.0   6:18.91 glusterfsd
[root@rhs-gp-srv2 scripts]# top -n 1 -b|egrep 'RES|gluster'
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6722 root      20   0   19.9g   3.8g   3392 S  2479  8.1 229896:38 glusterfsd
 7421 root      20   0   13.4g   5.0g   4416 S  31.6 10.6   5002:23 glusterfs
 6173 root      20   0  596100  23448   2400 S   0.0  0.0   0:20.29 glusterd
 7410 root      20   0 1453872  16424   2220 S   0.0  0.0   6:18.91 glusterfsd



Steps to Reproduce:
=================
1.have a setup where cpu consumption of gluster procs are high
2. run /usr/share/glusterfs/scripts/control-cpu-load.sh 
3. enter the pid of gluster proc and the limit under which the cpu must be consumed.

Actual results:
=============
script runs but cpu consumption not coming under control

Comment 9 Nag Pavan Chilakam 2019-10-24 12:40:51 UTC
tested with the script made available with 6.0.20 packages.
The cpu consumption is now under control with the usage of control-cpu-load.sh script
hence moving to verified

Comment 11 errata-xmlrpc 2019-10-30 12:23:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249