Bug 1463397 - "docker stats" command reports inaccurate CPU usage
"docker stats" command reports inaccurate CPU usage
Status: CLOSED WONTFIX
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers (Show other bugs)
3.2.1
Unspecified Unspecified
unspecified Severity low
: ---
: ---
Assigned To: Jhon Honce
DeShuai Ma
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-20 15:08 EDT by Steven Walter
Modified: 2017-06-27 12:55 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-27 12:55:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Steven Walter 2017-06-20 15:08:32 EDT
Description of problem:

`docker stats` command indicates CPU usage by a container consistently above 300% (in a multi-core system). However the `top` output indicates a sub-200% CPU usage, which is consistent with resource limits placed on the pod.

Version-Release number of selected component (if applicable):
3.2.1

How reproducible:
Confirmed

Steps to Reproduce:
1. run docker stats in one terminal
2. run top in another
3. Observe the two

Actual results:


CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O
cadaaad35832        314.27%             4.969 GB / 6.442 GB   77.14%              909.4 MB / 646 MB   49.76 MB / 0 B

TOP for container:

top - 15:53:01 up 111 days, 16:22,  0 users,  load average: 10.51, 22.90, 28.23
Tasks:  12 total,   1 running,  11 sleeping,   0 stopped,   0 zombie
%Cpu(s): 35.1 us, 39.6 sy,  0.0 ni, 22.0 id,  1.6 wa,  0.0 hi,  1.7 si,  0.0 st
KiB Mem : 65802252 total, 22708220 free, 28383288 used, 14710744 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 36450864 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                           
     1 1010010+  20   0 8151860 3.587g   8996 S 169.0  5.7   2459:13 mysqld                                                                                            
 15415 1010010+  20   0 3348008  24024   2828 S  31.3  0.0  10:02.22 sysbench                                                                                          
 19971 1010010+  20   0   53988   2008   1448 R   0.3  0.0   0:00.01 top                                                                                               
    24 1010010+  20   0    4356    340    268 S   0.0  0.0   3:53.19 tail                                                                                              
   970 1010010+  20   0   11776   1720   1360 S   0.0  0.0   0:00.32 sh                                                                                                
 15416 1010010+  20   0    4316    360    284 S   0.0  0.0   0:00.00 tee                                                                                               
 19938 1010010+  20   0   11772   1712   1368 S   0.0  0.0   0:00.06 bash                                                                                              
 65597 1010010+  20   0   11772   1800   1444 S   0.0  0.0   0:00.10 sh                                                                                                
 65876 1010010+  20   0    4356    356    284 S   0.0  0.0   0:06.39 tail                                                                                              
120360 1010010+  20   0   11776   1812   1448 S   0.0  0.0   0:00.13 sh                                                                                                
121383 1010010+  20   0   13532   1792   1356 S   0.0  0.0   0:00.01 runtest.sh                                                                                        
123515 1010010+  20   0    4356    356    284 S   0.0  0.0   0:00.72 tail                                                                                              




Expected results:
Expect that the CPU would be more closely aligned between the two outputs

Additional info:
# docker version
Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-44.el7.x86_64
 Go version:      go1.4.2
 Git commit:      7ffc8ee-unsupported
 Built:           Fri Jun 17 15:27:21 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-44.el7.x86_64
 Go version:      go1.4.2
 Git commit:      7ffc8ee-unsupported
 Built:           Fri Jun 17 15:27:21 2016
 OS/Arch:         linux/amd64
root@apsrp6465:/root
# openshift version
openshift v3.2.1.4-1-g1864c8f
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

# oc get dc mysqldb -n mysql-pxc -o yaml
apiVersion: v1
kind: DeploymentConfig
metadata:
  name: mysqldb
. . .
    spec:
      containers:
      - env:
        - name: MYSQL_USER
          value: admin
        - name: MYSQL_PASSWORD
          value: admin
        - name: MYSQL_ROOT_PASSWORD
          value: admin
        - name: MYSQL_DATABASE
          value: sampledb
        - name: MYSQL_MEM_REQUEST
          value: 6Gi
        - name: MYSQL_CPU_REQUEST
          value: 1500m
. . .
        resources:
          limits:
            cpu: "2"
            memory: 6Gi
          requests:
            cpu: 1500m
            memory: 6Gi
. . .
Comment 2 Steven Walter 2017-06-20 15:11:41 EDT
Also of note, the machine only has 2 CPUs so customer is concerned over 300%+
Comment 3 Steven Walter 2017-06-20 15:13:37 EDT
(In reply to Steven Walter from comment #2)
> Also of note, the machine only has 2 CPUs so customer is concerned over 300%+

Ignore this -- misread; only 2 CPUs are allocated in limits, but machine has more than 2 CPU

Note You need to log in before you can comment on or make changes to this bug.