Bug 1581409

Summary: Limit ranges are being applied with cpu-cfs-quota set to false
Product: OpenShift Container Platform Reporter: Taneem Ibrahim <tibrahim>
Component: NodeAssignee: Seth Jennings <sjenning>
Status: CLOSED ERRATA QA Contact: DeShuai Ma <dma>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.7.0CC: aos-bugs, bleanhar, jokerman, mmccomas
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Fixes an issue where cfs_quota might still be set on a pod even when cpu-cfs-quota is set to false on the node
 Story Points: ---
Clone Of:
: 1581860 1581861 1581862 1581863 1582486 1582487 (view as bug list) Environment:
Last Closed: 2018-07-30 19:16:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1581860, 1581861, 1581862, 1581863, 1582486, 1582487    

Description Taneem Ibrahim 2018-05-22 17:03:21 UTC 
Description of problem:

This issue was originally thought to be resolved as part of https://bugzilla.redhat.com/show_bug.cgi?id=1554471 . However, the same issue is still showing up on v3.7.42 build. 

Version-Release number of selected component (if applicable):

v3.7.42
kubernetes v1.7.6+a08f5eeb62

How reproducible:

Not easily. Certain pods show the cpu.cfs_quota_us set at -1 as expected with cpu-cfs-quota flag set to false on the kubelet but several other pods show actual cpu values 1000,2000, etc.

Steps to Reproduce:

I have not been able to re-produce it yet on my local cluster unfortunately.

Actual results:


Expected results:


Additional info:

- attached node logs, pod yamls from working and non working.
- print out of the cpu.cfs_quota_us from various pods is added as a private comment. below.
Comment 9 DeShuai Ma 2018-05-25 15:52:44 UTC 
Verify build https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=16434524

[root@ip-172-18-12-225 test]# rpm -qa|grep openshift
atomic-openshift-master-3.7.46-1.git.1.849c97d.el7.x86_64
atomic-openshift-docker-excluder-3.7.46-1.git.1.849c97d.el7.noarch
tuned-profiles-atomic-openshift-node-3.7.46-1.git.1.849c97d.el7.x86_64
atomic-openshift-excluder-3.7.46-1.git.1.849c97d.el7.noarch
atomic-openshift-clients-3.7.46-1.git.1.849c97d.el7.x86_64
atomic-openshift-node-3.7.46-1.git.1.849c97d.el7.x86_64
atomic-openshift-3.7.46-1.git.1.849c97d.el7.x86_64
atomic-openshift-sdn-ovs-3.7.46-1.git.1.849c97d.el7.x86_64

[root@ip-172-18-12-225 ~]# oc version
oc v3.7.46
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-12-225.ec2.internal:8443
openshift v3.7.46
kubernetes v1.7.6+a08f5eeb62

Verify steps:
//Scenario A cpu-cfs-quota=false
kubeletArguments:
  cpu-cfs-quota:
  - 'false'

1. Create a burstable pod 
[root@ip-172-18-12-225 ~]# oc create -f pod.yaml 
pod "burstable-vqpsr" created
[root@ip-172-18-12-225 ~]# 
[root@ip-172-18-12-225 ~]# oc get po -o yaml|grep -i uid
      openshift.io/scc: anyuid
    uid: 5a2dea60-6031-11e8-89e2-0eb7f09c6860

2. On node the pod located to check the cpu.cfs_quota_us in pod level cgroup.
[root@ip-172-18-5-70 test]# cat /sys/fs/cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2dea60_6031_11e8_89e2_0eb7f09c6860.slice/cpu.cfs_quota_us
-1

//Scenario B cpu-cfs-quota=true
kubeletArguments:
  cpu-cfs-quota:
  - 'true'
1. Create a burstable pod 
[root@ip-172-18-12-225 ~]# oc create -f pod.yaml 
pod "burstable-8tmdv" created
[root@ip-172-18-12-225 ~]# oc get po -o yaml|grep uid
      openshift.io/scc: anyuid
    uid: 9ec90ce2-6031-11e8-89e2-0eb7f09c6860
[root@ip-172-18-12-225 ~]# oc get po
NAME              READY     STATUS    RESTARTS   AGE
burstable-8tmdv   1/1       Running   0          22s

2. On node the pod located to check the cpu.cfs_quota_us in pod level cgroup.
[root@ip-172-18-5-70 test]# cat /sys/fs/cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec90ce2_6031_11e8_89e2_0eb7f09c6860.slice/cpu.cfs_quota_us
100000

//pod.yaml
apiVersion: v1
kind: Pod
metadata:
  generateName: burstable-
spec:
  containers:
  - image: busybox
    name: busybox
    command:
    - sleep
    - "3600"
    resources:
      requests:
        memory: 1500Mi
        cpu: 300m
      limits:
        cpu: 1
        memory: 3Gi
  restartPolicy: Always
  terminationGracePeriodSeconds: 0
Comment 12 DeShuai Ma 2018-06-06 08:03:13 UTC 
Verified on openshift v3.10.0-0.60.0 Step can ref comment 9
Comment 15 errata-xmlrpc 2018-07-30 19:16:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816