Bug 1554471

I will try to reproduce it locally and get back to you.

I can not reproduce in origin's 3.9 branch. I will check 3.7 now.

I am still investigating it and will get back you once I have something to share.

Here is my investigation so far:

I tested it on 3.9 and latest upstream kubernetes master branch and I am seeing same behaviour in both. And it seems it also matches the behaviour in 3.7. 

1. cpu-cfs-quota to true
1a) if pod specifies only request, there was no throttling (nr_periods 0, nr_throttled 0, throttled_time 0) and pod was able to use as much as cpu as it wanted. (the pod had 5cpu, and i ran dd if=/dev/zero of=/dev/null, it was able to go upto 20% (around 1cpu), which is expected with dd). Also, the pod set its CpuQuota to 0, CpuPeriod to 0. 

1b) if pod specifies limit, there was no throttling if it stayed below its limits. There was throttling (nr_periods, nr_throttled, throttled_time) kept increasing when pod trried to go beyond its limit and remained always within its limit. (the pod had 5cpu, and its limit was 500m, and i ran dd if=/dev/zero of=/dev/null, it was always around 10%). Also, the pod set its CpuQuota to 50000, CpuPeriod to 100000.  


2. cpu-cfs-quota to false:
2a) if pod specifies only request, there was no throttling (nr_periods 0, nr_throttled 0, throttled_time 0) and pod was able to use as much as cpu as it wanted. (the pod had 5cpu, and i ran dd if=/dev/zero of=/dev/null, it was able to go upto 20% (around 1cpu), which is expected with dd). Also, the pod set its CpuQuota to 0, CpuPeriod to 0. 

2b) if pod specifies limit, there was no throttling (nr_periods 0, nr_throttled 0, throttled_time 0) but remained always within its limit. (the pod had 5cpu, and its limit was 500m, and i ran dd if=/dev/zero of=/dev/null, it was always around 10%). Also, the pod set its CpuQuota to 0, CpuPeriod to 0.

As per above, all cases are working as expected except 2b which needs to be checked if its working as expected or not. In 2b, the main discrepancy is that cpu-cfs-quota was set to false, and the pod set its CpuQuota to 0, CpuPeriod to 0 as expected, then why it remained at 10% and did not go upto 20%, even though there was no throttling?

Just to capture some offline conversation:

Tested is done with docker 1.12.6. And i was checking values here https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go#L65-L71 and they seem to be set correctly as per my last comment.

I checked that values shown in docker inspect are same as what is being passed by kubelet to docker. I will keep investigating but so far I am not seeing if anything is wrong with kubelet.


I found another bz which seems to deal in the same area: https://bugzilla.redhat.com/show_bug.cgi?id=1455071 just putting here for reference as I go through the bz.

In 2b case (as described in the comment https://bugzilla.redhat.com/show_bug.cgi?id=1554471#c7), I see following:

# kubectl --kubeconfig /var/run/kubernetes/admin.kubeconfig exec nginx5-d2czw -- cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
# kubectl --kubeconfig /var/run/kubernetes/admin.kubeconfig exec nginx5-d2czw -- cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
-1
# kubectl --kubeconfig /var/run/kubernetes/admin.kubeconfig exec nginx5-d2czw -- cat /sys/fs/cgroup/cpu/cpu.shares
10


docker version: docker-1.12.6-71.git3e8e77d.el7.x86_64
kernel version: 3.10.0-709.el7.x86_64

I think I have isolated the root issue.

The container cgroup for cfs quota is unbound, but the pod level cgroup is bounded.  I will need some time to see how much needs to change upstream to keep this unbounded when pod level cgroups are enabled.

Upstream PR posted:
https://github.com/kubernetes/kubernetes/pull/61294

(In reply to Derek Carr from comment #13)
> Upstream PR posted:
> https://github.com/kubernetes/kubernetes/pull/61294

Thanks a lot Derek. I did some testing and it is working as expected.

Upstream PR merged.

Origin PR for 3.10 here:
https://github.com/openshift/origin/pull/19028

Will do pro-active picks with cloned bzs.

This shipped with https://access.redhat.com/errata/product/290/ver=3.7/rhel---7/x86_64/RHBA-2018:0636

Verify on ocp v3.7.43

Steps to verify:
Set cpu-cfs-quota=true in /etc/origin/node/node-config.yaml
kubeletArguments:
  cpu-cfs-quota:
  - 'true'
//Case 1 (cpu-cfs-quota=true + without limits)
1. Create a pod without limits
2. rsh into pod run 'dd if=/dev/zero of=/dev/null'
3. In another terminal rsh into pod and run 'while true; do sleep 4; ps aux|grep dd ; done' The cpu usage is about 99% for one core
sh-4.2# while true; do sleep 4; ps aux|grep dd ; done
root         19 98.0  0.0   4348   344 ?        R+   08:59   0:24 dd if=/dev/zero of=/dev/null
4. The the host which the pod running, using `htop` to watch the cpu usage. one of the four cpu core usage is about 99%

//Case 2 (cpu-cfs-quota=true + with limits)
1. Create a pod with limits.cpu=500m
2. rsh into pod run 'dd if=/dev/zero of=/dev/null'
3. In another terminal rsh into pod and run 'while true; do sleep 4; ps aux|grep dd ; done' The cpu usage is about 50% for one core
4. The the host which the pod running, using `htop` to watch the cpu usage. one of the four cpu core usage is about 50%


//Case 3 (cpu-cfs-quota=false + without limits)
1. Create a pod without limits.cpu=500m
2. rsh into pod run 'dd if=/dev/zero of=/dev/null'
3. In another terminal rsh into pod and run 'while true; do sleep 4; ps aux|grep dd ; done' The cpu usage is about 99% for one core
4. The the host which the pod running, using `htop` to watch the cpu usage. one of the four cpu core usage is about 99%

//Case 4 (cpu-cfs-quota=false + with limits)
1. Create a pod without limits
2. rsh into pod run 'dd if=/dev/zero of=/dev/null'
3. In another terminal rsh into pod and run 'while true; do sleep 4; ps aux|grep dd ; done' The cpu usage is about 99% for one core.
4. The the host which the pod running, using `htop` to watch the cpu usage. one of the four cpu core usage is about 99%.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2009