Bug 1459826

Summary: init-containers with resource requests/limits got error
Product: OpenShift Container Platform Reporter: Weihua Meng <wmeng>
Component: NodeAssignee: Seth Jennings <sjenning>
Status: CLOSED ERRATA QA Contact: DeShuai Ma <dma>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, decarr, jokerman, mmccomas
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 21:56:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Weihua Meng 2017-06-08 09:56:12 UTC
Description of problem:
init-containers with resource requests/limits got error
Message
Failed to start container with id 4b43d72e4376312cad410647323cb9513cf3440972e96c0ff58ca5886be9ff1a with error: rpc error: code = 2 desc = failed to start container "4b43d72e4376312cad410647323cb9513cf3440972e96c0ff58ca5886be9ff1a": Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:291: setting cgroup config for ready process caused \\\\\\\"failed to write 50000 to cpu.cfs_quota_us: write /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb5bca1_4c2b_11e7_9837_42010af00003.slice/docker-4b43d72e4376312cad410647323cb9513cf3440972e96c0ff58ca5886be9ff1a.scope/cpu.cfs_quota_us: invalid argument\\\\\\\"\\\"\\n\""}


Version-Release number of selected component (if applicable):
openshift v3.6.99
kubernetes v1.6.1+5115d708d7
etcd 3.1.0

How reproducible:
Always

Steps to Reproduce:
1. create a pod of init-containers with resource requests/limits
$ oc create -f init-containers.yaml
$ cat init-containers.yaml
apiVersion: v1
kind: Pod
metadata:
  name: init-limits
spec:
  containers:
  - image: docker.io/ocpqe/hello-pod
    name: hello-pod
    resources: 
      limits: 
        cpu: "150m"
        memory: "120Mi"
      requests: 
        cpu: "100m"
        memory: "100Mi"
  initContainers:
  - name: init
    image: centos:centos7
    command:
    - /bin/true
    resources:
      limits:
        cpu: "500m"
        memory: "400Mi"
      requests:
        cpu: "300m"
        memory: "200Mi"
2. exam pod status
$ oc get pod

Actual results:
Errors
$ oc get pod
NAME            READY     STATUS                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      RESTARTS   AGE
init-quota123   0/1       Init:rpc error: code = 2 desc = failed to start container "4b43d72e4376312cad410647323cb9513cf3440972e96c0ff58ca5886be9ff1a": Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:291: setting cgroup config for ready process caused \\\\\\\"failed to write 50000 to cpu.cfs_quota_us: write /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb5bca1_4c2b_11e7_9837_42010af00003.slice/docker-4b43d72e4376312cad410647323cb9513cf3440972e96c0ff58ca5886be9ff1a.scope/cpu.cfs_quota_us: invalid argument\\\\\\\"\\\"\\n\""}   0          4s
[root@dhcp-140-97 initcontainers]# oc get pod
NAME            READY     STATUS                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      RESTARTS   AGE
init-quota123   0/1       Init:rpc error: code = 2 desc = failed to start container "8d2b645c2f694639ad6744af38c3c3416ba2ce05a2973a6554b0456ffad5ec1a": Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:291: setting cgroup config for ready process caused \\\\\\\"failed to write 50000 to cpu.cfs_quota_us: write /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb5bca1_4c2b_11e7_9837_42010af00003.slice/docker-8d2b645c2f694639ad6744af38c3c3416ba2ce05a2973a6554b0456ffad5ec1a.scope/cpu.cfs_quota_us: invalid argument\\\\\\\"\\\"\\n\""}   1          6s
[root@dhcp-140-97 initcontainers]# oc get pod
NAME            READY     STATUS                  RESTARTS   AGE
init-quota123   0/1       Init:CrashLoopBackOff   1          7s


Expected results:
pod running, no error

Additional info:
$ oc describe pod
will show error message, too

Comment 3 Seth Jennings 2017-06-12 21:09:20 UTC
Upstream already has a fix:
https://github.com/kubernetes/kubernetes/commit/22ab45b5759b01341a2b030915da68842e9bd135

Comment 4 Seth Jennings 2017-06-12 21:23:49 UTC
Origin PR:
https://github.com/openshift/origin/pull/14605

Comment 6 Weihua Meng 2017-06-16 09:34:42 UTC
verified on openshift v3.6.112
Fixed.

# oc create -f init-containers.yaml
pod "init-limits" created
# oc get pod
NAME          READY     STATUS     RESTARTS   AGE
init-limits   0/1       Init:0/1   0          4s
# oc get pod
NAME          READY     STATUS     RESTARTS   AGE
init-limits   0/1       Init:0/1   0          7s
# oc get pod
NAME          READY     STATUS     RESTARTS   AGE
init-limits   0/1       Init:0/1   0          9s
[...]
# oc get pod
NAME          READY     STATUS    RESTARTS   AGE
init-limits   1/1       Running   0          2m

Comment 9 errata-xmlrpc 2017-11-28 21:56:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188