Bug 1714183

Summary: Sporadic CreateContainerError, container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"\"
Product: OpenShift Container Platform Reporter: Marek Schmidt <maschmid>
Component: ContainersAssignee: Mrunal Patel <mpatel>
Status: CLOSED NOTABUG QA Contact: weiwei jiang <wjiang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs, dwalsh, jokerman, mmccomas
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-21 21:14:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marek Schmidt 2019-05-27 10:36:00 UTC 
Description of problem:

Attempting to install knative on OCP 4.1.0.rc7 (on AWS) via

https://github.com/openshift-cloud-functions/knative-operators/blob/openshift-v0.5.0/etc/scripts/install.sh

On some runs, some of the knative-eventing controllers fail to start with CreateContainerError (see details below)

(they eventually start and seem to run fine after some time)


Version-Release number of selected component (if applicable):
4.1.0-rc.7

How reproducible:
Sometimes (also seen on 4.1.0-rc.5)

Steps to Reproduce:
1. install an istio workaround for https://issues.jboss.org/browse/MAISTRA-410

oc adm policy add-scc-to-user privileged -z default -n default
oc create -n default -f https://issues.jboss.org/secure/attachment/12450475/modprobe_daemonset.yaml

)

2. git clone git:openshift-cloud-functions/knative-operators.git
3. cd knative-operators
4. git checkout openshift-v0.5.0
5. ./etc/scripts/install.sh

(note, camel-controller-manager-0 is expected to fail at startup if no camelk is installed, but it should not fail with CreateContainerError )

Actual results:

sometimes the knative eventing kafka controller startup fails with CreateContainerError

Expected results:

knative eventing kafka controller should start successfully

Additional info:

Images that sometimes fail to start as containers are

quay.io/openshift-knative/knative-eventing-sources-kafka-source-controller:v0.5.0

and

quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0


oc describe pod kafka-controller-manager-0
Name:               kafka-controller-manager-0
Namespace:          knative-eventing
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-0-138-252.us-west-2.compute.internal/10.0.138.252
Start Time:         Mon, 27 May 2019 04:42:08 -0400
Labels:             control-plane=kafka-controller-manager
                    controller-revision-hash=kafka-controller-manager-6d97c6477f
                    statefulset.kubernetes.io/pod-name=kafka-controller-manager-0
Annotations:        k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "openshift-sdn",
                          "interface": "eth0",
                          "ips": [
                              "10.128.2.30"
                          ],
                          "default": true,
                          "dns": {}
                      }]
                    openshift.io/scc: restricted
Status:             Pending
IP:                 10.128.2.30
Controlled By:      StatefulSet/kafka-controller-manager
Containers:
  manager:
    Container ID:   
    Image:          quay.io/openshift-knative/knative-eventing-sources-kafka-source-controller:v0.5.0
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:     20m
      memory:  20Mi
    Environment:
      KAFKA_RA_IMAGE:  quay.io/openshift-knative/knative-eventing-sources-kafka-source-adapter:v0.5.0
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kafka-controller-manager-token-l8lz6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kafka-controller-manager-token-l8lz6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kafka-controller-manager-token-l8lz6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From                                                 Message
  ----     ------     ----  ----                                                 -------
  Normal   Scheduled  2m7s  default-scheduler                                    Successfully assigned knative-eventing/kafka-controller-manager-0 to ip-10-0-138-252.us-west-2.compute.internal
  Warning  Failed     111s  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Error: container create failed: panic: No error following JSON procError payload. [recovered]
           panic: No error following JSON procError payload.

goroutine 1 [running]:
panic(0x562e1b42af60, 0x562e1b4a94d0)
  /usr/lib/golang/src/runtime/panic.go:556 +0x2cf fp=0xc00015e780 sp=0xc00015e6f0 pc=0x562e1ae61c9f
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction.func1(0xc00015f890)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:478 +0x23c fp=0xc00015e818 sp=0xc00015e780 pc=0x562e1b0f0a8c
runtime.call32(0x0, 0x562e1b4a8900, 0xc0000780f0, 0x800000008)
  /usr/lib/golang/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00015e848 sp=0xc00015e818 pc=0x562e1ae8da9d
panic(0x562e1b42af60, 0x562e1b4a94d0)
  /usr/lib/golang/src/runtime/panic.go:513 +0x1bd fp=0xc00015e8d8 sp=0xc00015e848 pc=0x562e1ae61b8d
github.com/opencontainers/runc/libcontainer.parseSync(0x562e1b4ab8e0, 0xc0000820e0, 0xc00015e9d0, 0x0, 0xc000087d80)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/sync.go:99 +0x2ee fp=0xc00015e950 sp=0xc00015e8d8 pc=0x562e1b0d86de
github.com/opencontainers/runc/libcontainer.(*initProcess).start(0xc000087d80, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/process_linux.go:320 +0x3c4 fp=0xc00015ea00 sp=0xc00015e950 pc=0x562e1b0cc784
github.com/opencontainers/runc/libcontainer.(*linuxContainer).start(0xc00024a000, 0xc0002165a0, 0x0, 0x562e1ae60d84)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/container_linux.go:331 +0x86 fp=0xc00015eb70 sp=0xc00015ea00 pc=0x562e1b0b49f6
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start(0xc00024a000, 0xc0002165a0, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/container_linux.go:232 +0xa0 fp=0xc00015ebc0 sp=0xc00015eb70 pc=0x562e1b0b4170
main.(*runner).run(0xc00015f3a8, 0xc0000fac30, 0x0, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/utils_linux.go:331 +0x93c fp=0xc00015f2f8 sp=0xc00015ebc0 pc=0x562e1b1199fc
main.startContainer(0xc000148140, 0xc00012c150, 0x1, 0x0, 0x0, 0x16, 0xc0000f0800)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/utils_linux.go:457 +0x3cb fp=0xc00015f428 sp=0xc00015f2f8 pc=0x562e1b11a71b
main.glob..func2(0xc000148140, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/create.go:65 +0xb0 fp=0xc00015f470 sp=0xc00015f428 pc=0x562e1b11ad40
runtime.call32(0xc0000ec6c0, 0x562e1b4a8a20, 0xc0000f0840, 0x800000018)
  /usr/lib/golang/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00015f4a0 sp=0xc00015f470 pc=0x562e1ae8da9d
reflect.Value.call(0x562e1b437340, 0x562e1b4a8a20, 0x13, 0x562e1b12781a, 0x4, 0xc0000bf850, 0x1, 0x1, 0xc0000780c0, 0xc0001024e0, ...)
  /usr/lib/golang/src/reflect/value.go:447 +0x456 fp=0xc00015f768 sp=0xc00015f4a0 pc=0x562e1aeec486
reflect.Value.Call(0x562e1b437340, 0x562e1b4a8a20, 0x13, 0xc0000bf850, 0x1, 0x1, 0xc0000bf830, 0x562e1b0ecbe8, 0x562e1b12795a)
  /usr/lib/golang/src/reflect/value.go:308 +0xa6 fp=0xc00015f7d0 sp=0xc00015f768 pc=0x562e1aeebef6
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction(0x562e1b437340, 0x562e1b4a8a20, 0xc000148140, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:487 +0x1fa fp=0xc00015f878 sp=0xc00015f7d0 pc=0x562e1b0e4c5a
github.com/opencontainers/runc/vendor/github.com/urfave/cli.Command.Run(0x562e1b12804a, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x562e1b12c25c, 0x12, 0x0, ...)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:191 +0x91a fp=0xc00015fb10 sp=0xc00015f878 pc=0x562e1b0e5f2a
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc000118180, 0xc000086000, 0x8, 0x8, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:240 +0x530 fp=0xc00015fcc0 sp=0xc00015fb10 pc=0x562e1b0e2e80
main.main()
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/main.go:165 +0xae7 fp=0xc00015ff98 sp=0xc00015fcc0 pc=0x562e1b112467
runtime.main()
  /usr/lib/golang/src/runtime/proc.go:201 +0x20f fp=0xc00015ffe0 sp=0xc00015ff98 pc=0x562e1ae63a0f
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00015ffe8 sp=0xc00015ffe0 pc=0x562e1ae8f801

goroutine 2 [force gc (idle)]:
runtime.gopark(0x562e1b4a8f98, 0x562e1b6fc970, 0x1410, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002af80 sp=0xc00002af60 pc=0x562e1ae63e01
runtime.goparkunlock(0x562e1b6fc970, 0x1410, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002afb0 sp=0xc00002af80 pc=0x562e1ae63eb5
runtime.forcegchelper()
  /usr/lib/golang/src/runtime/proc.go:251 +0xb7 fp=0xc00002afe0 sp=0xc00002afb0 pc=0x562e1ae63c77
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002afe8 sp=0xc00002afe0 pc=0x562e1ae8f801
created by runtime.init.5
  /usr/lib/golang/src/runtime/proc.go:240 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x562e1b4a8f98, 0x562e1b6fcb00, 0x562e1ae5140c, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002b780 sp=0xc00002b760 pc=0x562e1ae63e01
runtime.goparkunlock(0x562e1b6fcb00, 0x562e1b17140c, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002b7b0 sp=0xc00002b780 pc=0x562e1ae63eb5
runtime.bgsweep(0xc000052000)
  /usr/lib/golang/src/runtime/mgcsweep.go:52 +0x93 fp=0xc00002b7d8 sp=0xc00002b7b0 pc=0x562e1ae56c93
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002b7e0 sp=0xc00002b7d8 pc=0x562e1ae8f801
created by runtime.gcenable
  /usr/lib/golang/src/runtime/mgc.go:216 +0x5a

goroutine 18 [finalizer wait]:
runtime.gopark(0x562e1b4a8f98, 0x562e1b719ff0, 0x140f, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc000026728 sp=0xc000026708 pc=0x562e1ae63e01
runtime.goparkunlock(0x562e1b719ff0, 0x140f, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc000026758 sp=0xc000026728 pc=0x562e1ae63eb5
runtime.runfinq()
  /usr/lib/golang/src/runtime/mfinal.go:175 +0x9d fp=0xc0000267e0 sp=0xc000026758 pc=0x562e1ae4e55d
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000267e8 sp=0xc0000267e0 pc=0x562e1ae8f801
created by runtime.createfing
  /usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 35 [IO wait]:
runtime.gopark(0x562e1b4a8f68, 0x7f97c6fcdec8, 0x1b02, 0x5)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc0000c1760 sp=0xc0000c1740 pc=0x562e1ae63e01
runtime.netpollblock(0x7f97c6fcdea0, 0x72, 0x0)
  /usr/lib/golang/src/runtime/netpoll.go:366 +0x9b fp=0xc0000c1798 sp=0xc0000c1760 pc=0x562e1ae5f06b
internal/poll.runtime_pollWait(0x7f97c6fcdea0, 0x72, 0xc000268270)
  /usr/lib/golang/src/runtime/netpoll.go:173 +0x68 fp=0xc0000c17c8 sp=0xc0000c1798 pc=0x562e1ae5e708
internal/poll.(*pollDesc).wait(0xc00016c318, 0x72, 0xc0001eb900, 0x1000, 0x0)
  /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:85 +0x9c fp=0xc0000c17f8 sp=0xc0000c17c8 pc=0x562e1aec93fc
internal/poll.(*pollDesc).waitRead(0xc00016c318, 0xc000268200, 0x10, 0x10)
           /usr/lib/golang/src/interna
  Warning  Failed  86s  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"\""
  Warning  Failed  40s  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Error: container create failed: panic: No error following JSON procError payload. [recovered]
           panic: No error following JSON procError payload.

goroutine 1 [running]:
panic(0x5581fa73ef60, 0x5581fa7bd4d0)
  /usr/lib/golang/src/runtime/panic.go:556 +0x2cf fp=0xc00014c780 sp=0xc00014c6f0 pc=0x5581fa175c9f
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction.func1(0xc00014d890)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:478 +0x23c fp=0xc00014c818 sp=0xc00014c780 pc=0x5581fa404a8c
runtime.call32(0x0, 0x5581fa7bc900, 0xc000012170, 0x800000008)
  /usr/lib/golang/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00014c848 sp=0xc00014c818 pc=0x5581fa1a1a9d
panic(0x5581fa73ef60, 0x5581fa7bd4d0)
  /usr/lib/golang/src/runtime/panic.go:513 +0x1bd fp=0xc00014c8d8 sp=0xc00014c848 pc=0x5581fa175b8d
github.com/opencontainers/runc/libcontainer.parseSync(0x5581fa7bf8e0, 0xc00000e0a8, 0xc00014c9d0, 0x0, 0xc000010400)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/sync.go:99 +0x2ee fp=0xc00014c950 sp=0xc00014c8d8 pc=0x5581fa3ec6de
github.com/opencontainers/runc/libcontainer.(*initProcess).start(0xc000010400, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/process_linux.go:320 +0x3c4 fp=0xc00014ca00 sp=0xc00014c950 pc=0x5581fa3e0784
github.com/opencontainers/runc/libcontainer.(*linuxContainer).start(0xc0001d21e0, 0xc0001ab680, 0x0, 0x5581fa174d84)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/container_linux.go:331 +0x86 fp=0xc00014cb70 sp=0xc00014ca00 pc=0x5581fa3c89f6
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start(0xc0001d21e0, 0xc0001ab680, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/container_linux.go:232 +0xa0 fp=0xc00014cbc0 sp=0xc00014cb70 pc=0x5581fa3c8170
main.(*runner).run(0xc00014d3a8, 0xc0000f6c30, 0x0, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/utils_linux.go:331 +0x93c fp=0xc00014d2f8 sp=0xc00014cbc0 pc=0x5581fa42d9fc
main.startContainer(0xc000136140, 0xc00011a150, 0x1, 0x0, 0x0, 0x16, 0xc0000ec7a0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/utils_linux.go:457 +0x3cb fp=0xc00014d428 sp=0xc00014d2f8 pc=0x5581fa42e71b
main.glob..func2(0xc000136140, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/create.go:65 +0xb0 fp=0xc00014d470 sp=0xc00014d428 pc=0x5581fa42ed40
runtime.call32(0xc0000da600, 0x5581fa7bca20, 0xc0000ec7e0, 0x800000018)
  /usr/lib/golang/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00014d4a0 sp=0xc00014d470 pc=0x5581fa1a1a9d
reflect.Value.call(0x5581fa74b340, 0x5581fa7bca20, 0x13, 0x5581fa43b81a, 0x4, 0xc0000b1850, 0x1, 0x1, 0xc000012140, 0xc0000fe4e0, ...)
  /usr/lib/golang/src/reflect/value.go:447 +0x456 fp=0xc00014d768 sp=0xc00014d4a0 pc=0x5581fa200486
reflect.Value.Call(0x5581fa74b340, 0x5581fa7bca20, 0x13, 0xc0000b1850, 0x1, 0x1, 0xc0000b1830, 0x5581fa400be8, 0x5581fa43b95a)
  /usr/lib/golang/src/reflect/value.go:308 +0xa6 fp=0xc00014d7d0 sp=0xc00014d768 pc=0x5581fa1ffef6
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction(0x5581fa74b340, 0x5581fa7bca20, 0xc000136140, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:487 +0x1fa fp=0xc00014d878 sp=0xc00014d7d0 pc=0x5581fa3f8c5a
github.com/opencontainers/runc/vendor/github.com/urfave/cli.Command.Run(0x5581fa43c04a, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5581fa44025c, 0x12, 0x0, ...)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:191 +0x91a fp=0xc00014db10 sp=0xc00014d878 pc=0x5581fa3f9f2a
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc000060480, 0xc000010080, 0x8, 0x8, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:240 +0x530 fp=0xc00014dcc0 sp=0xc00014db10 pc=0x5581fa3f6e80
main.main()
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/main.go:165 +0xae7 fp=0xc00014df98 sp=0xc00014dcc0 pc=0x5581fa426467
runtime.main()
  /usr/lib/golang/src/runtime/proc.go:201 +0x20f fp=0xc00014dfe0 sp=0xc00014df98 pc=0x5581fa177a0f
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00014dfe8 sp=0xc00014dfe0 pc=0x5581fa1a3801

goroutine 2 [force gc (idle)]:
runtime.gopark(0x5581fa7bcf98, 0x5581faa10970, 0x1410, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002af80 sp=0xc00002af60 pc=0x5581fa177e01
runtime.goparkunlock(0x5581faa10970, 0x1410, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002afb0 sp=0xc00002af80 pc=0x5581fa177eb5
runtime.forcegchelper()
  /usr/lib/golang/src/runtime/proc.go:251 +0xb7 fp=0xc00002afe0 sp=0xc00002afb0 pc=0x5581fa177c77
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002afe8 sp=0xc00002afe0 pc=0x5581fa1a3801
created by runtime.init.5
  /usr/lib/golang/src/runtime/proc.go:240 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x5581fa7bcf98, 0x5581faa10b00, 0x5581fa16140c, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002b780 sp=0xc00002b760 pc=0x5581fa177e01
runtime.goparkunlock(0x5581faa10b00, 0x5581fa49140c, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002b7b0 sp=0xc00002b780 pc=0x5581fa177eb5
runtime.bgsweep(0xc000052000)
  /usr/lib/golang/src/runtime/mgcsweep.go:52 +0x93 fp=0xc00002b7d8 sp=0xc00002b7b0 pc=0x5581fa16ac93
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002b7e0 sp=0xc00002b7d8 pc=0x5581fa1a3801
created by runtime.gcenable
  /usr/lib/golang/src/runtime/mgc.go:216 +0x5a

goroutine 4 [finalizer wait]:
runtime.gopark(0x5581fa7bcf98, 0x5581faa2dff0, 0x140f, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002bf28 sp=0xc00002bf08 pc=0x5581fa177e01
runtime.goparkunlock(0x5581faa2dff0, 0x140f, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002bf58 sp=0xc00002bf28 pc=0x5581fa177eb5
runtime.runfinq()
  /usr/lib/golang/src/runtime/mfinal.go:175 +0x9d fp=0xc00002bfe0 sp=0xc00002bf58 pc=0x5581fa16255d
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002bfe8 sp=0xc00002bfe0 pc=0x5581fa1a3801
created by runtime.createfing
  /usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 6 [timer goroutine (idle)]:
runtime.gopark(0x5581fa7bcf98, 0x5581faa14720, 0x536c1414, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002a728 sp=0xc00002a708 pc=0x5581fa177e01
runtime.goparkunlock(0x5581faa14720, 0xc000051414, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002a758 sp=0xc00002a728 pc=0x5581fa177eb5
runtime.timerproc(0x5581faa14720)
  /usr/lib/golang/src/runtime/time.go:280 +0x28c fp=0xc00002a7d8 sp=0xc00002a758 pc=0x5581fa19480c
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002a7e0 sp=0xc00002a7d8 pc=0x5581fa1a3801
created by runtime.(*timersBucket).addtimerLocked
  /usr/lib/golang/src/runtime/time.go:170 +0x116

goroutine 18 [syscall]:
runtime.notetsleepg(0x5581faa2e340, 0xffffffffffffffff, 0xffffffff
  Normal   Pulling  8s (x7 over 118s)  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Pulling image "quay.io/openshift-knative/knative-eventing-sources-kafka-source-controller:v0.5.0"
  Normal   Pulled   7s (x7 over 117s)  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Successfully pulled image "quay.io/openshift-knative/knative-eventing-sources-kafka-source-controller:v0.5.0"
  Warning  Failed   2s (x4 over 104s)  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"read init-p: connection reset by peer\""






oc describe pod camel-controller-manager-0
Name:               camel-controller-manager-0
Namespace:          knative-eventing
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-0-148-82.us-west-2.compute.internal/10.0.148.82
Start Time:         Mon, 27 May 2019 04:19:47 -0400
Labels:             control-plane=camel-controller-manager
                    controller-revision-hash=camel-controller-manager-c58b9488c
                    statefulset.kubernetes.io/pod-name=camel-controller-manager-0
Annotations:        k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "openshift-sdn",
                          "interface": "eth0",
                          "ips": [
                              "10.131.0.20"
                          ],
                          "default": true,
                          "dns": {}
                      }]
                    openshift.io/scc: restricted
Status:             Running
IP:                 10.131.0.20
Controlled By:      StatefulSet/camel-controller-manager
Containers:
  manager:
    Container ID:   cri-o://028f9202fff538e72f79103698b483b5f06b5eb7ecd1e982ceb6fd9090f438f8
    Image:          quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0
    Image ID:       quay.io/openshift-knative/knative-eventing-sources-camel-source-controller@sha256:ef22ffa2ef0ff6450e32699b43b473d8c43014c850eee9c8d54671fca8fc7521
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 27 May 2019 04:29:20 -0400
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:        20m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from camel-controller-manager-token-zn6qj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  camel-controller-manager-token-zn6qj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  camel-controller-manager-token-zn6qj
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                                                Message
  ----     ------     ----                  ----                                                -------
  Normal   Scheduled  129m                  default-scheduler                                   Successfully assigned knative-eventing/camel-controller-manager-0 to ip-10-0-148-82.us-west-2.compute.internal
  Normal   Pulling    128m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Pulling image "quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0"
  Normal   Pulled     128m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Successfully pulled image "quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0"
  Warning  Failed     128m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:313: sending config to init process caused \"write init-p: no buffer space available\""
  Warning  Failed     127m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"failed to write 0-3\\n to cpuset.cpus: open /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3093f6ab_8058_11e9_8d8e_06ad16ce2654.slice/crio-521d0a85969acead6d0a5d8f35a1feecfac98467c768bdfe32eccc2655e1f08f.scope/cpuset.cpus: cannot allocate memory\""
  Warning  Failed     126m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:313: sending config to init process caused \"write init-p: broken pipe\""
  Warning  Failed     125m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"no space left on device\\\"\""
  Normal   Pulled     125m (x11 over 128m)  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Container image "quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0" already present on machine
  Warning  Failed     125m (x2 over 127m)   kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for devices not found\""
  Warning  Failed     123m (x10 over 128m)  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"read init-p: connection reset by peer\""
Comment 1 Marek Schmidt 2019-05-27 10:43:28 UTC 
Also note, according to grafana, memory and disk are well below 30% on all nodes (as well as CPU besides a few spikes up to 50%), so this doesn't seem to be caused by actual resource pressures.
Comment 4 Marek Schmidt 2019-05-29 06:55:04 UTC 
Removing just the CPU limit doesn't seem to help ( 'Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"read init-p: connection reset by peer\"' still occured)

Bumping up CPU and memory requests make the pods came up reliably.