1714183 – Sporadic CreateContainerError, container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"\"

Bug 1714183 - Sporadic CreateContainerError, container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"\"

Summary: Sporadic CreateContainerError, container_linux.go:336: starting container pro...

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Containers
Sub Component:
Version:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	4.2.0
Assignee:	Mrunal Patel
QA Contact:	weiwei jiang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-05-27 10:36 UTC by Marek Schmidt
Modified:	2019-08-21 21:14 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-08-21 21:14:20 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Marek Schmidt 2019-05-27 10:36:00 UTC

Description of problem:

Attempting to install knative on OCP 4.1.0.rc7 (on AWS) via

https://github.com/openshift-cloud-functions/knative-operators/blob/openshift-v0.5.0/etc/scripts/install.sh

On some runs, some of the knative-eventing controllers fail to start with CreateContainerError (see details below)

(they eventually start and seem to run fine after some time)


Version-Release number of selected component (if applicable):
4.1.0-rc.7

How reproducible:
Sometimes (also seen on 4.1.0-rc.5)

Steps to Reproduce:
1. install an istio workaround for https://issues.jboss.org/browse/MAISTRA-410

oc adm policy add-scc-to-user privileged -z default -n default
oc create -n default -f https://issues.jboss.org/secure/attachment/12450475/modprobe_daemonset.yaml

)

2. git clone git:openshift-cloud-functions/knative-operators.git
3. cd knative-operators
4. git checkout openshift-v0.5.0
5. ./etc/scripts/install.sh

(note, camel-controller-manager-0 is expected to fail at startup if no camelk is installed, but it should not fail with CreateContainerError )

Actual results:

sometimes the knative eventing kafka controller startup fails with CreateContainerError

Expected results:

knative eventing kafka controller should start successfully

Additional info:

Images that sometimes fail to start as containers are

quay.io/openshift-knative/knative-eventing-sources-kafka-source-controller:v0.5.0

and

quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0


oc describe pod kafka-controller-manager-0
Name:               kafka-controller-manager-0
Namespace:          knative-eventing
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-0-138-252.us-west-2.compute.internal/10.0.138.252
Start Time:         Mon, 27 May 2019 04:42:08 -0400
Labels:             control-plane=kafka-controller-manager
                    controller-revision-hash=kafka-controller-manager-6d97c6477f
                    statefulset.kubernetes.io/pod-name=kafka-controller-manager-0
Annotations:        k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "openshift-sdn",
                          "interface": "eth0",
                          "ips": [
                              "10.128.2.30"
                          ],
                          "default": true,
                          "dns": {}
                      }]
                    openshift.io/scc: restricted
Status:             Pending
IP:                 10.128.2.30
Controlled By:      StatefulSet/kafka-controller-manager
Containers:
  manager:
    Container ID:   
    Image:          quay.io/openshift-knative/knative-eventing-sources-kafka-source-controller:v0.5.0
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:     20m
      memory:  20Mi
    Environment:
      KAFKA_RA_IMAGE:  quay.io/openshift-knative/knative-eventing-sources-kafka-source-adapter:v0.5.0
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kafka-controller-manager-token-l8lz6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kafka-controller-manager-token-l8lz6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kafka-controller-manager-token-l8lz6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From                                                 Message
  ----     ------     ----  ----                                                 -------
  Normal   Scheduled  2m7s  default-scheduler                                    Successfully assigned knative-eventing/kafka-controller-manager-0 to ip-10-0-138-252.us-west-2.compute.internal
  Warning  Failed     111s  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Error: container create failed: panic: No error following JSON procError payload. [recovered]
           panic: No error following JSON procError payload.

goroutine 1 [running]:
panic(0x562e1b42af60, 0x562e1b4a94d0)
  /usr/lib/golang/src/runtime/panic.go:556 +0x2cf fp=0xc00015e780 sp=0xc00015e6f0 pc=0x562e1ae61c9f
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction.func1(0xc00015f890)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:478 +0x23c fp=0xc00015e818 sp=0xc00015e780 pc=0x562e1b0f0a8c
runtime.call32(0x0, 0x562e1b4a8900, 0xc0000780f0, 0x800000008)
  /usr/lib/golang/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00015e848 sp=0xc00015e818 pc=0x562e1ae8da9d
panic(0x562e1b42af60, 0x562e1b4a94d0)
  /usr/lib/golang/src/runtime/panic.go:513 +0x1bd fp=0xc00015e8d8 sp=0xc00015e848 pc=0x562e1ae61b8d
github.com/opencontainers/runc/libcontainer.parseSync(0x562e1b4ab8e0, 0xc0000820e0, 0xc00015e9d0, 0x0, 0xc000087d80)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/sync.go:99 +0x2ee fp=0xc00015e950 sp=0xc00015e8d8 pc=0x562e1b0d86de
github.com/opencontainers/runc/libcontainer.(*initProcess).start(0xc000087d80, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/process_linux.go:320 +0x3c4 fp=0xc00015ea00 sp=0xc00015e950 pc=0x562e1b0cc784
github.com/opencontainers/runc/libcontainer.(*linuxContainer).start(0xc00024a000, 0xc0002165a0, 0x0, 0x562e1ae60d84)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/container_linux.go:331 +0x86 fp=0xc00015eb70 sp=0xc00015ea00 pc=0x562e1b0b49f6
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start(0xc00024a000, 0xc0002165a0, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/container_linux.go:232 +0xa0 fp=0xc00015ebc0 sp=0xc00015eb70 pc=0x562e1b0b4170
main.(*runner).run(0xc00015f3a8, 0xc0000fac30, 0x0, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/utils_linux.go:331 +0x93c fp=0xc00015f2f8 sp=0xc00015ebc0 pc=0x562e1b1199fc
main.startContainer(0xc000148140, 0xc00012c150, 0x1, 0x0, 0x0, 0x16, 0xc0000f0800)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/utils_linux.go:457 +0x3cb fp=0xc00015f428 sp=0xc00015f2f8 pc=0x562e1b11a71b
main.glob..func2(0xc000148140, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/create.go:65 +0xb0 fp=0xc00015f470 sp=0xc00015f428 pc=0x562e1b11ad40
runtime.call32(0xc0000ec6c0, 0x562e1b4a8a20, 0xc0000f0840, 0x800000018)
  /usr/lib/golang/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00015f4a0 sp=0xc00015f470 pc=0x562e1ae8da9d
reflect.Value.call(0x562e1b437340, 0x562e1b4a8a20, 0x13, 0x562e1b12781a, 0x4, 0xc0000bf850, 0x1, 0x1, 0xc0000780c0, 0xc0001024e0, ...)
  /usr/lib/golang/src/reflect/value.go:447 +0x456 fp=0xc00015f768 sp=0xc00015f4a0 pc=0x562e1aeec486
reflect.Value.Call(0x562e1b437340, 0x562e1b4a8a20, 0x13, 0xc0000bf850, 0x1, 0x1, 0xc0000bf830, 0x562e1b0ecbe8, 0x562e1b12795a)
  /usr/lib/golang/src/reflect/value.go:308 +0xa6 fp=0xc00015f7d0 sp=0xc00015f768 pc=0x562e1aeebef6
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction(0x562e1b437340, 0x562e1b4a8a20, 0xc000148140, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:487 +0x1fa fp=0xc00015f878 sp=0xc00015f7d0 pc=0x562e1b0e4c5a
github.com/opencontainers/runc/vendor/github.com/urfave/cli.Command.Run(0x562e1b12804a, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x562e1b12c25c, 0x12, 0x0, ...)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:191 +0x91a fp=0xc00015fb10 sp=0xc00015f878 pc=0x562e1b0e5f2a
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc000118180, 0xc000086000, 0x8, 0x8, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:240 +0x530 fp=0xc00015fcc0 sp=0xc00015fb10 pc=0x562e1b0e2e80
main.main()
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/main.go:165 +0xae7 fp=0xc00015ff98 sp=0xc00015fcc0 pc=0x562e1b112467
runtime.main()
  /usr/lib/golang/src/runtime/proc.go:201 +0x20f fp=0xc00015ffe0 sp=0xc00015ff98 pc=0x562e1ae63a0f
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00015ffe8 sp=0xc00015ffe0 pc=0x562e1ae8f801

goroutine 2 [force gc (idle)]:
runtime.gopark(0x562e1b4a8f98, 0x562e1b6fc970, 0x1410, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002af80 sp=0xc00002af60 pc=0x562e1ae63e01
runtime.goparkunlock(0x562e1b6fc970, 0x1410, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002afb0 sp=0xc00002af80 pc=0x562e1ae63eb5
runtime.forcegchelper()
  /usr/lib/golang/src/runtime/proc.go:251 +0xb7 fp=0xc00002afe0 sp=0xc00002afb0 pc=0x562e1ae63c77
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002afe8 sp=0xc00002afe0 pc=0x562e1ae8f801
created by runtime.init.5
  /usr/lib/golang/src/runtime/proc.go:240 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x562e1b4a8f98, 0x562e1b6fcb00, 0x562e1ae5140c, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002b780 sp=0xc00002b760 pc=0x562e1ae63e01
runtime.goparkunlock(0x562e1b6fcb00, 0x562e1b17140c, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002b7b0 sp=0xc00002b780 pc=0x562e1ae63eb5
runtime.bgsweep(0xc000052000)
  /usr/lib/golang/src/runtime/mgcsweep.go:52 +0x93 fp=0xc00002b7d8 sp=0xc00002b7b0 pc=0x562e1ae56c93
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002b7e0 sp=0xc00002b7d8 pc=0x562e1ae8f801
created by runtime.gcenable
  /usr/lib/golang/src/runtime/mgc.go:216 +0x5a

goroutine 18 [finalizer wait]:
runtime.gopark(0x562e1b4a8f98, 0x562e1b719ff0, 0x140f, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc000026728 sp=0xc000026708 pc=0x562e1ae63e01
runtime.goparkunlock(0x562e1b719ff0, 0x140f, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc000026758 sp=0xc000026728 pc=0x562e1ae63eb5
runtime.runfinq()
  /usr/lib/golang/src/runtime/mfinal.go:175 +0x9d fp=0xc0000267e0 sp=0xc000026758 pc=0x562e1ae4e55d
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000267e8 sp=0xc0000267e0 pc=0x562e1ae8f801
created by runtime.createfing
  /usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 35 [IO wait]:
runtime.gopark(0x562e1b4a8f68, 0x7f97c6fcdec8, 0x1b02, 0x5)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc0000c1760 sp=0xc0000c1740 pc=0x562e1ae63e01
runtime.netpollblock(0x7f97c6fcdea0, 0x72, 0x0)
  /usr/lib/golang/src/runtime/netpoll.go:366 +0x9b fp=0xc0000c1798 sp=0xc0000c1760 pc=0x562e1ae5f06b
internal/poll.runtime_pollWait(0x7f97c6fcdea0, 0x72, 0xc000268270)
  /usr/lib/golang/src/runtime/netpoll.go:173 +0x68 fp=0xc0000c17c8 sp=0xc0000c1798 pc=0x562e1ae5e708
internal/poll.(*pollDesc).wait(0xc00016c318, 0x72, 0xc0001eb900, 0x1000, 0x0)
  /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:85 +0x9c fp=0xc0000c17f8 sp=0xc0000c17c8 pc=0x562e1aec93fc
internal/poll.(*pollDesc).waitRead(0xc00016c318, 0xc000268200, 0x10, 0x10)
           /usr/lib/golang/src/interna
  Warning  Failed  86s  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"\""
  Warning  Failed  40s  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Error: container create failed: panic: No error following JSON procError payload. [recovered]
           panic: No error following JSON procError payload.

goroutine 1 [running]:
panic(0x5581fa73ef60, 0x5581fa7bd4d0)
  /usr/lib/golang/src/runtime/panic.go:556 +0x2cf fp=0xc00014c780 sp=0xc00014c6f0 pc=0x5581fa175c9f
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction.func1(0xc00014d890)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:478 +0x23c fp=0xc00014c818 sp=0xc00014c780 pc=0x5581fa404a8c
runtime.call32(0x0, 0x5581fa7bc900, 0xc000012170, 0x800000008)
  /usr/lib/golang/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00014c848 sp=0xc00014c818 pc=0x5581fa1a1a9d
panic(0x5581fa73ef60, 0x5581fa7bd4d0)
  /usr/lib/golang/src/runtime/panic.go:513 +0x1bd fp=0xc00014c8d8 sp=0xc00014c848 pc=0x5581fa175b8d
github.com/opencontainers/runc/libcontainer.parseSync(0x5581fa7bf8e0, 0xc00000e0a8, 0xc00014c9d0, 0x0, 0xc000010400)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/sync.go:99 +0x2ee fp=0xc00014c950 sp=0xc00014c8d8 pc=0x5581fa3ec6de
github.com/opencontainers/runc/libcontainer.(*initProcess).start(0xc000010400, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/process_linux.go:320 +0x3c4 fp=0xc00014ca00 sp=0xc00014c950 pc=0x5581fa3e0784
github.com/opencontainers/runc/libcontainer.(*linuxContainer).start(0xc0001d21e0, 0xc0001ab680, 0x0, 0x5581fa174d84)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/container_linux.go:331 +0x86 fp=0xc00014cb70 sp=0xc00014ca00 pc=0x5581fa3c89f6
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start(0xc0001d21e0, 0xc0001ab680, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/libcontainer/container_linux.go:232 +0xa0 fp=0xc00014cbc0 sp=0xc00014cb70 pc=0x5581fa3c8170
main.(*runner).run(0xc00014d3a8, 0xc0000f6c30, 0x0, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/utils_linux.go:331 +0x93c fp=0xc00014d2f8 sp=0xc00014cbc0 pc=0x5581fa42d9fc
main.startContainer(0xc000136140, 0xc00011a150, 0x1, 0x0, 0x0, 0x16, 0xc0000ec7a0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/utils_linux.go:457 +0x3cb fp=0xc00014d428 sp=0xc00014d2f8 pc=0x5581fa42e71b
main.glob..func2(0xc000136140, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/create.go:65 +0xb0 fp=0xc00014d470 sp=0xc00014d428 pc=0x5581fa42ed40
runtime.call32(0xc0000da600, 0x5581fa7bca20, 0xc0000ec7e0, 0x800000018)
  /usr/lib/golang/src/runtime/asm_amd64.s:522 +0x3d fp=0xc00014d4a0 sp=0xc00014d470 pc=0x5581fa1a1a9d
reflect.Value.call(0x5581fa74b340, 0x5581fa7bca20, 0x13, 0x5581fa43b81a, 0x4, 0xc0000b1850, 0x1, 0x1, 0xc000012140, 0xc0000fe4e0, ...)
  /usr/lib/golang/src/reflect/value.go:447 +0x456 fp=0xc00014d768 sp=0xc00014d4a0 pc=0x5581fa200486
reflect.Value.Call(0x5581fa74b340, 0x5581fa7bca20, 0x13, 0xc0000b1850, 0x1, 0x1, 0xc0000b1830, 0x5581fa400be8, 0x5581fa43b95a)
  /usr/lib/golang/src/reflect/value.go:308 +0xa6 fp=0xc00014d7d0 sp=0xc00014d768 pc=0x5581fa1ffef6
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction(0x5581fa74b340, 0x5581fa7bca20, 0xc000136140, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:487 +0x1fa fp=0xc00014d878 sp=0xc00014d7d0 pc=0x5581fa3f8c5a
github.com/opencontainers/runc/vendor/github.com/urfave/cli.Command.Run(0x5581fa43c04a, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5581fa44025c, 0x12, 0x0, ...)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:191 +0x91a fp=0xc00014db10 sp=0xc00014d878 pc=0x5581fa3f9f2a
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc000060480, 0xc000010080, 0x8, 0x8, 0x0, 0x0)
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:240 +0x530 fp=0xc00014dcc0 sp=0xc00014db10 pc=0x5581fa3f6e80
main.main()
  /builddir/build/BUILD/runc-2abd837c8c25b0102ac4ce14f17bc0bc7ddffba7/GOPATH/src/github.com/opencontainers/runc/GOPATH/src/github.com/opencontainers/runc/main.go:165 +0xae7 fp=0xc00014df98 sp=0xc00014dcc0 pc=0x5581fa426467
runtime.main()
  /usr/lib/golang/src/runtime/proc.go:201 +0x20f fp=0xc00014dfe0 sp=0xc00014df98 pc=0x5581fa177a0f
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00014dfe8 sp=0xc00014dfe0 pc=0x5581fa1a3801

goroutine 2 [force gc (idle)]:
runtime.gopark(0x5581fa7bcf98, 0x5581faa10970, 0x1410, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002af80 sp=0xc00002af60 pc=0x5581fa177e01
runtime.goparkunlock(0x5581faa10970, 0x1410, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002afb0 sp=0xc00002af80 pc=0x5581fa177eb5
runtime.forcegchelper()
  /usr/lib/golang/src/runtime/proc.go:251 +0xb7 fp=0xc00002afe0 sp=0xc00002afb0 pc=0x5581fa177c77
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002afe8 sp=0xc00002afe0 pc=0x5581fa1a3801
created by runtime.init.5
  /usr/lib/golang/src/runtime/proc.go:240 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x5581fa7bcf98, 0x5581faa10b00, 0x5581fa16140c, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002b780 sp=0xc00002b760 pc=0x5581fa177e01
runtime.goparkunlock(0x5581faa10b00, 0x5581fa49140c, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002b7b0 sp=0xc00002b780 pc=0x5581fa177eb5
runtime.bgsweep(0xc000052000)
  /usr/lib/golang/src/runtime/mgcsweep.go:52 +0x93 fp=0xc00002b7d8 sp=0xc00002b7b0 pc=0x5581fa16ac93
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002b7e0 sp=0xc00002b7d8 pc=0x5581fa1a3801
created by runtime.gcenable
  /usr/lib/golang/src/runtime/mgc.go:216 +0x5a

goroutine 4 [finalizer wait]:
runtime.gopark(0x5581fa7bcf98, 0x5581faa2dff0, 0x140f, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002bf28 sp=0xc00002bf08 pc=0x5581fa177e01
runtime.goparkunlock(0x5581faa2dff0, 0x140f, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002bf58 sp=0xc00002bf28 pc=0x5581fa177eb5
runtime.runfinq()
  /usr/lib/golang/src/runtime/mfinal.go:175 +0x9d fp=0xc00002bfe0 sp=0xc00002bf58 pc=0x5581fa16255d
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002bfe8 sp=0xc00002bfe0 pc=0x5581fa1a3801
created by runtime.createfing
  /usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 6 [timer goroutine (idle)]:
runtime.gopark(0x5581fa7bcf98, 0x5581faa14720, 0x536c1414, 0x1)
  /usr/lib/golang/src/runtime/proc.go:302 +0xf1 fp=0xc00002a728 sp=0xc00002a708 pc=0x5581fa177e01
runtime.goparkunlock(0x5581faa14720, 0xc000051414, 0x1)
  /usr/lib/golang/src/runtime/proc.go:308 +0x55 fp=0xc00002a758 sp=0xc00002a728 pc=0x5581fa177eb5
runtime.timerproc(0x5581faa14720)
  /usr/lib/golang/src/runtime/time.go:280 +0x28c fp=0xc00002a7d8 sp=0xc00002a758 pc=0x5581fa19480c
runtime.goexit()
  /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00002a7e0 sp=0xc00002a7d8 pc=0x5581fa1a3801
created by runtime.(*timersBucket).addtimerLocked
  /usr/lib/golang/src/runtime/time.go:170 +0x116

goroutine 18 [syscall]:
runtime.notetsleepg(0x5581faa2e340, 0xffffffffffffffff, 0xffffffff
  Normal   Pulling  8s (x7 over 118s)  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Pulling image "quay.io/openshift-knative/knative-eventing-sources-kafka-source-controller:v0.5.0"
  Normal   Pulled   7s (x7 over 117s)  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Successfully pulled image "quay.io/openshift-knative/knative-eventing-sources-kafka-source-controller:v0.5.0"
  Warning  Failed   2s (x4 over 104s)  kubelet, ip-10-0-138-252.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"read init-p: connection reset by peer\""






oc describe pod camel-controller-manager-0
Name:               camel-controller-manager-0
Namespace:          knative-eventing
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-0-148-82.us-west-2.compute.internal/10.0.148.82
Start Time:         Mon, 27 May 2019 04:19:47 -0400
Labels:             control-plane=camel-controller-manager
                    controller-revision-hash=camel-controller-manager-c58b9488c
                    statefulset.kubernetes.io/pod-name=camel-controller-manager-0
Annotations:        k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "openshift-sdn",
                          "interface": "eth0",
                          "ips": [
                              "10.131.0.20"
                          ],
                          "default": true,
                          "dns": {}
                      }]
                    openshift.io/scc: restricted
Status:             Running
IP:                 10.131.0.20
Controlled By:      StatefulSet/camel-controller-manager
Containers:
  manager:
    Container ID:   cri-o://028f9202fff538e72f79103698b483b5f06b5eb7ecd1e982ceb6fd9090f438f8
    Image:          quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0
    Image ID:       quay.io/openshift-knative/knative-eventing-sources-camel-source-controller@sha256:ef22ffa2ef0ff6450e32699b43b473d8c43014c850eee9c8d54671fca8fc7521
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 27 May 2019 04:29:20 -0400
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:        20m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from camel-controller-manager-token-zn6qj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  camel-controller-manager-token-zn6qj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  camel-controller-manager-token-zn6qj
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                                                Message
  ----     ------     ----                  ----                                                -------
  Normal   Scheduled  129m                  default-scheduler                                   Successfully assigned knative-eventing/camel-controller-manager-0 to ip-10-0-148-82.us-west-2.compute.internal
  Normal   Pulling    128m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Pulling image "quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0"
  Normal   Pulled     128m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Successfully pulled image "quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0"
  Warning  Failed     128m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:313: sending config to init process caused \"write init-p: no buffer space available\""
  Warning  Failed     127m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"failed to write 0-3\\n to cpuset.cpus: open /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3093f6ab_8058_11e9_8d8e_06ad16ce2654.slice/crio-521d0a85969acead6d0a5d8f35a1feecfac98467c768bdfe32eccc2655e1f08f.scope/cpuset.cpus: cannot allocate memory\""
  Warning  Failed     126m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:313: sending config to init process caused \"write init-p: broken pipe\""
  Warning  Failed     125m                  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"no space left on device\\\"\""
  Normal   Pulled     125m (x11 over 128m)  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Container image "quay.io/openshift-knative/knative-eventing-sources-camel-source-controller:v0.5.0" already present on machine
  Warning  Failed     125m (x2 over 127m)   kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for devices not found\""
  Warning  Failed     123m (x10 over 128m)  kubelet, ip-10-0-148-82.us-west-2.compute.internal  Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"read init-p: connection reset by peer\""

Comment 1 Marek Schmidt 2019-05-27 10:43:28 UTC

Also note, according to grafana, memory and disk are well below 30% on all nodes (as well as CPU besides a few spikes up to 50%), so this doesn't seem to be caused by actual resource pressures.

Comment 4 Marek Schmidt 2019-05-29 06:55:04 UTC

Removing just the CPU limit doesn't seem to help ( 'Error: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"read init-p: connection reset by peer\"' still occured)

Bumping up CPU and memory requests make the pods came up reliably.

Note You need to log in before you can comment on or make changes to this bug.