Bug 1388883

Summary: CNS: Not able to start a pod - RunContainerError
Product: OpenShift Container Platform Reporter: Apeksha <akhakhar>
Component: ContainersAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED DUPLICATE QA Contact: DeShuai Ma <dma>
Severity: high Docs Contact:
Priority: medium    
Version: 3.4.0CC: agoldste, aos-bugs, eparis, jokerman, mmccomas, rcyriac
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1388897 (view as bug list) Environment:
Last Closed: 2016-10-31 08:19:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1388897    

Description Apeksha 2016-10-26 11:06:58 UTC 
Description of problem:
CNS: Not able to setup router - RunContainerError
On rhel7.3

Version-Release number of selected component (if applicable):
[root@rhsauto049 ~]# oadm version
oadm v3.4.0.15+9c963ec
kubernetes v1.4.0+776c994

Server https://10.70.40.128:8443
openshift v3.4.0.15+9c963ec
kubernetes v1.4.0+776c994

Red Hat Enterprise Linux Server release 7.3 (Maipo)

[root@rhsauto049 ~]# oc get pods
NAME                                                      READY     STATUS              RESTARTS   AGE
aplo-router-1-deploy                                      0/1       ContainerCreating   0          3h

[root@rhsauto049 ~]# oc describe pod aplo-router-1-deploy 
Name:			aplo-router-1-deploy
Namespace:		aplo
Security Policy:	restricted
Node:			rhsauto048.lab.eng.blr.redhat.com/10.70.40.123
Start Time:		Wed, 26 Oct 2016 13:03:43 +0530
Labels:			openshift.io/deployer-pod-for.name=aplo-router-1
Status:			Pending
IP:			
Controllers:		<none>
Containers:
  deployment:
    Container ID:	
    Image:		openshift3/ose-deployer:v3.4.0.15
    Image ID:		
    Port:		
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from deployer-token-yiqmb (ro)
    Environment Variables:
      KUBERNETES_MASTER:	https://rhsauto053.lab.eng.blr.redhat.com:8443
      OPENSHIFT_MASTER:		https://rhsauto053.lab.eng.blr.redhat.com:8443
      BEARER_TOKEN_FILE:	/var/run/secrets/kubernetes.io/serviceaccount/token
      OPENSHIFT_CA_DATA:	-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE0Nzc0NjY2NTUwHhcNMTYxMDI2MDcyNDE1WhcNMjExMDI1
MDcyNDE2WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE0Nzc0NjY2NTUw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCTFo8IBKsKZ1WT2Za/yHR5
v9uPpeV/rN9RnWl4OB4/ZcPHVGcSsTCk4ClQ4HS8Fxeiybs/bJoqNtP6vrrOweNk
fSa9F9ExvYgPSLYIqZJSxKXCuU4OM4bXBNkA+E6skQ0i29n0Gp0mBHudE7DUJSeE
YkxSkOAdgnwcHxcshek+R+Z77J084yIsEs+JIgh7fRftYa7mMp7lWpve2P/dLHyH
Zqn+yvnmWqgoC0VdcmU39ihzlseD1RtZv0nSlwzdqs98IaaqdZNAXvf3e58oDUTM
rH97hPJmB0OLoMn0B1aA7velhKt6mp4eks07qd9i3avvfwwmViLz6YsEem9Qabib
AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQABPEq1TV0JQrnQeKGNp4oEIZ5jFiPqP6abc9g+UHZmXMBa
iGvNlYreAtDHEmS2mV6zSDjzV0XZC4h1MwKNp7zpD159hPb2+MwqbWYnGWKQzzof
sJxXi+Pv8UP3jgJ27sptYg9weEPoFqrzMRqVBibrNUQZZl488Lf+Yt2m9aK3vQ21
b0BknN0oz193el3AzVYSlszTJ3s4ET3MB1BOzjbsyCWnPP3V0LUwr8J1NHAEj2az
WqWub/7niVv7186H1UZWE4iuRuewH0EttBMbIlkq4la550sg4AyBMob3U0USmBkS
yeyxHFSVN63lqxNFl7u+L8n1R1jer41D+RsQ3RX2
-----END CERTIFICATE-----

      OPENSHIFT_DEPLOYMENT_NAME:	aplo-router-1
      OPENSHIFT_DEPLOYMENT_NAMESPACE:	aplo
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	False 
  PodScheduled 	True 
Volumes:
  deployer-token-yiqmb:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	deployer-token-yiqmb
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From						SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----						-------------	--------	------		-------
  1h		1h		1	{kubelet rhsauto048.lab.eng.blr.redhat.com}			Warning		FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
5437a621c2be370aae7e916ff9227768f07fab08035f9e1d5bb73cc82c56ed6f: [9] System error: invalid character 'a' looking for
beginning of value"

  1h	1h	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
4ae0e84ccb280e77dd0fb3bbc16555154d13e2562a59824ac6bf4180b03cbc0f: [9] System error: invalid character 'a' looking for
beginning of value"

  37m	37m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
bd3af951c9942856b47aa9ef6caeb10c56c9f8a948716779c025de4aaeba5b66: [9] System error: invalid character 'a' looking for
beginning of value"

  35m	35m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
bc4d8c8df142a873192789bbcd6923124067f399399a994d32effea321387224: [9] System error: invalid character 'a' looking for
beginning of value"

  30m	30m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
333331525b6af359098541d71015697f944aea81161bfd2fc3909c194e79d96d: [9] System error: invalid character 'a' looking for
beginning of value"

  26m	26m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: Error response
from daemon: devmapper: Unknown device bf6d8ff5aebdf1c57e377a0b35e371e5ce43c4fe37bfa7821b82a0fe4a23210c
  24m	24m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	(events with common reason combined)
  3h	16s	1089	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Container command could
not be invoked."
Comment 1 Apeksha 2016-10-26 11:31:38 UTC 
sosreports - http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1388883/
Comment 2 Andy Goldstein 2016-10-26 14:25:40 UTC 
I think this is what the "invalid character 'a'" is about:

[root@rhsauto048 ~]# docker pull docker.io/busybox
Using default tag: latest
Trying to pull repository docker.io/library/busybox ...
latest: Pulling from docker.io/library/busybox
56bec22e3559: Pull complete
Digest: sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912
Status: Downloaded newer image for docker.io/busybox:latest
[root@rhsauto048 ~]# docker run --rm docker.io/busybox date
docker: Error response from daemon: failed to create endpoint hopeful_lovelace on network bridge: adding interface veth5b5eab6 to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface.

I think the 'a' is the start of 'adding interface veth5b5eab6 to bridge docker0 failed: could not find bridge docker0: route ip+net: no such network interface.'
Comment 3 Eric Paris 2016-10-26 14:44:42 UTC 
Andy's comment is a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1388839
Comment 4 Eric Paris 2016-10-26 18:18:08 UTC 
This is likely a dup of 1388839. Since we were explicitly deleting the docker0 bridge, fixed in the linked openshift PR
Comment 5 Antonio Murdaca 2016-10-31 08:19:48 UTC 

*** This bug has been marked as a duplicate of bug 1388839 ***