Bug 1388897

Summary: CNS: Not able to start a pod - RunContainerError
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Prasanth <pprakash>
Component: CNS-deploymentAssignee: Mohamed Ashiq <mliyazud>
Status: CLOSED CURRENTRELEASE QA Contact: Prasanth <pprakash>
Severity: high Docs Contact:
Priority: unspecified    
Version: cns-3.4CC: akhakhar, annair, aos-bugs, dma, hchiramm, jarrpa, jhonce, jokerman, madam, mliyazud, mmccomas, mzywusko, nerawat, pprakash, rcyriac, rhs-bugs, rmekala, rreddy, rtalur, sankarshan
Target Milestone: ---   
Target Release: CNS 3.4   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1388883 Environment:
Last Closed: 2017-01-23 07:12:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1388839, 1388883    
Bug Blocks: 1385247    

Description Prasanth 2016-10-26 11:39:58 UTC 
+++ This bug was initially created as a clone of Bug #1388883 +++

Description of problem:
CNS: Not able to setup router - RunContainerError
On rhel7.3

Version-Release number of selected component (if applicable):
[root@rhsauto049 ~]# oadm version
oadm v3.4.0.15+9c963ec
kubernetes v1.4.0+776c994

Server https://10.70.40.128:8443
openshift v3.4.0.15+9c963ec
kubernetes v1.4.0+776c994

Red Hat Enterprise Linux Server release 7.3 (Maipo)

[root@rhsauto049 ~]# oc get pods
NAME                                                      READY     STATUS              RESTARTS   AGE
aplo-router-1-deploy                                      0/1       ContainerCreating   0          3h

[root@rhsauto049 ~]# oc describe pod aplo-router-1-deploy 
Name:			aplo-router-1-deploy
Namespace:		aplo
Security Policy:	restricted
Node:			rhsauto048.lab.eng.blr.redhat.com/10.70.40.123
Start Time:		Wed, 26 Oct 2016 13:03:43 +0530
Labels:			openshift.io/deployer-pod-for.name=aplo-router-1
Status:			Pending
IP:			
Controllers:		<none>
Containers:
  deployment:
    Container ID:	
    Image:		openshift3/ose-deployer:v3.4.0.15
    Image ID:		
    Port:		
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from deployer-token-yiqmb (ro)
    Environment Variables:
      KUBERNETES_MASTER:	https://rhsauto053.lab.eng.blr.redhat.com:8443
      OPENSHIFT_MASTER:		https://rhsauto053.lab.eng.blr.redhat.com:8443
      BEARER_TOKEN_FILE:	/var/run/secrets/kubernetes.io/serviceaccount/token
      OPENSHIFT_CA_DATA:	-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE0Nzc0NjY2NTUwHhcNMTYxMDI2MDcyNDE1WhcNMjExMDI1
MDcyNDE2WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE0Nzc0NjY2NTUw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCTFo8IBKsKZ1WT2Za/yHR5
v9uPpeV/rN9RnWl4OB4/ZcPHVGcSsTCk4ClQ4HS8Fxeiybs/bJoqNtP6vrrOweNk
fSa9F9ExvYgPSLYIqZJSxKXCuU4OM4bXBNkA+E6skQ0i29n0Gp0mBHudE7DUJSeE
YkxSkOAdgnwcHxcshek+R+Z77J084yIsEs+JIgh7fRftYa7mMp7lWpve2P/dLHyH
Zqn+yvnmWqgoC0VdcmU39ihzlseD1RtZv0nSlwzdqs98IaaqdZNAXvf3e58oDUTM
rH97hPJmB0OLoMn0B1aA7velhKt6mp4eks07qd9i3avvfwwmViLz6YsEem9Qabib
AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQABPEq1TV0JQrnQeKGNp4oEIZ5jFiPqP6abc9g+UHZmXMBa
iGvNlYreAtDHEmS2mV6zSDjzV0XZC4h1MwKNp7zpD159hPb2+MwqbWYnGWKQzzof
sJxXi+Pv8UP3jgJ27sptYg9weEPoFqrzMRqVBibrNUQZZl488Lf+Yt2m9aK3vQ21
b0BknN0oz193el3AzVYSlszTJ3s4ET3MB1BOzjbsyCWnPP3V0LUwr8J1NHAEj2az
WqWub/7niVv7186H1UZWE4iuRuewH0EttBMbIlkq4la550sg4AyBMob3U0USmBkS
yeyxHFSVN63lqxNFl7u+L8n1R1jer41D+RsQ3RX2
-----END CERTIFICATE-----

      OPENSHIFT_DEPLOYMENT_NAME:	aplo-router-1
      OPENSHIFT_DEPLOYMENT_NAMESPACE:	aplo
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	False 
  PodScheduled 	True 
Volumes:
  deployer-token-yiqmb:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	deployer-token-yiqmb
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From						SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----						-------------	--------	------		-------
  1h		1h		1	{kubelet rhsauto048.lab.eng.blr.redhat.com}			Warning		FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
5437a621c2be370aae7e916ff9227768f07fab08035f9e1d5bb73cc82c56ed6f: [9] System error: invalid character 'a' looking for
beginning of value"

  1h	1h	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
4ae0e84ccb280e77dd0fb3bbc16555154d13e2562a59824ac6bf4180b03cbc0f: [9] System error: invalid character 'a' looking for
beginning of value"

  37m	37m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
bd3af951c9942856b47aa9ef6caeb10c56c9f8a948716779c025de4aaeba5b66: [9] System error: invalid character 'a' looking for
beginning of value"

  35m	35m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
bc4d8c8df142a873192789bbcd6923124067f399399a994d32effea321387224: [9] System error: invalid character 'a' looking for
beginning of value"

  30m	30m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Cannot start container
333331525b6af359098541d71015697f944aea81161bfd2fc3909c194e79d96d: [9] System error: invalid character 'a' looking for
beginning of value"

  26m	26m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: Error response
from daemon: devmapper: Unknown device bf6d8ff5aebdf1c57e377a0b35e371e5ce43c4fe37bfa7821b82a0fe4a23210c
  24m	24m	1	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	(events with common reason combined)
  3h	16s	1089	{kubelet rhsauto048.lab.eng.blr.redhat.com}		Warning	FailedSync	Error syncing pod, skipping: failed to
"StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Container command could
not be invoked."

--- Additional comment from Apeksha on 2016-10-26 07:31:38 EDT ---

sosreports - http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1388883/
Comment 7 Prasanth 2016-11-10 09:32:06 UTC 
Verified as fixed in the latest OCP build

oc v3.4.0.24+52fd77b
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dhcp47-127.lab.eng.blr.redhat.com:8443
openshift v3.4.0.24+52fd77b
kubernetes v1.4.0+776c994