Bug 1470695

Summary:	Error during pod listing: rpc error: code = 2 desc = An error occurred trying to connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=k8s_sti-build_cakephp-example-5-build_dakinitest-20170425-17_38c1051f-5c45-11e7-9627-02306c0cdc4b
Product:	OpenShift Container Platform	Reporter:	Justin Pierce <jupierce>
Component:	Node	Assignee:	Seth Jennings <sjenning>
Status:	CLOSED DUPLICATE	QA Contact:	DeShuai Ma <dma>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.6.0	CC:	aos-bugs, jokerman, mmccomas
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-07-14 21:17:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Justin Pierce 2017-07-13 13:28:39 UTC

Description of problem:
Running 'oc get pods --all-namespaces' on our free-stg cluster shows an odd error for for one of the pods.

-----------------------------------------------
.....
dakinitest-20170425-17   cakephp-example-4-jsz9j               1/1       Running                                                                                                                                                                                                                                        0          7d
dakinitest-20170425-17   cakephp-example-5-build               0/1       rpc error: code = 2 desc = An error occurred trying to connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=k8s_sti-build_cakephp-example-5-build_dakinitest-20170425-17_38c1051f-5c45-11e7-9627-02306c0cdc4b_0: EOF   0          14d
dakinitest-20170425-17   cakephp-example-6-build               0/1       Completed                                                                                                                                                                                                                                      0          7d
dakinitest-20170425-18   cakephp-example-4-zk1gx               1/1       Running                                                                                                                                                                                                                                        0          7d
dakinitest-20170425-18   cakephp-example-5-build               0/1       ContainerCannotRun                                                                                                                                                                                                                             0          14d
.....
-----------------------------------------------

Version-Release number of selected component (if applicable):
oc v3.6.126.1
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://internal.api.free-stg.openshift.com:443
openshift v3.6.126.1
kubernetes v1.6.1+5115d708d7


How reproducible:
On this cluster, the state seems persistent.


Steps to Reproduce:
1. Unknown
2.
3.

Actual results:
Status reported as: dakinitest-20170425-17   cakephp-example-5-build               0/1       rpc error: code = 2 desc = An error occurred trying to connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=k8s_sti-build_cakephp-example-5-build_dakinitest-20170425-17_38c1051f-5c45-11e7-9627-02306c0cdc4b_0: EOF   0          14d


Expected results:
Expecting a valid status for the pod. 


Additional info:
----------------
apiVersion: v1
kind: Pod
metadata:
  annotations:
    openshift.io/build.name: cakephp-example-5
    openshift.io/scc: privileged
  creationTimestamp: 2017-06-28T21:03:23Z
  labels:
    openshift.io/build.name: cakephp-example-5
  name: cakephp-example-5-build
  namespace: dakinitest-20170425-17
  ownerReferences:
  - apiVersion: build.openshift.io/v1
    controller: true
    kind: Build
    name: cakephp-example-5
    uid: 38b22187-5c45-11e7-9627-02306c0cdc4b
  resourceVersion: "22856943"
  selfLink: /api/v1/namespaces/dakinitest-20170425-17/pods/cakephp-example-5-build
  uid: 38c1051f-5c45-11e7-9627-02306c0cdc4b
spec:
  activeDeadlineSeconds: 3600
  containers:
  - args:
    - --loglevel=0
    env:
    - name: BUILD
      value: |
        {"kind":"Build","apiVersion":"v1","metadata":{"name":"cakephp-example-5","namespace":"dakinitest-20170425-17","selfLink":"/apis/build.openshift.io/v1/namespaces/dakinitest-20170425-17/builds/cakephp-example-5","uid":"38b22187-5c45-11e7-9627-02306c0cdc4b","resourceVersion":"22841496","creationTimestamp":"2017-06-28T21:03:23Z","labels":{"app":"cakephp-example","buildconfig":"cakephp-example","openshift.io/build-config.name":"cakephp-example","openshift.io/build.start-policy":"Serial","template":"cakephp-example"},"annotations":{"openshift.io/build-config.name":"cakephp-example","openshift.io/build.number":"5"},"ownerReferences":[{"apiVersion":"build.openshift.io/v1","kind":"BuildConfig","name":"cakephp-example","uid":"189f5919-29fc-11e7-90ae-027558dba8e1","controller":true}]},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/openshift/cakephp-ex.git"}},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"registry.access.redhat.com/rhscl/php-56-rhel7@sha256:351e3809aced37c24aafa3af361dc0d24c12a65d44a18a7186fc226ff0cf78e5"},"forcePull":true}},"output":{"to":{"kind":"DockerImage","name":"172.30.44.192:5000/dakinitest-20170425-17/cakephp-example:latest"},"pushSecret":{"name":"builder-dockercfg-gkh39"}},"resources":{},"postCommit":{},"nodeSelector":null,"triggeredBy":[{"message":"Image change","imageChangeBuild":{"fromRef":{"kind":"ImageStreamTag","namespace":"openshift","name":"php:5.6"}}}]},"status":{"phase":"New","outputDockerImageReference":"172.30.44.192:5000/dakinitest-20170425-17/cakephp-example:latest","config":{"kind":"BuildConfig","namespace":"dakinitest-20170425-17","name":"cakephp-example"},"output":{}}}
    - name: SOURCE_REPOSITORY
      value: https://github.com/openshift/cakephp-ex.git
    - name: SOURCE_URI
      value: https://github.com/openshift/cakephp-ex.git
    - name: ORIGIN_VERSION
      value: v3.6.126.1
    - name: PUSH_DOCKERCFG_PATH
      value: /var/run/secrets/openshift.io/push
    image: registry.ops.openshift.com/openshift3/ose-sti-builder:v3.6.126.1
    imagePullPolicy: Always
    name: sti-build
    resources: {}
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: docker-socket
    - mountPath: /var/run/secrets/openshift.io/push
      name: builder-dockercfg-gkh39-push
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: builder-token-zfnfh
      readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
  - name: builder-dockercfg-gkh39
  nodeName: ip-172-31-71-195.us-east-2.compute.internal
  nodeSelector:
    type: compute
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: builder
  serviceAccountName: builder
  terminationGracePeriodSeconds: 30
  volumes:
  - hostPath:
      path: /var/run/docker.sock
    name: docker-socket
  - name: builder-dockercfg-gkh39-push
    secret:
      defaultMode: 420
      secretName: builder-dockercfg-gkh39
  - name: builder-token-zfnfh
    secret:
      defaultMode: 420
      secretName: builder-token-zfnfh
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2017-06-28T21:13:17Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2017-06-28T21:13:17Z
    message: 'containers with unready status: [sti-build]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2017-06-28T21:13:17Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://4c92e103b9e78c2e5e25806027bc8635afc6c9b911c98744a7bb13dec291822c
    image: registry.ops.openshift.com/openshift3/ose-sti-builder:v3.6.126.1
    imageID: docker-pullable://registry.ops.openshift.com/openshift3/ose-sti-builder@sha256:982e8cabf5575e34280bdf06e1dfd068c8d472c73f9b44c7ab7458ce996cb071
    lastState:
      terminated:
        containerID: docker://4c92e103b9e78c2e5e25806027bc8635afc6c9b911c98744a7bb13dec291822c
        exitCode: 0
        finishedAt: null
        startedAt: null
    name: sti-build
    ready: false
    restartCount: 0
    state:
      waiting:
        message: Create Container Failed
        reason: 'rpc error: code = 2 desc = An error occurred trying to connect: Post
          http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=k8s_sti-build_cakephp-example-5-build_dakinitest-20170425-17_38c1051f-5c45-11e7-9627-02306c0cdc4b_0:
          EOF'
  hostIP: 172.31.71.195
  phase: Failed
  qosClass: BestEffort
  startTime: 2017-06-28T21:13:17Z
----------------

Comment 1 Derek Carr 2017-07-14 13:41:08 UTC

well, the reason in this case is that the container could not be created...

I suspect this is already fixed via:
https://github.com/openshift/origin/pull/14887

Seth can you confirm?

Comment 2 Seth Jennings 2017-07-14 19:11:09 UTC

Derek,

The bug was verified as fixed in v3.6.114 and this is v3.6.126.1.

Looking further...

Comment 3 Seth Jennings 2017-07-14 19:19:18 UTC

Meh, QE comment in the original bug was wrong.  Fix for 1449820 didn't go in until v3.6.128

$ git describe --contains 573d2d6e4d805a3bcc57956fe386d61037875033
v3.6.128-1~2^2~44^2

This should be fixed in v3.6.128 and later.

Comment 4 Seth Jennings 2017-07-14 21:17:30 UTC

If you can recreate on 3.6.128 or higher, please reopen.

*** This bug has been marked as a duplicate of bug 1449820 ***