Hide Forgot
Created attachment 1238649 [details] heketi.log Description of problem: CNS: New pvc create failed post upgarde glusterfs: create volume err: error creating volume Failed to get list of pods 1. I deleted the old tempaltes and heketi dc and recreated heketi dc. 2. then created storageclass fast and secret 3. then tried to create pvc storage-claim6, it failed. [root@dhcp37-175 ~]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES AGE storage-claim1 Bound glusterfs-5a2edaba 10Gi RWX 3d storage-claim2 Bound glusterfs-6c411a7b 10Gi RWX 3d storage-claim3 Bound glusterfs-c02b48ea 10Gi RWX 3d storage-claim4 Bound glusterfs-9aadafad 20Gi RWX 3d storage-claim5 Bound glusterfs-a9f864da 20Gi RWX 3d storage-claim6 Pending 18m [root@dhcp37-175 ~]# oc describe pvc storage-claim6 Name: storage-claim6 Namespace: aplo StorageClass: fast Status: Pending Volume: Labels: <none> Capacity: Access Modes: Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 12m 1s 50 {persistentvolume-controller } Warning ProvisioningFailed Failed to provision volume with StorageClass "fast": glusterfs: create volume err: error creating volume Failed to get list of pods. [root@dhcp37-175 ~]# oc logs heketi-1-jso6j [sshexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:158: Failed to get list of pods [kubeexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:237: User "system:anonymous" cannot list pods in project "aplo" [sshexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:158: Failed to get list of pods [kubeexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:237: User "system:anonymous" cannot list pods in project "aplo" [sshexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:158: Failed to get list of pods [heketi] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:131: Failed to create volume: Failed to get list of pods [asynchttp] INFO 2017/01/09 05:38:04 Completed job 87b7908aab51d2e6f6cd23e50b322725 in 176.912713ms [negroni] Started GET /queue/87b7908aab51d2e6f6cd23e50b322725 [negroni] Completed 500 Internal Server Error in 208.389µs [root@dhcp37-175 ~]# oc get pods NAME READY STATUS RESTARTS AGE aplo-router-3-d6h2d 1/1 Running 2 2d glusterfs-5xwkm 1/1 Running 1 2d glusterfs-py4rs 1/1 Running 2 2d glusterfs-xy1tu 1/1 Running 1 2d heketi-1-jso6j 1/1 Running 0 22m mongodb-1-ymkbi 1/1 Running 1 2d nginx1 1/1 Running 2 2d [root@dhcp37-175 ~]# oc describe pod heketi Name: heketi-1-jso6j Namespace: aplo Security Policy: privileged Node: dhcp37-138.lab.eng.blr.redhat.com/10.70.37.138 Start Time: Mon, 09 Jan 2017 15:47:56 +0530 Labels: deployment=heketi-1 deploymentconfig=heketi glusterfs=heketi-pod name=heketi Status: Running IP: 10.1.1.7 Controllers: ReplicationController/heketi-1 Containers: heketi: Container ID: docker://ea73b9e401ea6e1f828cbb1b58b9f106a6f58cd27368504a3580d60009151c13 Image: rhgs3/rhgs-volmanager-rhel7 Image ID: docker-pullable://registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7@sha256:7e5d2fefd6e5debd796b684b9afa239d14210bf99ae5a5e6154e8251f55eb11a Port: 8080/TCP State: Running Started: Mon, 09 Jan 2017 15:48:19 +0530 Ready: True Restart Count: 0 Liveness: http-get http://:8080/hello delay=30s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/hello delay=3s timeout=3s period=10s #success=1 #failure=3 Volume Mounts: /var/lib/heketi from db (rw) /var/lib/heketi/secret from secret (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-cgwgu (ro) Environment Variables: HEKETI_USER_KEY: HEKETI_ADMIN_KEY: HEKETI_EXECUTOR: kubernetes HEKETI_FSTAB: /var/lib/heketi/fstab HEKETI_SNAPSHOT_LIMIT: 14 HEKETI_KUBE_CERTFILE: HEKETI_KUBE_INSECURE: y HEKETI_KUBE_GLUSTER_DAEMONSET: y HEKETI_KUBE_USE_SECRET: y HEKETI_KUBE_TOKENFILE: /var/lib/heketi/secret/token HEKETI_KUBE_NAMESPACE: aplo HEKETI_KUBE_APIHOST: https://10.70.37.149:8443 Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: db: Type: Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime) EndpointsName: heketi-storage-endpoints Path: heketidbstorage ReadOnly: false secret: Type: Secret (a volume populated by a Secret) SecretName: heketi-service-account-token-j7h3z default-token-cgwgu: Type: Secret (a volume populated by a Secret) SecretName: default-token-cgwgu QoS Class: BestEffort Tolerations: <none> Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 23m 23m 1 {default-scheduler } Normal Scheduled Successfully assigned heketi-1-jso6j to dhcp37-138.lab.eng.blr.redhat.com 22m 22m 1 {kubelet dhcp37-138.lab.eng.blr.redhat.com} spec.containers{heketi} Normal Pulling pulling image "rhgs3/rhgs-volmanager-rhel7" 22m 22m 1 {kubelet dhcp37-138.lab.eng.blr.redhat.com} spec.containers{heketi} Normal Pulled Successfully pulled image "rhgs3/rhgs-volmanager-rhel7" 22m 22m 1 {kubelet dhcp37-138.lab.eng.blr.redhat.com} spec.containers{heketi} Normal Created Created container with docker id ea73b9e401ea; Security:[seccomp=unconfined] 22m 22m 1 {kubelet dhcp37-138.lab.eng.blr.redhat.com} spec.containers{heketi} Normal Started Started container with docker id ea73b9e401ea [root@dhcp37-175 ~]# oc get dc NAME REVISION DESIRED CURRENT TRIGGERED BY aplo-router 3 1 1 config heketi 1 1 1 config mongodb 1 1 1 config,image(mongodb:3.2) [root@dhcp37-175 ~]# oc describe dc heketi Name: heketi Namespace: aplo Created: 23 minutes ago Labels: glusterfs=heketi-dc template=heketi Description: Defines how to deploy Heketi Annotations: <none> Latest Version: 1 Selector: name=heketi Replicas: 1 Triggers: Config Strategy: Rolling Template: Labels: glusterfs=heketi-pod name=heketi Containers: heketi: Image: rhgs3/rhgs-volmanager-rhel7 Port: 8080/TCP Liveness: http-get http://:8080/hello delay=30s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/hello delay=3s timeout=3s period=10s #success=1 #failure=3 Volume Mounts: /var/lib/heketi from db (rw) /var/lib/heketi/secret from secret (rw) Environment Variables: HEKETI_USER_KEY: HEKETI_ADMIN_KEY: HEKETI_EXECUTOR: kubernetes HEKETI_FSTAB: /var/lib/heketi/fstab HEKETI_SNAPSHOT_LIMIT: 14 HEKETI_KUBE_CERTFILE: HEKETI_KUBE_INSECURE: y HEKETI_KUBE_GLUSTER_DAEMONSET: y HEKETI_KUBE_USE_SECRET: y HEKETI_KUBE_TOKENFILE: /var/lib/heketi/secret/token HEKETI_KUBE_NAMESPACE: aplo HEKETI_KUBE_APIHOST: https://10.70.37.149:8443 Volumes: db: Type: Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime) EndpointsName: heketi-storage-endpoints Path: heketidbstorage ReadOnly: false secret: Type: Secret (a volume populated by a Secret) SecretName: heketi-service-account-token-j7h3z Deployment #1 (latest): Name: heketi-1 Created: 23 minutes ago Status: Complete Replicas: 1 current / 1 desired Selector: deployment=heketi-1,deploymentconfig=heketi,name=heketi Labels: glusterfs=heketi-dc,openshift.io/deployment-config.name=heketi,template=heketi Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 23m 23m 1 {deploymentconfig-controller } Normal DeploymentCreated Created new replication controller "heketi-1" for version 1
The setup was analysed and found an issue with the image mentioned in the template. I am closing this bug as NOTABUG.