Bug 1411282 - CNS: New pvc create failed post CNS upgrade
Summary: CNS: New pvc create failed post CNS upgrade
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: CNS-deployment
Version: cns-3.4
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Michael Adam
QA Contact: Anoop
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-09 10:44 UTC by Apeksha
Modified: 2017-01-09 11:58 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-09 11:58:29 UTC
Target Upstream Version:


Attachments (Terms of Use)
heketi.log (1001.17 KB, text/plain)
2017-01-09 10:44 UTC, Apeksha
no flags Details

Description Apeksha 2017-01-09 10:44:26 UTC
Created attachment 1238649 [details]
heketi.log

Description of problem:
CNS: New pvc create failed post upgarde 
glusterfs: create volume err: error creating volume Failed to get list of pods

1. I deleted the old tempaltes and heketi dc and recreated heketi dc.
2. then created storageclass fast and secret
3. then tried to create pvc storage-claim6, it failed.

[root@dhcp37-175 ~]# oc get pvc
NAME             STATUS    VOLUME               CAPACITY   ACCESSMODES   AGE
storage-claim1   Bound     glusterfs-5a2edaba   10Gi       RWX           3d
storage-claim2   Bound     glusterfs-6c411a7b   10Gi       RWX           3d
storage-claim3   Bound     glusterfs-c02b48ea   10Gi       RWX           3d
storage-claim4   Bound     glusterfs-9aadafad   20Gi       RWX           3d
storage-claim5   Bound     glusterfs-a9f864da   20Gi       RWX           3d
storage-claim6   Pending                                                 18m

[root@dhcp37-175 ~]# oc describe pvc storage-claim6 
Name:		storage-claim6
Namespace:	aplo
StorageClass:	fast
Status:		Pending
Volume:		
Labels:		<none>
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  12m		1s		50	{persistentvolume-controller }			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "fast": glusterfs: create volume err: error creating volume Failed to get list of pods.


[root@dhcp37-175 ~]# oc logs heketi-1-jso6j
[sshexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:158: Failed to get list of pods
[kubeexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:237: User "system:anonymous" cannot list pods in project "aplo"
[sshexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:158: Failed to get list of pods
[kubeexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:237: User "system:anonymous" cannot list pods in project "aplo"
[sshexec] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:158: Failed to get list of pods
[heketi] ERROR 2017/01/09 05:38:04 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:131: Failed to create volume: Failed to get list of pods
[asynchttp] INFO 2017/01/09 05:38:04 Completed job 87b7908aab51d2e6f6cd23e50b322725 in 176.912713ms
[negroni] Started GET /queue/87b7908aab51d2e6f6cd23e50b322725
[negroni] Completed 500 Internal Server Error in 208.389µs



[root@dhcp37-175 ~]# oc get pods
NAME                  READY     STATUS    RESTARTS   AGE
aplo-router-3-d6h2d   1/1       Running   2          2d
glusterfs-5xwkm       1/1       Running   1          2d
glusterfs-py4rs       1/1       Running   2          2d
glusterfs-xy1tu       1/1       Running   1          2d
heketi-1-jso6j        1/1       Running   0          22m
mongodb-1-ymkbi       1/1       Running   1          2d
nginx1                1/1       Running   2          2d


[root@dhcp37-175 ~]# oc describe pod heketi
Name:			heketi-1-jso6j
Namespace:		aplo
Security Policy:	privileged
Node:			dhcp37-138.lab.eng.blr.redhat.com/10.70.37.138
Start Time:		Mon, 09 Jan 2017 15:47:56 +0530
Labels:			deployment=heketi-1
			deploymentconfig=heketi
			glusterfs=heketi-pod
			name=heketi
Status:			Running
IP:			10.1.1.7
Controllers:		ReplicationController/heketi-1
Containers:
  heketi:
    Container ID:	docker://ea73b9e401ea6e1f828cbb1b58b9f106a6f58cd27368504a3580d60009151c13
    Image:		rhgs3/rhgs-volmanager-rhel7
    Image ID:		docker-pullable://registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7@sha256:7e5d2fefd6e5debd796b684b9afa239d14210bf99ae5a5e6154e8251f55eb11a
    Port:		8080/TCP
    State:		Running
      Started:		Mon, 09 Jan 2017 15:48:19 +0530
    Ready:		True
    Restart Count:	0
    Liveness:		http-get http://:8080/hello delay=30s timeout=3s period=10s #success=1 #failure=3
    Readiness:		http-get http://:8080/hello delay=3s timeout=3s period=10s #success=1 #failure=3
    Volume Mounts:
      /var/lib/heketi from db (rw)
      /var/lib/heketi/secret from secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cgwgu (ro)
    Environment Variables:
      HEKETI_USER_KEY:			
      HEKETI_ADMIN_KEY:			
      HEKETI_EXECUTOR:			kubernetes
      HEKETI_FSTAB:			/var/lib/heketi/fstab
      HEKETI_SNAPSHOT_LIMIT:		14
      HEKETI_KUBE_CERTFILE:		
      HEKETI_KUBE_INSECURE:		y
      HEKETI_KUBE_GLUSTER_DAEMONSET:	y
      HEKETI_KUBE_USE_SECRET:		y
      HEKETI_KUBE_TOKENFILE:		/var/lib/heketi/secret/token
      HEKETI_KUBE_NAMESPACE:		aplo
      HEKETI_KUBE_APIHOST:		https://10.70.37.149:8443
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	True 
  PodScheduled 	True 
Volumes:
  db:
    Type:		Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:	heketi-storage-endpoints
    Path:		heketidbstorage
    ReadOnly:		false
  secret:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	heketi-service-account-token-j7h3z
  default-token-cgwgu:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-cgwgu
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From						SubobjectPath		Type		Reason		Message
  ---------	--------	-----	----						-------------		--------	------		-------
  23m		23m		1	{default-scheduler }							Normal		Scheduled	Successfully assigned heketi-1-jso6j to dhcp37-138.lab.eng.blr.redhat.com
  22m		22m		1	{kubelet dhcp37-138.lab.eng.blr.redhat.com}	spec.containers{heketi}	Normal		Pulling		pulling image "rhgs3/rhgs-volmanager-rhel7"
  22m		22m		1	{kubelet dhcp37-138.lab.eng.blr.redhat.com}	spec.containers{heketi}	Normal		Pulled		Successfully pulled image "rhgs3/rhgs-volmanager-rhel7"
  22m		22m		1	{kubelet dhcp37-138.lab.eng.blr.redhat.com}	spec.containers{heketi}	Normal		Created		Created container with docker id ea73b9e401ea; Security:[seccomp=unconfined]
  22m		22m		1	{kubelet dhcp37-138.lab.eng.blr.redhat.com}	spec.containers{heketi}	Normal		Started		Started container with docker id ea73b9e401ea


[root@dhcp37-175 ~]# oc get dc
NAME          REVISION   DESIRED   CURRENT   TRIGGERED BY
aplo-router   3          1         1         config
heketi        1          1         1         config
mongodb       1          1         1         config,image(mongodb:3.2)

[root@dhcp37-175 ~]# oc describe dc heketi 
Name:		heketi
Namespace:	aplo
Created:	23 minutes ago
Labels:		glusterfs=heketi-dc
		template=heketi
Description:	Defines how to deploy Heketi
Annotations:	<none>
Latest Version:	1
Selector:	name=heketi
Replicas:	1
Triggers:	Config
Strategy:	Rolling
Template:
  Labels:	glusterfs=heketi-pod
		name=heketi
  Containers:
   heketi:
    Image:	rhgs3/rhgs-volmanager-rhel7
    Port:	8080/TCP
    Liveness:	http-get http://:8080/hello delay=30s timeout=3s period=10s #success=1 #failure=3
    Readiness:	http-get http://:8080/hello delay=3s timeout=3s period=10s #success=1 #failure=3
    Volume Mounts:
      /var/lib/heketi from db (rw)
      /var/lib/heketi/secret from secret (rw)
    Environment Variables:
      HEKETI_USER_KEY:			
      HEKETI_ADMIN_KEY:			
      HEKETI_EXECUTOR:			kubernetes
      HEKETI_FSTAB:			/var/lib/heketi/fstab
      HEKETI_SNAPSHOT_LIMIT:		14
      HEKETI_KUBE_CERTFILE:		
      HEKETI_KUBE_INSECURE:		y
      HEKETI_KUBE_GLUSTER_DAEMONSET:	y
      HEKETI_KUBE_USE_SECRET:		y
      HEKETI_KUBE_TOKENFILE:		/var/lib/heketi/secret/token
      HEKETI_KUBE_NAMESPACE:		aplo
      HEKETI_KUBE_APIHOST:		https://10.70.37.149:8443
  Volumes:
   db:
    Type:		Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:	heketi-storage-endpoints
    Path:		heketidbstorage
    ReadOnly:		false
   secret:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	heketi-service-account-token-j7h3z

Deployment #1 (latest):
	Name:		heketi-1
	Created:	23 minutes ago
	Status:		Complete
	Replicas:	1 current / 1 desired
	Selector:	deployment=heketi-1,deploymentconfig=heketi,name=heketi
	Labels:		glusterfs=heketi-dc,openshift.io/deployment-config.name=heketi,template=heketi
	Pods Status:	1 Running / 0 Waiting / 0 Succeeded / 0 Failed

Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  23m		23m		1	{deploymentconfig-controller }			Normal		DeploymentCreated	Created new replication controller "heketi-1" for version 1

Comment 1 Humble Chirammal 2017-01-09 11:58:29 UTC
The setup was analysed and found an issue with the image mentioned in the template. I am closing this bug as NOTABUG.


Note You need to log in before you can comment on or make changes to this bug.