Bug 1549140

Summary:	heketi-pod fails to come back, after we kill and restart the heketi pod
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Apeksha <akhakhar>
Component:	heketi	Assignee:	Michael Adam <madam>
Status:	CLOSED DUPLICATE	QA Contact:	Apeksha <akhakhar>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	cns-3.9	CC:	hchiramm, jcall, kramdoss, mliyazud, rgeorge, rhs-bugs, rtalur, storage-qa-internal
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-05 09:21:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Apeksha 2018-02-26 13:55:15 UTC

Description of problem:
heketi-pod fails to come back, after we kill and restart the heketi pod

Version-Release number of selected component (if applicable):
heketi-client-6.0.0-3.el7rhgs.x86_64
cns-deploy-6.0.0-3.el7rhgs.x86_64
atomic-openshift-3.9.0-0.47.0.git.0.965194d.el7.x86_64
openshift-ansible-3.9.0-0.47.0.git.0.f8847bb.el7.noarch

How reproducible: Once


Steps to Reproduce:
Run the automated testcase for heketi-pod failure, test fails.

1. have the setup up and running, 1 app-pod created.
2. Now delete the heketi pod
   command: oc delete dc,service,route heketi
3. now cretate a pvc it goes to pending state
4. Recreate heketi process
5. heketi pod fails to come back.

[root@dhcp46-24 ~]# oc get pods
NAME                                  READY     STATUS             RESTARTS   AGE
glusterblock-provisioner-dc-1-c5hcl   1/1       Running            0          3d
glusterfs-mgp2k                       1/1       Running            0          3d
glusterfs-tt5rs                       1/1       Running            1          3d
glusterfs-wbgzs                       1/1       Running            1          3d
heketi-1-tblcg                        0/1       CrashLoopBackOff   66         5h
storage-project-router-1-z5tg8        1/1       Running            0          3d


Actual results: heketi pod fails to come back.

Expected results: heketi pod should come back and pvc should be in bound state.

Comment 2 Apeksha 2018-02-26 13:57:23 UTC

Additional info:

    [root@dhcp46-24 ~]# oc logs heketi-1-tblcg
    Heketi 6.0.0
    [heketi] INFO 2018/02/26 08:36:07 Loaded kubernetes executor
    [heketi] INFO 2018/02/26 08:36:07 Please refer to the Heketi troubleshooting documentation for more information on how to resolve this issue.
    [heketi] ERROR 2018/02/26 08:36:07 /src/github.com/heketi/heketi/apps/glusterfs/app.go:150: Heketi terminated while performing one or more operations. Server will not start as long as pending operations are present in the db.
    panic: Heketi terminated while performing one or more operations. Server will not start as long as pending operations are present in the db.
     
    goroutine 1 [running]:
    github.com/heketi/heketi/apps/glusterfs.NewApp(0x236bf40, 0xc42000e198, 0x0)
            /builddir/build/BUILD/heketi-6.0.0/src/github.com/heketi/heketi/apps/glusterfs/app.go:154 +0xc0e
    main.main()
            /builddir/build/BUILD/heketi-6.0.0/main.go:273 +0x46d

Comment 6 Rachael 2018-02-28 14:30:42 UTC

Without a new build with fix for this issue, the negative test cases cannot be carried out.

Comment 8 Raghavendra Talur 2018-03-05 09:21:05 UTC


*** This bug has been marked as a duplicate of bug 1548322 ***