Bug 1549140 - heketi-pod fails to come back, after we kill and restart the heketi pod
Summary: heketi-pod fails to come back, after we kill and restart the heketi pod
Keywords:
Status: CLOSED DUPLICATE of bug 1548322
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: heketi
Version: cns-3.9
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Michael Adam
QA Contact: Apeksha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-26 13:55 UTC by Apeksha
Modified: 2018-03-08 12:26 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-05 09:21:05 UTC
Embargoed:

Attachments (Terms of Use)

Description Apeksha 2018-02-26 13:55:15 UTC 
Description of problem:
heketi-pod fails to come back, after we kill and restart the heketi pod

Version-Release number of selected component (if applicable):
heketi-client-6.0.0-3.el7rhgs.x86_64
cns-deploy-6.0.0-3.el7rhgs.x86_64
atomic-openshift-3.9.0-0.47.0.git.0.965194d.el7.x86_64
openshift-ansible-3.9.0-0.47.0.git.0.f8847bb.el7.noarch

How reproducible: Once


Steps to Reproduce:
Run the automated testcase for heketi-pod failure, test fails.

1. have the setup up and running, 1 app-pod created.
2. Now delete the heketi pod
   command: oc delete dc,service,route heketi
3. now cretate a pvc it goes to pending state
4. Recreate heketi process
5. heketi pod fails to come back.

[root@dhcp46-24 ~]# oc get pods
NAME                                  READY     STATUS             RESTARTS   AGE
glusterblock-provisioner-dc-1-c5hcl   1/1       Running            0          3d
glusterfs-mgp2k                       1/1       Running            0          3d
glusterfs-tt5rs                       1/1       Running            1          3d
glusterfs-wbgzs                       1/1       Running            1          3d
heketi-1-tblcg                        0/1       CrashLoopBackOff   66         5h
storage-project-router-1-z5tg8        1/1       Running            0          3d


Actual results: heketi pod fails to come back.

Expected results: heketi pod should come back and pvc should be in bound state.
Comment 2 Apeksha 2018-02-26 13:57:23 UTC 
Additional info:

    [root@dhcp46-24 ~]# oc logs heketi-1-tblcg
    Heketi 6.0.0
    [heketi] INFO 2018/02/26 08:36:07 Loaded kubernetes executor
    [heketi] INFO 2018/02/26 08:36:07 Please refer to the Heketi troubleshooting documentation for more information on how to resolve this issue.
    [heketi] ERROR 2018/02/26 08:36:07 /src/github.com/heketi/heketi/apps/glusterfs/app.go:150: Heketi terminated while performing one or more operations. Server will not start as long as pending operations are present in the db.
    panic: Heketi terminated while performing one or more operations. Server will not start as long as pending operations are present in the db.
     
    goroutine 1 [running]:
    github.com/heketi/heketi/apps/glusterfs.NewApp(0x236bf40, 0xc42000e198, 0x0)
            /builddir/build/BUILD/heketi-6.0.0/src/github.com/heketi/heketi/apps/glusterfs/app.go:154 +0xc0e
    main.main()
            /builddir/build/BUILD/heketi-6.0.0/main.go:273 +0x46d
Comment 6 Rachael 2018-02-28 14:30:42 UTC 
Without a new build with fix for this issue, the negative test cases cannot be carried out.
Comment 8 Raghavendra Talur 2018-03-05 09:21:05 UTC 

*** This bug has been marked as a duplicate of bug 1548322 ***
Note You need to log in before you can comment on or make changes to this bug.