Bug 1434838
Summary: | heketi-cli command returns error post cns deployment | ||
---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Prasanth <pprakash> |
Component: | cns-deploy-tool | Assignee: | Michael Adam <madam> |
Status: | CLOSED ERRATA | QA Contact: | Prasanth <pprakash> |
Severity: | urgent | Docs Contact: | |
Priority: | unspecified | ||
Version: | cns-3.4 | CC: | akhakhar, hchiramm, jarrpa, madam, mliyazud, pprakash, rcyriac, rhs-bugs, rtalur, sankarshan, storage-qa-internal, vinug |
Target Milestone: | --- | ||
Target Release: | CNS 3.5 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | cns-deploy-4.0.0-13.el7rhgs | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2017-04-20 18:28:47 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1415600 |
Description
Prasanth
2017-03-22 13:04:25 UTC
Thanks for sharing the machine. DB was clean. Some how it was not loaded properly, Will try to reproduce the issue from my end. We had a small debugging session in our scrum and could not find the root cause and how ever restarting of heketi pod helped. Will try again so that we can be sure it was something spurious and can be neglected. or handle it from the heketi volmanager script if we could find exact case of this issue being hit. trying more setups will update If hit again. I have built a volmanager scratch build : brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-volmanager-rhel7:rhgs-3.2-rhel-7-docker-candidate-20170323042518 This image will log the presence of the db and the log can be verified with : # oc rsh <heketi-pod-name> cat /var/lib/heketi/dbstat.log 1) Please confirm that log says the DB is not present 2) check if the db is present in location /var/lib/heketi in the heketi pod. 3) if present, size of the DB is above 32K(empty db size). If all the above is confirmed Then we just need a delay. (In reply to Mohamed Ashiq from comment #8) > I have built a volmanager scratch build : > brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-volmanager- > rhel7:rhgs-3.2-rhel-7-docker-candidate-20170323042518 > > This image will log the presence of the db and the log can be verified with : > > # oc rsh <heketi-pod-name> cat /var/lib/heketi/dbstat.log > > 1) Please confirm that log says the DB is not present > 2) check if the db is present in location /var/lib/heketi in the heketi pod. > 3) if present, size of the DB is above 32K(empty db size). > > If all the above is confirmed Then we just need a delay. Ashiq, I've used the above volmanager scratch build and did a manual deployment for collecting the requested details. Please find the results below: ######### [root@dhcp46-145 templates]# heketi-cli cluster list Error: [root@dhcp46-145 ]# oc get pods NAME READY STATUS RESTARTS AGE glusterfs-44nlc 1/1 Running 0 4m glusterfs-jr0f0 1/1 Running 0 4m glusterfs-xp20r 1/1 Running 0 4m heketi-1-lhqfh 1/1 Running 0 23s storage-project-router-1-r7bwz 1/1 Running 1 58m [root@dhcp46-145 ]# oc rsh heketi-1-lhqfh cat /var/lib/heketi/dbstat.log File: '/var/lib/heketi/heketi.db' Size: 53248 Blocks: 104 IO Block: 131072 regular file Device: 59h/89d Inode: 10471067870456742553 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-03-24 11:57:25.027425828 +0000 Modify: 2017-03-24 11:57:25.032428014 +0000 Change: 2017-03-24 11:57:25.034428889 +0000 Birth: - [root@dhcp46-145 ]# oc rsh heketi-1-lhqfh ls -al /var/lib/heketi/heketi.db -rw-r--r--. 1 root root 53248 Mar 24 11:57 /var/lib/heketi/heketi.db [root@dhcp46-145 ]# oc rsh heketi-1-lhqfh du -sch /var/lib/heketi/heketi.db 52K /var/lib/heketi/heketi.db 52K total ######### Let me know if you want me to collect and attach any additional details from the setup for your further debugging. If required, I can also share the setup for sometime, which is in the same state, for your debugging. Tested this with the latest image rhgs-volmanager-docker-3.2.0-4 and I still see this issue. See below: ############## [root@dhcp46-195 opt]# export HEKETI_CLI_SERVER=http://heketi-storage-project.cloudapps.mystorage.com [root@dhcp46-195 opt]# heketi-cli cluster list Clusters: [root@dhcp46-195 opt]# heketi-cli volume list [root@dhcp46-195 opt]# date Tue Mar 28 15:56:18 IST 2017 [root@dhcp46-195 opt]# curl http://heketi-storage-project.cloudapps.mystorage.com/hello Hello from Heketi[root@dhcp46-195 opt]# [root@dhcp46-195 opt]# [root@dhcp46-195 opt]# heketi-cli volume list Error: ############## Moving it back to Assigned for further investigation and fix. Prasanth, Please give this scratch build a try. brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhgs3/rhgs-volmanager-rhel7:rhgs-3.2-rhel-7-docker-candidate-20170328150404 This will not let the pod to start of the db is not found and the volume is mounted. Verified Patches posted upstream at https://github.com/gluster/gluster-kubernetes/pull/241 https://github.com/gluster/gluster-kubernetes/pull/245 Verified as fixed in the latest build: cns-deploy-4.0.0-13.el7rhgs. Marking it as verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1112 |