Red Hat Bugzilla – Bug 1432340
heketi-cli doesn't output the reason of error
Last modified: 2018-09-12 05:23:26 EDT
In some condition, heketi-cli doesn't answer the error to the client. Please see the output of error: [root@knakayam-ose34-master1 ~]# heketi-cli topology info Error: [root@knakayam-ose34-master1 ~]# heketi-cli volume list Error: [root@knakayam-ose34-master1 ~]# heketi-cli cluster list Error: Expected results: Whenever heketi-cli returns error, please output some clue for the error.
For example (this is just one example), if server returns 404, heketi-cli didn't output any information about the error. [root@knakayam-ose34-master1 ~]# heketi-cli topology info --server=http://access.redhat.com Error: [root@knakayam-ose34-master1 ~]#
Can you please share the env variable value of HEKETI_CLI_SERVER ?
I'm sorry, but why do you need to know it? I have already debugged the reason why I got the "Error:". What I wanted to ask you is that improvement of the error message. For example, if 404 was returned as c#2, please output the "404 error".
(In reply to Kenjiro Nakayama from comment #4) > I'm sorry, but why do you need to know it? I have already debugged the > reason why I got the "Error:". What I wanted to ask you is that improvement > of the error message. For example, if 404 was returned as c#2, please output > the "404 error". Can you please let me know which version of Heketi is in use here ? If you are not using heekti version 4, can you reproduce the same with v4?
heketi version 4 is provided via RH repository? I think I am using the latest package. # rpm -qf `which heketi-cli` heketi-client-3.1.0-14.el7rhgs.x86_64 # heketi-cli -v heketi-cli 3.1.0
(In reply to Kenjiro Nakayama from comment #6) > heketi version 4 is provided via RH repository? I think I am using the > latest package. > > # rpm -qf `which heketi-cli` > heketi-client-3.1.0-14.el7rhgs.x86_64 > > # heketi-cli -v > heketi-cli 3.1.0 Its available in RH repository.Please retest with the same.
That's QE/QA's work.
I'm seeing this issue also. Let me know if you need any additional info: root@10.19.231.221 ~ $ echo $HEKETI_CLI_SERVER http://heketi-cns0.cloudapps.mystorage.com root@10.19.231.221 ~ $ heketi-cli volume list Error: root@10.19.231.221 ~ $ heketi-cli -v heketi-cli 4.0.0
Any update for this issue? I'm seeing this happen after certain CNS node fail/restore tests and it's blocking me from getting volume status, etc. I can provide my config info if needed.
(In reply to Mark LaBonte from comment #11) > Any update for this issue? > > I'm seeing this happen after certain CNS node fail/restore tests and it's > blocking me from getting volume status, etc. > > I can provide my config info if needed. Thanks Mark! Actually the communication is from heketi-cli to heketi service. The client has to get return from heketi server on the queries. What we normally do to make sure the server give response is via 'curl' command. For eg# [root@client] curl http://deploy-heketi-storage-project.cloudapps.mystorage.com/hello Hello from Heketi Reference #https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html-single/container-native_storage_for_openshift_container_platform/#idm140543670015568 So, for your specifc setup, can you issue curl command against your heketi service and give a check ?
Thanks Humble - the root cause in my case is a failed heketi-db volume. Regarding the empty error, the issue exists in heketi-cli 4.0.0 - what release is the fix targeted for?
(In reply to Mark LaBonte from comment #13) > Thanks Humble - the root cause in my case is a failed heketi-db volume. Does that mean, heketi service was not running properly? > > Regarding the empty error, the issue exists in heketi-cli 4.0.0 - what > release is the fix targeted for? Mark, I am not completely sure on to what extent this has been fixed. However I recollected some enhancements in this area as you can see here https://bugzilla.redhat.com/show_bug.cgi?id=1351646 and was asking to test latest version of heketi. I still have to check on what are covered and what is not.
Humble, Yes, the service was down but we should really be returning something other than an empty error here.
This seems to need small fixes in the heketi-client side in order to prevent an empty "Error: " message. It is possible that a router/proxy returns an HTTP error status (like 404), but without contents (error message) in the body. From client/api/go-client/volume.go: 132 func (c *Client) VolumeList() (*api.VolumeListResponse, error) { 133 134 // Create request 135 req, err := http.NewRequest("GET", c.host+"/volumes", nil) 136 if err != nil { 137 return nil, err 138 } The err variable can get set to an empty string (""), which in turn gets displayed as an error message through heketi-cli. We'll need to review all http.NewRequest() calls and add a more descriptive error message in case none is set.
https://github.com/heketi/heketi/commit/ed3c92e4d635c4b644864344471a0e197f7e4804 has been merged upstream.
The following commands were run to reproduce the issue: [root@dhcp46-144 ~]# heketi-cli topology info --server=http://abc.com Error: server did not provide a message (status 404: Not Found) [root@dhcp46-144 ~]# heketi-cli volume list --server=http://example.com Error: server did not provide a message (status 404: Not Found) [root@dhcp46-144 ~]# heketi-cli cluster list --server=http://example.com Error: server did not provide a message (status 404: Not Found) heketidbstorage volume was stopped and the same commands were run: [root@dhcp46-144 ~]# heketi-cli volume list Error: Get http://172.31.217.233:8080/volumes: dial tcp 172.31.217.233:8080: getsockopt: connection refused [root@dhcp46-144 ~]# heketi-cli cluster list Error: Get http://172.31.217.233:8080/clusters: dial tcp 172.31.217.233:8080: getsockopt: connection refused [root@dhcp46-144 ~]# heketi-cli topology info Error: Get http://172.31.217.233:8080/clusters: dial tcp 172.31.217.233:8080: getsockopt: connection refused Since no empty error messages were seen, moving this bug to verified.
Updated doc text in the Doc Text field. Please review for technical accuracy.
Thanks John , I have updated the doc text based on your feed back
Doc Text looks OK
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2686