Bug 1430786 - Error messages seen on heketi logs complaining 'Unable to update database to secret'
Summary: Error messages seen on heketi logs complaining 'Unable to update database to ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: heketi
Version: cns-3.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: CNS 3.5
Assignee: Mohamed Ashiq
QA Contact: krishnaram Karthick
URL:
Whiteboard:
Depends On:
Blocks: 1415600
TreeView+ depends on / blocked
 
Reported: 2017-03-09 15:28 UTC by krishnaram Karthick
Modified: 2019-02-13 09:18 UTC (History)
9 users (show)

Fixed In Version: heketi-4.0.0-2.el7rhgs
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-20 18:38:07 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1111 0 normal SHIPPED_LIVE heketi bug fix and enhancement update 2017-04-20 22:37:02 UTC

Description krishnaram Karthick 2017-03-09 15:28:24 UTC
Description of problem:
On a 6 node CNS setup with 2 TSPs with 90+ volumes on each TSP, lot of error messages are seen on heketi logs as shown below.

[heketi] ERROR 2017/03/09 11:11:57 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:06 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:07 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:17 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:17 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:26 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=19, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:28 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=9, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:37 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:38 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=21, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""

Version-Release number of selected component (if applicable):
rpm -qa | grep 'heketi'
heketi-client-4.0.0-1.el7rhgs.x86_64


How reproducible:
Yet to determine

Possible steps to Reproduce:
1. create a CNS setup with 2 TSPs and have 90+ volumes on each of the TSPs
2. observe heketi logs


Actual results:

Following error message is seen.

[heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""

Expected results:
No such error message should be seen

Additional info:

Comment 2 Humble Chirammal 2017-03-09 15:36:54 UTC
Karthick, What is the size of the DB now ? It could be the same bug as https://bugzilla.redhat.com/show_bug.cgi?id=1430704

Comment 3 krishnaram Karthick 2017-03-09 15:44:26 UTC
(In reply to Humble Chirammal from comment #2)
> Karthick, What is the size of the DB now ? It could be the same bug as
> https://bugzilla.redhat.com/show_bug.cgi?id=1430704

[root@dhcp46-202 claim_files]# oc describe secrets/heketi-db-backup
Name:           heketi-db-backup
Namespace:      storage-project
Labels:         <none>
Annotations:    <none>

Type:   Opaque

Data
====
heketi.db:      774144 bytes

Looks like the size is less than 1 MB.

Comment 5 Mohamed Ashiq 2017-03-13 09:48:22 UTC
We have to make the backup to secret configurable from heketi. As the secret can hold only 1MB. heketi db in my test environment hit 1MB with 1 TSP with 3 nodes and 4 devices each, having 130 replica 3 volume crossed 1MB. So It is better to follow heketidbvolume model.

Well make that changes in heketi.

Comment 10 krishnaram Karthick 2017-04-10 07:44:05 UTC
These error messages are not seen anymore with the latest builds of heketi (heketi-4.0.0-3.el7rhgs and above).

Moving the bug to verified.

Comment 12 errata-xmlrpc 2017-04-20 18:38:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1111


Note You need to log in before you can comment on or make changes to this bug.