Bug 1430786

Summary:	Error messages seen on heketi logs complaining 'Unable to update database to secret'
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	krishnaram Karthick <kramdoss>
Component:	heketi	Assignee:	Mohamed Ashiq <mliyazud>
Status:	CLOSED ERRATA	QA Contact:	krishnaram Karthick <kramdoss>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	cns-3.5	CC:	hchiramm, kramdoss, madam, mliyazud, pprakash, rhs-bugs, rtalur, storage-qa-internal, vinug
Target Milestone:	---
Target Release:	CNS 3.5
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	heketi-4.0.0-2.el7rhgs	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-04-20 18:38:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1415600

Description krishnaram Karthick 2017-03-09 15:28:24 UTC

Description of problem:
On a 6 node CNS setup with 2 TSPs with 90+ volumes on each TSP, lot of error messages are seen on heketi logs as shown below.

[heketi] ERROR 2017/03/09 11:11:57 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:06 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:07 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:17 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:17 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:26 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=19, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:28 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=9, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:37 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:38 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=21, ErrCode=COMPRESSION_ERROR, debug=""
[heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""

Version-Release number of selected component (if applicable):
rpm -qa | grep 'heketi'
heketi-client-4.0.0-1.el7rhgs.x86_64


How reproducible:
Yet to determine

Possible steps to Reproduce:
1. create a CNS setup with 2 TSPs and have 90+ volumes on each of the TSPs
2. observe heketi logs


Actual results:

Following error message is seen.

[heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug=""

Expected results:
No such error message should be seen

Additional info:

Comment 2 Humble Chirammal 2017-03-09 15:36:54 UTC

Karthick, What is the size of the DB now ? It could be the same bug as https://bugzilla.redhat.com/show_bug.cgi?id=1430704

Comment 3 krishnaram Karthick 2017-03-09 15:44:26 UTC

(In reply to Humble Chirammal from comment #2)
> Karthick, What is the size of the DB now ? It could be the same bug as
> https://bugzilla.redhat.com/show_bug.cgi?id=1430704

[root@dhcp46-202 claim_files]# oc describe secrets/heketi-db-backup
Name:           heketi-db-backup
Namespace:      storage-project
Labels:         <none>
Annotations:    <none>

Type:   Opaque

Data
====
heketi.db:      774144 bytes

Looks like the size is less than 1 MB.

Comment 5 Mohamed Ashiq 2017-03-13 09:48:22 UTC

We have to make the backup to secret configurable from heketi. As the secret can hold only 1MB. heketi db in my test environment hit 1MB with 1 TSP with 3 nodes and 4 devices each, having 130 replica 3 volume crossed 1MB. So It is better to follow heketidbvolume model.

Well make that changes in heketi.

Comment 10 krishnaram Karthick 2017-04-10 07:44:05 UTC

These error messages are not seen anymore with the latest builds of heketi (heketi-4.0.0-3.el7rhgs and above).

Moving the bug to verified.

Comment 12 errata-xmlrpc 2017-04-20 18:38:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1111