Description of problem: On a 6 node CNS setup with 2 TSPs with 90+ volumes on each TSP, lot of error messages are seen on heketi logs as shown below. [heketi] ERROR 2017/03/09 11:11:57 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:06 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:07 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:17 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:17 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:26 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=19, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:28 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=9, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:37 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:38 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=21, ErrCode=COMPRESSION_ERROR, debug="" [heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug="" Version-Release number of selected component (if applicable): rpm -qa | grep 'heketi' heketi-client-4.0.0-1.el7rhgs.x86_64 How reproducible: Yet to determine Possible steps to Reproduce: 1. create a CNS setup with 2 TSPs and have 90+ volumes on each of the TSPs 2. observe heketi logs Actual results: Following error message is seen. [heketi] ERROR 2017/03/09 11:12:48 /src/github.com/heketi/heketi/apps/glusterfs/app_middleware.go:69: Unable to backup database to kubernetes secret: Unable to update database to secret: client: etcd cluster is unavailable or misconfigured; error #0: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=COMPRESSION_ERROR, debug="" Expected results: No such error message should be seen Additional info:
Karthick, What is the size of the DB now ? It could be the same bug as https://bugzilla.redhat.com/show_bug.cgi?id=1430704
(In reply to Humble Chirammal from comment #2) > Karthick, What is the size of the DB now ? It could be the same bug as > https://bugzilla.redhat.com/show_bug.cgi?id=1430704 [root@dhcp46-202 claim_files]# oc describe secrets/heketi-db-backup Name: heketi-db-backup Namespace: storage-project Labels: <none> Annotations: <none> Type: Opaque Data ==== heketi.db: 774144 bytes Looks like the size is less than 1 MB.
We have to make the backup to secret configurable from heketi. As the secret can hold only 1MB. heketi db in my test environment hit 1MB with 1 TSP with 3 nodes and 4 devices each, having 130 replica 3 volume crossed 1MB. So It is better to follow heketidbvolume model. Well make that changes in heketi.
These error messages are not seen anymore with the latest builds of heketi (heketi-4.0.0-3.el7rhgs and above). Moving the bug to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1111