Bug 1633930

Summary: ASan (address sanitizer) fixes - Blanket bug
Product: [Community] GlusterFS Reporter: Amar Tumballi <atumball>
Component: coreAssignee: bugs <bugs>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: urgent    
Version: mainlineCC: amukherj, bugs, srangana
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1635373 (view as bug list) Environment:
Last Closed: 2019-03-25 16:30:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1635373    

Description Amar Tumballi 2018-09-28 06:57:27 UTC 
Description of problem:

As of now, glusterfs has an option to have '--enable-asan' during configure, which can be used to run the system with stricter memory check flags. There is a nightly job which runs with this flag @ https://build.gluster.org/job/asan/

Let us capture all the fixes which get in to the codebase as part of fixing ASan errors, using this bug!

Version-Release number of selected component (if applicable):
mainline

How reproducible:
Currently 100%


Steps to Reproduce:
1. Run any regression tests after doing --enable-asan option!
Comment 1 Amar Tumballi 2018-09-28 06:58:56 UTC 
Marking priority as 'Urgent' and severity as 'High' for this, as any issue reported in asan job should be treated with highest focus, as in most of the cases, it would mean there is a memory corruption/leak, which is critical for stability of the product!
Comment 2 Worker Ant 2018-09-28 17:51:09 UTC 
REVIEW: https://review.gluster.org/21299 (mdcache: Fix asan reported potential heap buffer overflow) posted (#1) for review on master by Shyamsundar Ranganathan
Comment 3 Worker Ant 2018-09-28 17:59:39 UTC 
REVIEW: https://review.gluster.org/21300 (configure: add more options to asan section) posted (#1) for review on master by Amar Tumballi
Comment 4 Worker Ant 2018-09-28 18:00:57 UTC 
REVIEW: https://review.gluster.org/21301 (cli: fix the clang warnings) posted (#1) for review on master by Amar Tumballi
Comment 5 Worker Ant 2018-09-29 09:35:39 UTC 
COMMIT: https://review.gluster.org/21299 committed in master by "Shyamsundar Ranganathan" <srangana> with a commit message- mdcache: Fix asan reported potential heap buffer overflow

The char pointer mdc_xattr_str in function mdc_xattr_list_populate
is malloc'd and doing a strcat into a malloc'd region can
overflow content allocated based on prior contents of the
memory region.

Added a NULL terimation to the malloc'd region to prevent
the overflow, and treat it as an empty string.

Change-Id: If0decab669551581230a8ede4c44c319ff04bac9
Updates: bz#1633930
Signed-off-by: ShyamsundarR <srangana>
Comment 6 Worker Ant 2018-10-09 19:17:54 UTC 
COMMIT: https://review.gluster.org/21301 committed in master by "Shyamsundar Ranganathan" <srangana> with a commit message- cli: memory leak issues reported by asan

With this fix, a run on 'rpc-coverage.t' passes properly.
This should help to get started with other fixes soon!

Change-Id: I257ae4e28b9974998a451d3b490cc18c02650ba2
updates: bz#1633930
Signed-off-by: Amar Tumballi <amarts>
Comment 7 Worker Ant 2018-10-10 04:25:49 UTC 
REVIEW: https://review.gluster.org/21378 (glusterfsd: fix the asan leak message) posted (#1) for review on master by Amar Tumballi
Comment 8 Worker Ant 2018-10-10 08:39:05 UTC 
REVIEW: https://review.gluster.org/21383 (cli: Fix memory leaks reported by asan) posted (#1) for review on master by Kotresh HR
Comment 9 Worker Ant 2018-10-12 05:40:47 UTC 
COMMIT: https://review.gluster.org/21383 committed in master by "Amar Tumballi" <amarts> with a commit message- cli: Fix memory leaks reported by ASAN

'gf_cli_rsp' structure has 'op_errstr' and 'dict.dict_val'
which could get allocated by xdr and should be freed. This
patch takes care of all instances of the same.

Updates: bz#1633930
Change-Id: I2a5d129ffe85cfca743c89edb45b658f3ce688cb
Signed-off-by: Kotresh HR <khiremat>
Comment 10 Worker Ant 2018-10-12 07:05:50 UTC 
REVIEW: https://review.gluster.org/21397 (cli: Fix memory leaks reported by ASAN) posted (#1) for review on master by Kotresh HR
Comment 11 Worker Ant 2018-10-12 09:06:47 UTC 
REVIEW: https://review.gluster.org/21399 (glusterd/geo-rep: Fix memory corruption) posted (#1) for review on master by Kotresh HR
Comment 12 Worker Ant 2018-10-12 09:26:31 UTC 
REVIEW: https://review.gluster.org/21400 (run-tests.sh: add another option to have failure count configurable) posted (#1) for review on master by Amar Tumballi
Comment 13 Worker Ant 2018-10-12 15:39:40 UTC 
COMMIT: https://review.gluster.org/21397 committed in master by "Amar Tumballi" <amarts> with a commit message- cli: Fix memory leaks reported by ASAN

This patch addresses memory leaks other than
'gf_cli_rsp' response structure.

Updates: bz#1633930
Change-Id: Idc5940dca623e33bc01004a9324bc435fc60b006
Signed-off-by: Kotresh HR <khiremat>
Comment 14 Worker Ant 2018-10-12 15:40:02 UTC 
COMMIT: https://review.gluster.org/21399 committed in master by "Amar Tumballi" <amarts> with a commit message- glusterd/geo-rep: Fix memory corruption

Problem and Analysis:
The length of canonical format of uuid is 36 but
'GF_UUID_BUF_SIZE 50' was being used everywhere.
glusterd/geo-rep code was earlier using strncpy,
but recently changes to memcpy with the drive
to avoid strncpys. This leads to memory corruption
and glusterd is crashing without a core with geo-rep
creation with ASAN build.

Fix:
'GF_UUID_BUF_SIZE 37' (+ 1 for NULL byte)
And change geo-rep to use UUID_CANONICAL_FORM_LEN
instead

Updates: bz#1633930
Change-Id: Ibd347d542b92e64a96ce06780cda643557233bc7
Signed-off-by: Kotresh HR <khiremat>
Comment 15 Worker Ant 2018-10-15 06:11:26 UTC 
REVIEW: https://review.gluster.org/21406 (cli: Fix heap-buffer-overflow issue reported by ASAN) posted (#1) for review on master by Kotresh HR
Comment 16 Worker Ant 2018-10-15 07:19:50 UTC 
REVIEW: https://review.gluster.org/21407 (cli: memory leak issue reported by asan) posted (#1) for review on master by Harpreet Kaur Lalwani
Comment 17 Worker Ant 2018-10-15 07:33:48 UTC 
REVIEW: https://review.gluster.org/21408 (libglusterfs: memory leak issue reported by asan) posted (#1) for review on master by Iraj Jamali
Comment 18 Worker Ant 2018-10-16 12:00:18 UTC 
COMMIT: https://review.gluster.org/21378 committed in master by "Amar Tumballi" <amarts> with a commit message- glusterfsd: fix the asan leak message

Fixes below trace of ASan:

Direct leak of 130 byte(s) in 1 object(s) allocated from:
    #0 0x7fa794bb5850 in malloc (/lib64/libasan.so.4+0xde850)
    #1 0x7fa7944e5de9 in __gf_malloc ../../../libglusterfs/src/mem-pool.c:136
    #2 0x40b85c in gf_strndup ../../../libglusterfs/src/mem-pool.h:166
    #3 0x40b85c in gf_strdup ../../../libglusterfs/src/mem-pool.h:183
    #4 0x40b85c in parse_opts ../../../glusterfsd/src/glusterfsd.c:1049
    #5 0x7fa792a98720 in argp_parse (/lib64/libc.so.6+0x101720)
    #6 0x40d89f in parse_cmdline ../../../glusterfsd/src/glusterfsd.c:2041
    #7 0x406d07 in main ../../../glusterfsd/src/glusterfsd.c:2625


updates: bz#1633930
Change-Id: I394b3fc24b7a994c1b03635cb5e973e7290491d3
Signed-off-by: Amar Tumballi <amarts>
Comment 19 Worker Ant 2018-10-24 05:06:59 UTC 
COMMIT: https://review.gluster.org/21406 committed in master by "Amar Tumballi" <amarts> with a commit message- cli: Fix heap-buffer-overflow issue reported by ASAN

GF_MALLOC was being used to allocate memory which is
not initialized. strcat is used on it which could
result in buffer overflow if it contains garbage before
'\0'. So changed it to GF_CALLOC.

Traceback:

==23427==ERROR: AddressSanitizer: heap-buffer-overflow ...
WRITE of size 5 at 0x6080000083fe thread T3
 #0 0x7fb60966991c in __interceptor_strcat ...
 #1 0x48adc0 in config_parse ...
 #2 0x48cde8 in cli_cmd_gsync_set_parse ...
...

Updates: bz#1633930
Change-Id: I3710f011d8139984b1898265d84d150c9bdc962b
Signed-off-by: Kotresh HR <khiremat>
Comment 20 Amar Tumballi 2018-11-21 09:18:12 UTC 
Need Fix:
-----
==24676==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 256 byte(s) in 1 object(s) allocated from:
    #0 0x7fdac7d25848 in __interceptor_malloc (/lib64/libasan.so.5+0xef848)
    #1 0x7fdac79dfbba in mem_get_from_pool ../../../libglusterfs/src/mem-pool.c:817
    #2 0x7fdac79dfd19 in mem_get ../../../libglusterfs/src/mem-pool.c:843
    #3 0x7fdac79df601 in mem_get0 ../../../libglusterfs/src/mem-pool.c:742
    #4 0x7fdac7927755 in get_new_dict_full ../../../libglusterfs/src/dict.c:75
    #5 0x7fdac792798c in dict_new ../../../libglusterfs/src/dict.c:124
    #6 0x7fdab352cbe5 in changelog_rpc_server_init ../../../../../xlators/features/changelog/src/changelog-rpc-common.c:302
    #7 0x7fdab3529a44 in changelog_init_rpc_listener ../../../../../xlators/features/changelog/src/changelog-rpc.c:210
    #8 0x7fdab35190b7 in changelog_init_rpc ../../../../../xlators/features/changelog/src/changelog.c:2657
    #9 0x7fdab3519a36 in init ../../../../../xlators/features/changelog/src/changelog.c:2734
    #10 0x7fdac793feb3 in __xlator_init ../../../libglusterfs/src/xlator.c:718
    #11 0x7fdac7940167 in xlator_init ../../../libglusterfs/src/xlator.c:745
    #12 0x7fdac7a071ca in glusterfs_graph_init ../../../libglusterfs/src/graph.c:359
    #13 0x7fdac7a089b0 in glusterfs_graph_activate ../../../libglusterfs/src/graph.c:722
    #14 0x412a7d in glusterfs_process_volfp ../../../glusterfsd/src/glusterfsd.c:2592
    #15 0x420cff in mgmt_getspec_cbk ../../../glusterfsd/src/glusterfsd-mgmt.c:2085
    #16 0x7fdac788928a in rpc_clnt_handle_reply ../../../../rpc/rpc-lib/src/rpc-clnt.c:755
    #17 0x7fdac7889c79 in rpc_clnt_notify ../../../../rpc/rpc-lib/src/rpc-clnt.c:922
    #18 0x7fdac787faa2 in rpc_transport_notify ../../../../rpc/rpc-lib/src/rpc-transport.c:525
    #19 0x7fdac45aedf2 in socket_event_poll_in ../../../../../rpc/rpc-transport/socket/src/socket.c:2530
    #20 0x7fdac45b13cc in socket_event_handler ../../../../../rpc/rpc-transport/socket/src/socket.c:2931
    #21 0x7fdac7a7f6a3 in event_dispatch_epoll_handler ../../../libglusterfs/src/event-epoll.c:591
    #22 0x7fdac7a7fd60 in event_dispatch_epoll_worker ../../../libglusterfs/src/event-epoll.c:668
    #23 0x7fdac762458d in start_thread (/lib64/libpthread.so.0+0x858d)
---
Direct leak of 256 byte(s) in 1 object(s) allocated from:
    #0 0x7fa044c06848 in __interceptor_malloc (/lib64/libasan.so.5+0xef848)
    #1 0x7fa0448c0bba in mem_get_from_pool ../../../libglusterfs/src/mem-pool.c:817
    #2 0x7fa0448c0d19 in mem_get ../../../libglusterfs/src/mem-pool.c:843
    #3 0x7fa0448c0601 in mem_get0 ../../../libglusterfs/src/mem-pool.c:742
    #4 0x7fa044826c7e in log_buf_new ../../../libglusterfs/src/logging.c:304
    #5 0x7fa04482fc20 in _gf_msg_internal ../../../libglusterfs/src/logging.c:2014
    #6 0x7fa044830606 in _gf_msg ../../../libglusterfs/src/logging.c:2124
    #7 0x7fa0309ec040 in init ../../../../../xlators/mgmt/glusterd/src/glusterd.c:1484
    #8 0x7fa044820eb3 in __xlator_init ../../../libglusterfs/src/xlator.c:718
    #9 0x7fa044821167 in xlator_init ../../../libglusterfs/src/xlator.c:745
    #10 0x7fa0448e81ca in glusterfs_graph_init ../../../libglusterfs/src/graph.c:359
    #11 0x7fa0448e99b0 in glusterfs_graph_activate ../../../libglusterfs/src/graph.c:722
    #12 0x412a7d in glusterfs_process_volfp ../../../glusterfsd/src/glusterfsd.c:2592
    #13 0x412c52 in glusterfs_volumes_init ../../../glusterfsd/src/glusterfsd.c:2645
    #14 0x413553 in main ../../../glusterfsd/src/glusterfsd.c:2798
    #15 0x7fa04407f412 in __libc_start_main (/lib64/libc.so.6+0x24412)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x7fa044c06848 in __interceptor_malloc (/lib64/libasan.so.5+0xef848)
    #1 0x7fa03047eef7  (<unknown module>)
----
Direct leak of 624 byte(s) in 1 object(s) allocated from:
    #0 0x7ff924fc5a50 in __interceptor_calloc (/lib64/libasan.so.5+0xefa50)
    #1 0x7ff924ce2f0f in __gf_calloc ../../../libglusterfs/src/mem-pool.c:111
    #2 0x7ff921b8f6b4 in fuse_thread_proc ../../../../../xlators/mount/fuse/src/fuse-bridge.c:5639
    #3 0x7ff92498a58d in start_thread (/lib64/libpthread.so.0+0x858d)

----
Comment 21 Worker Ant 2018-11-25 13:25:21 UTC 
REVIEW: https://review.gluster.org/21716 (posix: fix memory leak) posted (#1) for review on master by Amar Tumballi
Comment 22 Worker Ant 2018-11-28 03:30:12 UTC 
REVIEW: https://review.gluster.org/21716 (posix: fix memory leak) posted (#2) for review on master by Amar Tumballi
Comment 23 Worker Ant 2018-11-30 11:28:01 UTC 
REVIEW: https://review.gluster.org/21764 (cli : Memeory leak fix reported by ASAN) posted (#1) for review on master by Sunny Kumar
Comment 24 Worker Ant 2018-11-30 12:21:47 UTC 
REVIEW: https://review.gluster.org/21766 (cli : Memeory leak fix reported by ASAN) posted (#1) for review on master by Sunny Kumar
Comment 25 Worker Ant 2018-12-02 07:24:13 UTC 
REVIEW: https://review.gluster.org/21764 (cli : Memeory leak fix reported by ASAN) posted (#2) for review on master by Atin Mukherjee
Comment 26 Worker Ant 2018-12-03 05:49:04 UTC 
REVIEW: https://review.gluster.org/21407 (cli: memory leak issue reported by asan) posted (#3) for review on master by Amar Tumballi
Comment 27 Worker Ant 2018-12-03 15:58:14 UTC 
REVIEW: https://review.gluster.org/21766 (cli : Memeory leak fix reported by ASAN) posted (#3) for review on master by Atin Mukherjee
Comment 28 Worker Ant 2018-12-03 18:54:38 UTC 
REVIEW: https://review.gluster.org/21788 (cli: fix memory leak in cli-cmd-system.c) posted (#1) for review on master by Sunny Kumar
Comment 29 Worker Ant 2018-12-04 10:04:15 UTC 
REVIEW: https://review.gluster.org/21793 (cli: memory leak issue reported by asan) posted (#1) for review on master by Harpreet Kaur Lalwani
Comment 30 Worker Ant 2018-12-04 12:53:47 UTC 
REVIEW: https://review.gluster.org/21796 (cli: memory leak issue reported by asan) posted (#1) for review on master by Harpreet Kaur Lalwani
Comment 31 Worker Ant 2018-12-04 14:21:32 UTC 
REVIEW: https://review.gluster.org/21798 (cli: memory leak issue reported by asan) posted (#1) for review on master by None
Comment 32 Worker Ant 2018-12-05 03:49:47 UTC 
REVIEW: https://review.gluster.org/21788 (cli: fix memory leak in cli-cmd-system.c) posted (#3) for review on master by Amar Tumballi
Comment 33 Worker Ant 2018-12-05 07:18:56 UTC 
REVIEW: https://review.gluster.org/21802 (cli: fix memory leak in cli/src/cli-rpc-ops.c) posted (#1) for review on master by Sunny Kumar
Comment 34 Worker Ant 2018-12-05 13:16:31 UTC 
REVIEW: https://review.gluster.org/21803 (cli: Fix mem-leaks reported by ASAN) posted (#1) for review on master by Kotresh HR
Comment 35 Worker Ant 2018-12-05 13:19:12 UTC 
REVIEW: https://review.gluster.org/21804 (cli: Fix memory leaks reported by ASAN) posted (#1) for review on master by Kotresh HR
Comment 36 Worker Ant 2018-12-05 13:21:41 UTC 
REVIEW: https://review.gluster.org/21805 (cluster/afr: Fix mem leak reported by ASAN) posted (#1) for review on master by Kotresh HR
Comment 37 Worker Ant 2018-12-05 13:58:36 UTC 
REVIEW: https://review.gluster.org/21806 (This patch fixes options leak in cli_cmd_volume_remove_brick_cbk. SUMMARY: AddressSanitizer: 1152 byte(s) leaked in 8 allocation(s)) posted (#1) for review on master by None
Comment 38 Worker Ant 2018-12-05 16:03:38 UTC 
REVIEW: https://review.gluster.org/21808 (cli: memory leak issue reported by asan This patch fixes xdata leak in cli_cmd_quota_handle_list_all and options leak in cli_cmd_quota_cbk.) posted (#1) for review on master by None
Comment 39 Worker Ant 2018-12-05 21:35:32 UTC 
REVIEW: https://review.gluster.org/21796 (cli: memory leak issue reported by asan) posted (#2) for review on master by Amar Tumballi
Comment 40 Worker Ant 2018-12-05 21:38:12 UTC 
REVIEW: https://review.gluster.org/21802 (cli: fix memory leak in cli/src/cli-rpc-ops.c) posted (#3) for review on master by Amar Tumballi
Comment 41 Worker Ant 2018-12-05 22:18:47 UTC 
REVIEW: https://review.gluster.org/21798 (cli: memory leak issue reported by asan) posted (#8) for review on master by Amar Tumballi
Comment 42 Worker Ant 2018-12-06 10:28:14 UTC 
REVIEW: https://review.gluster.org/21803 (cli: Fix mem-leaks reported by ASAN) posted (#5) for review on master by Amar Tumballi
Comment 43 Worker Ant 2018-12-06 16:37:21 UTC 
REVIEW: https://review.gluster.org/21806 (cli: fix a memory leak reported by ASan.) posted (#6) for review on master by Rinku Kothiya
Comment 44 Worker Ant 2018-12-06 16:41:30 UTC 
REVIEW: https://review.gluster.org/21818 (cli: fix memory leak in cli rpc ops) posted (#1) for review on master by MOHIT AGRAWAL
Comment 45 Worker Ant 2018-12-07 03:49:02 UTC 
REVIEW: https://review.gluster.org/21818 (cli: fix memory leak in cli rpc ops) posted (#2) for review on master by Atin Mukherjee
Comment 46 Worker Ant 2018-12-07 05:37:43 UTC 
REVIEW: https://review.gluster.org/21821 (cli : fix memory leak in cli-cmd-volume.c) posted (#1) for review on master by Sunny Kumar
Comment 47 Worker Ant 2018-12-07 07:09:52 UTC 
REVIEW: https://review.gluster.org/21823 (glusterd: Resolve memory leak in some gluster functions) posted (#1) for review on master by MOHIT AGRAWAL
Comment 48 Worker Ant 2018-12-11 11:03:31 UTC 
REVIEW: https://review.gluster.org/21823 (glusterd: Resolve memory leak in some glusterd functions) posted (#4) for review on master by Atin Mukherjee
Comment 49 Worker Ant 2018-12-11 11:11:03 UTC 
REVIEW: https://review.gluster.org/21844 (tools: stack-buffer-overflow reported by asan) posted (#1) for review on master by Harpreet Kaur Lalwani
Comment 50 Worker Ant 2018-12-11 11:11:52 UTC 
REVIEW: https://review.gluster.org/21839 (nfs: memory leak issue reported by asan) posted (#2) for review on master by Amar Tumballi
Comment 51 Worker Ant 2018-12-12 04:45:34 UTC 
REVIEW: https://review.gluster.org/21844 (tools: stack-buffer-overflow reported by asan) posted (#2) for review on master by Kotresh HR
Comment 52 Worker Ant 2018-12-17 04:46:45 UTC 
REVIEW: https://review.gluster.org/21805 (cluster/afr: Fix mem leak reported by ASAN) posted (#9) for review on master by Pranith Kumar Karampuri
Comment 53 Worker Ant 2018-12-19 21:28:56 UTC 
REVIEW: https://review.gluster.org/21894 (cli: fix memory leak) posted (#1) for review on master by Sunny Kumar
Comment 54 Worker Ant 2018-12-26 02:17:57 UTC 
REVIEW: https://review.gluster.org/21858 (posix: stack-buffer-overflow reported by asan) posted (#4) for review on master by Amar Tumballi
Comment 55 Worker Ant 2018-12-26 02:18:38 UTC 
REVIEW: https://review.gluster.org/21894 (cli: fix memory leak) posted (#5) for review on master by Amar Tumballi
Comment 56 Worker Ant 2018-12-30 05:54:43 UTC 
REVIEW: https://review.gluster.org/21300 (configure: add a message about asan option) posted (#5) for review on master by Niels de Vos
Comment 57 Worker Ant 2019-01-08 11:38:51 UTC 
REVIEW: https://review.gluster.org/22002 (afr : fix memory leak) posted (#1) for review on master by Sunny Kumar
Comment 58 Worker Ant 2019-01-11 02:57:14 UTC 
REVIEW: https://review.gluster.org/22002 (afr : fix memory leak) posted (#4) for review on master by Pranith Kumar Karampuri
Comment 59 Worker Ant 2019-01-24 08:13:35 UTC 
REVIEW: https://review.gluster.org/22089 (cli : fix mem leak during cli_cmd_volume_gsync_set_cbk) posted (#1) for review on master by Sunny Kumar
Comment 60 Worker Ant 2019-02-19 04:10:27 UTC 
REVIEW: https://review.gluster.org/22089 (cli : fix mem leak during cli_cmd_volume_gsync_set_cbk) merged (#4) on master by Amar Tumballi
Comment 61 Worker Ant 2019-02-20 11:49:24 UTC 
REVIEW: https://review.gluster.org/22239 (fuse : fix memory leak) posted (#1) for review on master by Sunny Kumar
Comment 62 Worker Ant 2019-02-25 07:06:47 UTC 
REVIEW: https://review.gluster.org/22239 (fuse : fix memory leak) merged (#4) on master by Amar Tumballi
Comment 63 Shyamsundar 2019-03-25 16:30:59 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-6.0, please open a new bug report.

glusterfs-6.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2019-March/000120.html
[2] https://www.gluster.org/pipermail/gluster-users/