Less than 24 hours later, a second crash nearly identical with the exception that this time around, the glusterfsd process on both nodes crashed within a few seconds of each other. I had just added four new bricks and was about 8 hours into a rebalance when it crashed:

###########################
# log from first node:
[2019-08-09 17:06:48.926301] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-gv0-server: disconnecting connection from CTX_ID:b36e17ac-62e5-4171-bfdc-1707aa5ec5a9-GRAPH_ID:0-PID:31299-HOST:gluster01-PC_NAME:gv0-client-2-RECON_NO:-0
[2019-08-09 17:06:48.926480] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-gv0-server: Shutting down connection CTX_ID:b36e17ac-62e5-4171-bfdc-1707aa5ec5a9-GRAPH_ID:0-PID:31299-HOST:gluster01-PC_NAME:gv0-client-2-RECON_NO:-0
[2019-08-09 17:06:48.956140] I [dict.c:541:dict_get] (-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f5f80443241] -->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f5f80669219] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f5f900860b4] ) 0-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:49.038307] I [dict.c:541:dict_get] (-->/lib64/libglusterfs.so.0(default_fremovexattr+0xe7) [0x7f5f901251f7] -->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1d5dd) [0x7f5f8066a5dd] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f5f900860b4] ) 0-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:49.099221] I [dict.c:541:dict_get] (-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f5f80443241] -->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f5f80669219] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f5f900860b4] ) 0-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
pending frames:
frame : type(1) op(INODELK)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash: 
2019-08-09 17:06:49
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 6.4
/lib64/libglusterfs.so.0(+0x26e00)[0x7f5f90091e00]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f5f9009c804]
/lib64/libc.so.6(+0x36340)[0x7f5f8e6d1340]
/usr/lib64/glusterfs/6.4/rpc-transport/socket.so(+0xa4cc)[0x7f5f844564cc]
/lib64/libglusterfs.so.0(+0x8b4d6)[0x7f5f900f64d6]
/lib64/libpthread.so.0(+0x7dd5)[0x7f5f8eed1dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f5f8e79902d]
---------

###########################
# log from second node:
[2019-08-09 17:06:48.926010] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-gv0-server: disconnecting connection from CTX_ID:b36e17ac-62e5-4171-bfdc-1707aa5ec5a9-GRAPH_ID:0-PID:31299-HOST:gluster01-PC_NAME:gv0-client-3-RECON_NO:-0
[2019-08-09 17:06:48.926231] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-gv0-server: Shutting down connection CTX_ID:b36e17ac-62e5-4171-bfdc-1707aa5ec5a9-GRAPH_ID:0-PID:31299-HOST:gluster01-PC_NAME:gv0-client-3-RECON_NO:-0
[2019-08-09 17:06:48.955802] I [dict.c:541:dict_get] (-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f47b8c64241] -->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f47b8e8a219] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:49.038024] I [dict.c:541:dict_get] (-->/lib64/libglusterfs.so.0(default_fremovexattr+0xe7) [0x7f47c89461f7] -->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1d5dd) [0x7f47b8e8b5dd] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:49.098930] I [dict.c:541:dict_get] (-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f47b8c64241] -->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f47b8e8a219] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:50.042220] I [MSGID: 115072] [server-rpc-fops_v2.c:1681:server4_setattr_cbk] 0-gv0-server: 349203: SETATTR /munin/var/lib (418e15b7-70d5-4f11-86ac-c1d8aeb6aa12), client: CTX_ID:d6150421-3ac2-4dbc-a665-d619f13dbb15-GRAPH_ID:0-PID:538-HOST:munin-PC_NAME:gv0-client-3-RECON_NO:-0, error-xlator: gv0-access-control [Operation not permitted]
[2019-08-09 17:06:50.962416] I [MSGID: 115072] [server-rpc-fops_v2.c:1681:server4_setattr_cbk] 0-gv0-server: 349225: SETATTR /munin/var/lib (418e15b7-70d5-4f11-86ac-c1d8aeb6aa12), client: CTX_ID:d6150421-3ac2-4dbc-a665-d619f13dbb15-GRAPH_ID:0-PID:538-HOST:munin-PC_NAME:gv0-client-3-RECON_NO:-0, error-xlator: gv0-access-control [Operation not permitted]
[2019-08-09 17:06:52.073700] I [dict.c:541:dict_get] (-->/lib64/libglusterfs.so.0(default_fremovexattr+0xe7) [0x7f47c89461f7] -->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1d5dd) [0x7f47b8e8b5dd] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:52.222678] I [dict.c:541:dict_get] (-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f47b8c64241] -->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f47b8e8a219] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
pending frames:
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash: 
2019-08-09 17:06:52
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 6.4
/lib64/libglusterfs.so.0(+0x26e00)[0x7f47c88b2e00]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f47c88bd804]
/lib64/libc.so.6(+0x36340)[0x7f47c6ef2340]
/usr/lib64/glusterfs/6.4/rpc-transport/socket.so(+0xa4cc)[0x7f47bcc774cc]
/lib64/libglusterfs.so.0(+0x8b4d6)[0x7f47c89174d6]
/lib64/libpthread.so.0(+0x7dd5)[0x7f47c76f2dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f47c6fba02d]
---------


###########################
# backtrace in core dump on first node:
(gdb) bt
#0  0x00007f5f844564cc in socket_event_handler () from /usr/lib64/glusterfs/6.4/rpc-transport/socket.so
#1  0x00007f5f900f64d6 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2  0x00007f5f8eed1dd5 in start_thread (arg=0x7f5f66ffd700) at pthread_create.c:307
#3  0x00007f5f8e79902d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) 


###########################
# backtrace in core dump on second node:
(gdb) bt
#0  0x00007f47bcc774cc in socket_event_handler () from /usr/lib64/glusterfs/6.4/rpc-transport/socket.so
#1  0x00007f47c89174d6 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2  0x00007f47c76f2dd5 in start_thread (arg=0x7f47b1053700) at pthread_create.c:307
#3  0x00007f47c6fba02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

Thanks for the report. If you still have the core, I request you to post the output of '(gdb) thread apply all bt full'. That gives more details on the crash.

Also, if it possible for letting us know a pattern of access (so that we can try to reproduce the same) it would be great.

Hi, I still have the core dumps, so I will include the output of your commands as attachments.  

I'm also going to attach the cluster configuration as well. Native clients are mounted with the 'reader-thread-count=4' option.

Only additional notes are that we had switched from Copper 1GbE to Fiber 10GbE about a week before the first crash. At the same time we also added 18 additional disks per node, which would eventually comprise two additional bricks per node. 

I've been using custom Ansible playbooks to manage not only the system, but Gluster as well. When I used Ansible (https://docs.ansible.com/ansible/latest/modules/gluster_volume_module.html) to add the two additional bricks at the same time, it incorrectly paired them (bug?).

Before the incorrect addition, my cluster configuration was as follows:

  Brick1: gluster00:/export/brick0/srv
  Brick2: gluster01:/export/brick0/srv

After Ansible incorrectly paired the disks, it was something like this, IIRC:

  Brick1: gluster00:/export/brick0/srv
  Brick2: gluster01:/export/brick0/srv
  Brick3: gluster00:/export/brick1/srv
  Brick4: gluster00:/export/brick2/srv
  Brick5: gluster01:/export/brick1/srv
  Brick6: gluster01:/export/brick2/srv

After adding the bricks, I issued a rebalance command (not realizing the incorrect pairing) but about a minute into the rebalance I realized that something was amiss. I immediately realized what happened and issued a:

  gluster volume remove-brick gv0 gluster01:/export/brick2/srv gluster01:/export/brick1/srv gluster00:/export/brick2/srv gluster00:/export/brick1/srv start

After the remove completed, I did a commit to confirm the remove-brick command.  After the commit I was back to the original configuration:

  Brick1: gluster00:/export/brick0/srv
  Brick2: gluster01:/export/brick0/srv

While the data was intact, my directory permissions and and file ownership were wiped out due to a bug that may have been related to Bug #1716848. After correcting the directory permissions and ownership, the cluster ran fine for several hours, and I had planned to reattempt the brick add (via Ansible) but with one brick pair at a time so I didn't end up with mismatched brick pairs again. At the end of the day however, before I was able to re-add the brick pair, Gluster crashed with the first core dump. It was still in the two brick setup, as I had not yet re-attempted the brick add. 

(Note: I reformatted the bricks before attempting to re-use them.)

I rebooted the cluster, and upon coming back up, the self heal daemon resync'd everything. After examining the volume, I was happy with everything so I went ahead and added a brick pair via Ansible. It worked and everything was paired correctly. I then added the next pair to Ansible and ran the playbook again. Again, everything paired correctly. At this point I had the correct brick setup:

  Brick1: gluster00:/export/brick0/srv
  Brick2: gluster01:/export/brick0/srv
  Brick3: gluster00:/export/brick1/srv
  Brick4: gluster01:/export/brick1/srv
  Brick5: gluster00:/export/brick2/srv
  Brick6: gluster01:/export/brick2/srv

From here I issued a rebalance command and watched. Everything was working fine for about 10 hours, which was when the second crash happened. That is, the second crash happened in the middle of a rebalance. After everything came back up, the self heal daemon did its thing. I examined the volume, saw no issues and went ahead and started the rebalance again. This time the rebalance ran to completion (took somewhere between 1-2 days). I've had zero crashes since then.

Not sure if there is any pattern of access that caused any of this, but the timing of it around some administrative work is interesting and why I covered it in such detail above.

I should also note that I'm also using both munin-glusterfs (https://github.com/burner1024/munin-gluster) and gluster-prometheus (https://github.com/gluster/gluster-prometheus) plugins on the nodes for monitoring (although Munin is legacy at this point, and will be going away once Prometheus is fully built out).

Created attachment 1603901 [details]
gdb backtrace from gluster00 crash #1

gluster00 crash #1 core dump

Comment on attachment 1603901 [details]
gdb backtrace from gluster00 crash #1

Output of '(gdb) thread apply all bt full' from crash #1 on gluster00

Created attachment 1603902 [details]
gdb backtrace from gluster01 crash #1

Output of '(gdb) thread apply all bt full' from crash #1 on gluster01

Created attachment 1603903 [details]
gdb backtrace from gluster00 crash #2

Output of '(gdb) thread apply all bt full' from crash #2 on gluster00

Created attachment 1603904 [details]
gdb backtrace from gluster01 crash #2

Output of '(gdb) thread apply all bt full' from crash #2 on gluster01

Created attachment 1603905 [details]
lvm config

LVM configuration

Created attachment 1603906 [details]
gluster volume info

output of 'gluster volume info'

Created attachment 1603907 [details]
gluster volume status client-list

output of 'gluster volume status gv0 client-list'

Created attachment 1603909 [details]
gluster volume status detail

output of 'gluster volume status detail'

From the current report, it is not clear which thread crashed and for what reason.

Please do the following.

# Install gluster-debuginfo package of the same version as your glusterfs server.

# type the following command into terminal :
   "gdb glusterfsd $core-file" and copy the text from the terminal which should have the crashed thread bt. It looks something like this.

  
Core was generated by `/usr/local/sbin/glusterfs --process-name fuse --volfile-server=vm3 --volfile-id'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f12be8c5d9b in dht_create (frame=0x7f12a4001e48, this=0x0, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, params=0x7f12ac0020c8) at dht-common.c:8677
8677	    conf = this->private;
[Current thread is 1 (Thread 0x7f12ce3ff700 (LWP 30689))]
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.26-28.fc27.x86_64 openssl-libs-1.1.0h-3.fc27.x86_64 pcre2-10.31-7.fc27.x86_64 sssd-client-1.16.2-4.fc27.x86_64
(gdb) bt
#0  0x00007f12be8c5d9b in dht_create (frame=0x7f12a4001e48, this=0x0, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, params=0x7f12ac0020c8) at dht-common.c:8677
#1  0x00007f12be633d8a in gf_utime_create (frame=0x7f12a4006cb8, this=0x7f12b0010b70, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at utime-autogen-fops.c:172
#2  0x00007f12ce11b86c in default_create (frame=0x7f12a4006cb8, this=0x7f12b0012890, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at defaults.c:2601
#3  0x00007f12be201846 in ra_create (frame=0x7f12a4001608, this=0x7f12b0014540, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at read-ahead.c:194
#4  0x00007f12ce11b86c in default_create (frame=0x7f12a4001608, this=0x7f12b0016120, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at defaults.c:2601
#5  0x00007f12bddd896f in ioc_create (frame=0x7f12a4008ff8, this=0x7f12b0018240, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at io-cache.c:919
#6  0x00007f12ce11b86c in default_create (frame=0x7f12a4008ff8, this=0x7f12b0019e30, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at defaults.c:2601
#7  0x00007f12ce11b86c in default_create (frame=0x7f12a4008ff8, this=0x7f12b001ba30, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at defaults.c:2601
#8  0x00007f12bd798698 in mdc_create (frame=0x7f12a4000cb8, this=0x7f12b001d610, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at md-cache.c:1969
#9  0x00007f12ce1102ed in default_create_resume (frame=0x7f12b8008328, this=0x7f12b001f210, loc=0x7f12b8009130, flags=34881, mode=33188, umask=0, fd=0x7f12b8001948, xdata=0x7f12ac0020c8) at defaults.c:1873
#10 0x00007f12ce05fbf8 in call_resume_wind (stub=0x7f12b80090e8) at call-stub.c:2033
#11 0x00007f12ce071bee in call_resume (stub=0x7f12b80090e8) at call-stub.c:2555
#12 0x00007f12bd5809c6 in iot_worker (data=0x7f12b002efc0) at io-threads.c:232
#13 0x00007f12ccbf550b in start_thread () from /lib64/libpthread.so.0
#14 0x00007f12cc4a516f in clone () from /lib64/libc.so.6


And then get another report with "thread apply all bt". 

Once we check that info we will figure out the next steps.

Regards,
Susant

I had run

  debuginfo-install glusterfs-server

But it looks like it only installed the debuginfo packages for glusterfs dependencies and not glusterfs itself.  Is this a gluster packaging bug?

I'll rerun the backtrace with the glusterfs-debuginfo package installed.

I wasn't able to pull the gluster-debuginfo package via yum, even with --enablerepo=centos-storage-debuginfo.  The baseurl in the CentOS-Storage-common.repo file looks correct, so I'm not sure if there is a problem with the repodata file or something else... Anyway, I manually downloaded the glusterfs-debuginfo file installed it and am attaching the output of:

gdb glusterfsd $core-file

As well as the 

thread apply all bt 

commands

Created attachment 1606731 [details]
gluster00: gdb glusterfsd core.6563

Created attachment 1606732 [details]
gluster00: gdb glusterfsd core.31704

Created attachment 1606733 [details]
gluster01: gdb glusterfsd core.6560

Created attachment 1606734 [details]
gluster01: gdb glusterfsd core.23686

Created attachment 1606755 [details]
gdb backtrace from gluster00 crash #1 (full)

Created attachment 1606757 [details]
gdb backtrace from gluster01 crash #1 (full)

Created attachment 1606759 [details]
gdb backtrace from gluster00 crash #2 (full)

Created attachment 1606760 [details]
gdb backtrace from gluster01 crash #2 (full)

Thanks for the report.

Here is the crashed thread.

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfsd -s gluster00 --volfile-id gv0.gluster00.export-brick1-srv'.
Program terminated with signal 11, Segmentation fault.
#0  socket_is_connected (this=0x7f5f500af3c0) at socket.c:2619
2619	    if (priv->use_ssl) {
[?1034h(gdb) quit


Form this it looks like "priv" is NULL.  Can you print the value of priv for clarity (command to type after opening core with gdb: "p priv").


Assigning this to Mohit to take it further.

Susant

Hi,

I checked all four core dumps, and they each have the same value for priv:

  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib64/libthread_db.so.1".
  Core was generated by `/usr/sbin/glusterfsd -s gluster00 --volfile-id gv0.gluster00.export-brick0-srv'.
  Program terminated with signal 11, Segmentation fault.
  #0  socket_is_connected (this=0x7fb40c0ef460) at socket.c:2619
  2619	      if (priv->use_ssl) {
  (gdb) p priv
  $1 = (socket_private_t *) 0x0
  (gdb)

Hi, we can also reproduce this crash, caused by a security scanner probing the brick.

#0  socket_is_connected (this=0x7f2d5000a490) at socket.c:2619
2619        if (priv->use_ssl) {
 
(gdb) print ((rpc_transport_t *)data)->private
$4 = (void *) 0x0

Hi,

 Can you please share the dump of /var/log/glusterfs directory from the node on that 
 glusterfsd is crashed?

Thanks,
Mohit Agrawal

(In reply to Mohit Agrawal from comment #27)
> Hi,
> 
>  Can you please share the dump of /var/log/glusterfs directory from the node
> on that 
>  glusterfsd is crashed?

I'll try first sharing snippets, just so I can eyeball them for details that shouldn't be made public.  This is the brick crash, showing some error messages indicating that the service is receiving security probe-related garbage.

[2019-09-11 02:39:52.615174] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong MSG-TYPE (1929576541) received from 10.234.96.36:7933
[2019-09-11 02:39:52.618865] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong MSG-TYPE (1996685661) received from 10.234.96.36:7936
[2019-09-11 02:39:52.623556] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong MSG-TYPE (8192) received from 10.234.96.36:7939
[2019-09-11 02:39:52.627840] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong MSG-TYPE (8192) received from 10.234.96.36:7942
pending frames:
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash: 
2019-09-11 02:39:52
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 6.4
/lib64/libglusterfs.so.0(+0x26e00)[0x7f2d6c6bde00]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f2d6c6c8804]
/lib64/libc.so.6(+0x36340)[0x7f2d6acfd340]
/usr/lib64/glusterfs/6.4/rpc-transport/socket.so(+0xa4cc)[0x7f2d60a824cc]
/lib64/libglusterfs.so.0(+0x8b4d6)[0x7f2d6c7224d6]
/lib64/libpthread.so.0(+0x7dd5)[0x7f2d6b4fddd5]
/lib64/libc.so.6(clone+0x6d)[0x7f2d6adc502d]
---------

Hi,

 Is it possible for you to backport the patch(https://review.gluster.org/#/c/glusterfs/+/23775/) on top of your current version and share the result?



Thanks,
Mohit Agrawal

Hi,

Hi, I'm not willing to try a patch on a production system. Since I initially created this report we've retired our old NAS systems and are running 100% off of gluster. Furthermore, I've upgraded the production current cluster to 6.6 to get a fix from another bug I was working around. 

I could apply the patch to a test VM but the configuration, although similar, would not be the same as what I'm using in production. Also, I use the pre-built binary RPMs, so I'm not sure if you'd want me to build glusterfs from source if I went that route. If I did build from source, I'd need to match the build flags any everything else that comes with the CentOS RPMs, no?

We have identified the problem. There's a patch [1] to fix it. It will be available on next release.

[1] https://review.gluster.org/c/glusterfs/+/23861

The patch is already present on v6.7

*** This bug has been marked as a duplicate of bug 1749625 ***