Description of problem: We are running a distributed replicated volume: 16 pairs of bricks (rep count 2), 2 nodes. On Friday, 2 pairs of brick daemons seg-faulted within minutes of each other, leading to 2 subvolumes down (no replicas left). We tried to bring them up again by doing a "volume start force”, which worked, but roughly 4 hours later this happened again, but to two other pairs of bricks. There is nothing of note in brick logs for the downed bricks, except that it just suddenly stops logging. In the other logs (nfs, glusterhd, etc), we simply start seeing errors saying “All sub volumes down” for those replicates. This is on Ubuntu 16.04 Version-Release number of selected component (if applicable): 3.8.2 How reproducible: It happened three rounds in total so far. Steps to Reproduce: 1. force start volume 2. wait for crash Additional info: Core file too large to attach here (60-70M), is there an alternative way to submit it? Did not see any stacktraces anywhere.
please post the log files. You can get a stacktrace using gdb with something like `gdb /usr/bin/glusterfsd /core` then `bt` Once we have that we can better determine the next steps. Thanks
Created attachment 1213948 [details] stack trace from one of the cores
Created attachment 1214063 [details] brick daemon log
This bug is getting closed because the 3.8 version is marked End-Of-Life. There will be no further updates to this version. Please open a new bug against a version that still receives bugfixes if you are still facing this issue in a more current release.