Bug 1388221

Summary: gluster brick daemon segfaulted in pairs
Product: [Community] GlusterFS Reporter: Jackie Tung <jackie>
Component: trash-xlatorAssignee: bugs <bugs>
Status: CLOSED EOL QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.8CC: bugs, jackie, vbellur
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-07 10:41:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
stack trace from one of the cores
none
brick daemon log none

Description Jackie Tung 2016-10-24 18:59:45 UTC 
Description of problem:

We are running a distributed replicated volume: 16 pairs of bricks (rep count 2), 2 nodes.

On Friday, 2 pairs of brick daemons seg-faulted within minutes of each other, leading to 2 subvolumes down (no replicas left).  We tried to bring them up again by doing a "volume start force”, which worked, but roughly 4 hours later this happened again, but to two other pairs of bricks.

There is nothing of note in brick logs for the downed bricks, except that it just suddenly stops logging.  In the other logs (nfs, glusterhd, etc), we simply start seeing errors saying “All sub volumes down” for those replicates.

This is on Ubuntu 16.04

Version-Release number of selected component (if applicable):
3.8.2

How reproducible:

It happened three rounds in total so far.

Steps to Reproduce:
1. force start volume
2. wait for crash

Additional info:
Core file too large to attach here (60-70M), is there an alternative way to submit it?

Did not see any stacktraces anywhere.
Comment 1 Kaleb KEITHLEY 2016-10-25 12:25:09 UTC 
please post the log files.

You can get a stacktrace using gdb with something like `gdb /usr/bin/glusterfsd /core` then `bt`

Once we have that we can better determine the next steps.  Thanks
Comment 2 Jackie Tung 2016-10-25 15:17:09 UTC 
Created attachment 1213948 [details]
stack trace from one of the cores
Comment 3 Jackie Tung 2016-10-25 21:04:09 UTC 
Created attachment 1214063 [details]
brick daemon log
Comment 4 Niels de Vos 2017-11-07 10:41:56 UTC 
This bug is getting closed because the 3.8 version is marked End-Of-Life. There will be no further updates to this version. Please open a new bug against a version that still receives bugfixes if you are still facing this issue in a more current release.