Bug 1434000

Summary: File marked bad by bitrot on both bricks of replica leading to EIO on the client.
Product: [Community] GlusterFS Reporter: Bernhard <1linuxengineer>
Component: bitrotAssignee: Raghavendra Bhat <rabhat>
Status: CLOSED EOL QA Contact:
Severity: low Docs Contact: bugs <bugs>
Priority: unspecified    
Version: 3.8CC: bugs, ravishankar
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-07 10:42:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
xattrs of some file on each brick none

Description Bernhard 2017-03-20 14:08:24 UTC 
Description of problem: the volume logfile reports a possible split-brain but when I try to heal it fails because the file is not in split-brain


Version-Release number of selected component (if applicable):3.8.8


How reproducible:don't know


Steps to Reproduce:
1.
2.
3.

Actual results:affected files are not accessible, I/O error


Expected results:files fully accessible


Additional info:I don't know how we got into this situation. we have 2 machines with 60 disks each. 42 of them are in 42 replicated gluster volumes. the machines are running several lxc and the gluster volumes are presented into the containers. the machines are under high memory pressure, so it could be that at some point in time a critical process was killed by the OOM Killer.

root@chastcvtprd04:~# uname -a
Linux chastcvtprd04 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@chastcvtprd04:~# dpkg -l | grep gluster
ii  glusterfs-client                    3.8.8-ubuntu1~xenial1              amd64        clustered file-system (client package)
ii  glusterfs-common                    3.8.8-ubuntu1~xenial1              amd64        GlusterFS common libraries and translator modules
ii  glusterfs-server                    3.8.8-ubuntu1~xenial1              amd64        clustered file-system (server package)


I have about 4GB of logs on my machines. Let me know what you need and I'll upload it
Comment 1 Bernhard 2017-03-20 14:36:10 UTC 
Created attachment 1264772 [details]
xattrs of some file on each brick
Comment 2 Ravishankar N 2017-03-20 15:54:02 UTC 
Discussion on gluster-users: http://lists.gluster.org/pipermail/gluster-users/2017-March/030352.html
Comment 3 Bernhard 2017-03-21 15:59:04 UTC 
you can find the gluster logs from both nodes at
https://drive.google.com/drive/folders/0B9xZMimoXkY1RFpBaTJiWXRSbUE?usp=sharing

there are 2 ZIP files, each ~2GB with the logs of all involved volumes and bitd
Comment 4 Niels de Vos 2017-11-07 10:42:53 UTC 
This bug is getting closed because the 3.8 version is marked End-Of-Life. There will be no further updates to this version. Please open a new bug against a version that still receives bugfixes if you are still facing this issue in a more current release.