Bug 1434000 - File marked bad by bitrot on both bricks of replica leading to EIO on the client.
Summary: File marked bad by bitrot on both bricks of replica leading to EIO on the cli...
Keywords:
Status: CLOSED EOL
Alias: None
Product: GlusterFS
Classification: Community
Component: bitrot
Version: 3.8
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: Raghavendra Bhat
QA Contact:
bugs@gluster.org
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-20 14:08 UTC by Bernhard
Modified: 2017-11-07 10:42 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-07 10:42:53 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)
xattrs of some file on each brick (29.23 KB, application/zip)
2017-03-20 14:36 UTC, Bernhard
no flags Details

Description Bernhard 2017-03-20 14:08:24 UTC
Description of problem: the volume logfile reports a possible split-brain but when I try to heal it fails because the file is not in split-brain


Version-Release number of selected component (if applicable):3.8.8


How reproducible:don't know


Steps to Reproduce:
1.
2.
3.

Actual results:affected files are not accessible, I/O error


Expected results:files fully accessible


Additional info:I don't know how we got into this situation. we have 2 machines with 60 disks each. 42 of them are in 42 replicated gluster volumes. the machines are running several lxc and the gluster volumes are presented into the containers. the machines are under high memory pressure, so it could be that at some point in time a critical process was killed by the OOM Killer.

root@chastcvtprd04:~# uname -a
Linux chastcvtprd04 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@chastcvtprd04:~# dpkg -l | grep gluster
ii  glusterfs-client                    3.8.8-ubuntu1~xenial1              amd64        clustered file-system (client package)
ii  glusterfs-common                    3.8.8-ubuntu1~xenial1              amd64        GlusterFS common libraries and translator modules
ii  glusterfs-server                    3.8.8-ubuntu1~xenial1              amd64        clustered file-system (server package)


I have about 4GB of logs on my machines. Let me know what you need and I'll upload it

Comment 1 Bernhard 2017-03-20 14:36:10 UTC
Created attachment 1264772 [details]
xattrs of some file on each brick

Comment 2 Ravishankar N 2017-03-20 15:54:02 UTC
Discussion on gluster-users: http://lists.gluster.org/pipermail/gluster-users/2017-March/030352.html

Comment 3 Bernhard 2017-03-21 15:59:04 UTC
you can find the gluster logs from both nodes at
https://drive.google.com/drive/folders/0B9xZMimoXkY1RFpBaTJiWXRSbUE?usp=sharing

there are 2 ZIP files, each ~2GB with the logs of all involved volumes and bitd

Comment 4 Niels de Vos 2017-11-07 10:42:53 UTC
This bug is getting closed because the 3.8 version is marked End-Of-Life. There will be no further updates to this version. Please open a new bug against a version that still receives bugfixes if you are still facing this issue in a more current release.


Note You need to log in before you can comment on or make changes to this bug.