Bug 1326956

Summary: A 3.1.2, during stress tests files are disappearing and then coming back.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Oonkwee Lim <olim>
Component: coreAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Anoop <annair>
Severity: urgent Docs Contact:
Priority: urgent    
Version: rhgs-3.1CC: hamiller, olim, pkarampu, rhs-bugs, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-02 00:56:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Oonkwee Lim 2016-04-13 22:17:21 UTC 
Description of problem:
From the customer:

After the upgrade it looks like we are getting the following error:

[2016-04-12 17:00:51.425700] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-dfs01-replicate-0: Unreadable subvolume -1 found with event generation 2 for gfid 39529c1a-ffe1-4652-b733-66e4ee17925d. (Possible split-brain)

We have seen several cases where files disappear while our application runs.

Our RHEL 5 clients appears to be the only ones impacted. If I run the following on a RHEL 5 node:

Export TMPDIR=/glustervolume/tmp
echo "" | mail -s test Ryan.MattesoN

I receive a message with no subject. If I run the exact same commands on a RHEL 7 node the messages arrives with a subject. I've also seen cases where files will disappear from the file system and come back at a later time.

Version-Release number of selected component (if applicable):
Servers:
Red Hat Enterprise Linux Server release 6.7 (Santiago)
Red Hat Gluster Storage Server 3.1 Update 2
operating-version=30707 for both servers, dsrhs01 and dsrhs02

Client:
Red Hat Enterprise Linux Server release 5.11 (Tikanga)
glusterfs-3.7.5-19.el5.x86_64                               Tue Apr 12 10:35:18 2016
glusterfs-client-xlators-3.7.5-19.el5.x86_64                Tue Apr 12 10:35:17 2016
glusterfs-fuse-3.7.5-19.el5.x86_64                          Tue Apr 12 10:35:19 2016
glusterfs-libs-3.7.5-19.el5.x86_64                          Tue Apr 12 10:35:16 2016


How reproducible:
Easily

Steps to Reproduce:
1. Export TMPDIR=/glustervolume/tmp
2. echo "" | mail -s test Ryan.MattesoN
3.

Actual results:
messages arrives with no subject

Expected results:
messages arrives with a subject

Additional info:
sosreports and etc can be accessed through http://collab-shell.usersys.redhat.com/01549155/ or through

    # ssh your_kerb.redhat.com
    # cd /cases/01549155