Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1054668

Summary: One brick always has alot of entries on `gluster volume heal gv0 info`
Product: [Community] GlusterFS Reporter: Diep Pham Van <imeo>
Component: replicateAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED INSUFFICIENT_DATA QA Contact:
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: gluster-bugs, imeo
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-05-09 09:36:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log
none
/var/log/glusterfs/glustershd.log
none
xattrs
none
brick process
none
mount log
none
/var/log/glusterfs/glustershd.log (svr387-214)
none
mount log none

Description Diep Pham Van 2014-01-17 08:54:30 UTC 
Description of problem:
I have a cluster with 40 bricks running glusterfs 3.4.0 on Centos 6.4.
One brick always has about 20-30 entries, while all other bricks have 0 or 1 entries.
The list of files that need to heal on that brick change, but the total number is always bigger than 20.

Version-Release number of selected component (if applicable):
GlusterFS 3.4.0

How reproducible:
Always

Steps to Reproduce:
1.Run gluster volume heal gv0 info

Actual results:
Brick svr386-213.localdomain:/export/sdg1/brick
Number of entries is always bigger than 20.

Expected results:
Number of entries is the same as other bricks.

Additional info:
There are some E level lines in etc-glusterfs-glusterd.vol.log and glustershd.log.
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log: http://pastebin.com/qriYa3pr
/var/log/glusterfs/glustershd.log: http://pastebin.com/58uVqGdt

Seem that some xattr is missing on brick svr386-214.localdomain:/export/sdg1/brick
Sample xattrs: http://pastebin.com/AcriYHc6
The brick is running on both servers: http://pastebin.com/Nu3yjqCp

The brick log /var/log/glusterfs/bricks/export-sdg1-brick.log is empty.

Mount options: svr385-212.localdomain:/gv0 /glusterfs glusterfs defaults,backupvolfile-server=svr386-213.localdomain,_netdev 0 0
The mount log: http://pastebin.com/nY7dLXad

If you need more information, just let me know.
Comment 1 Pranith Kumar K 2014-01-17 09:03:19 UTC 
hi Diep Pham Van,
    Could you please attach logs for bricks and mount please to this bug. Pastebin has a limit on when it will delete the pages, so it would be good if you could attach the documents to this bug itself.

Pranith
Comment 2 Diep Pham Van 2014-01-17 09:21:48 UTC 
Created attachment 851474 [details]
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log
Comment 3 Diep Pham Van 2014-01-17 09:22:32 UTC 
Created attachment 851475 [details]
/var/log/glusterfs/glustershd.log
Comment 4 Diep Pham Van 2014-01-17 09:23:19 UTC 
Created attachment 851476 [details]
xattrs
Comment 5 Diep Pham Van 2014-01-17 09:23:51 UTC 
Created attachment 851477 [details]
brick process
Comment 6 Diep Pham Van 2014-01-17 09:24:20 UTC 
Created attachment 851478 [details]
mount log
Comment 7 Diep Pham Van 2014-01-17 09:25:38 UTC 
(In reply to Pranith Kumar K from comment #1)
> hi Diep Pham Van,
>     Could you please attach logs for bricks and mount please to this bug.
> Pastebin has a limit on when it will delete the pages, so it would be good
> if you could attach the documents to this bug itself.
> 
> Pranith

I've attached all the log files.
The bricks log (/var/log/glusterfs/bricks/export-sdg1-brick.log) is empty.
Comment 8 Pranith Kumar K 2014-01-17 10:53:03 UTC 
Diep,
      It seems like these are partial logs. Could you please attach the full log files. If they exceed the limit please compress the file and upload them.

Pranith
Comment 9 Diep Pham Van 2014-01-18 03:02:52 UTC 
Created attachment 851891 [details]
/var/log/glusterfs/glustershd.log (svr387-214)
Comment 10 Diep Pham Van 2014-01-18 03:07:02 UTC 
Created attachment 851892 [details]
mount log
Comment 11 Diep Pham Van 2014-02-07 07:23:46 UTC 
Do you need more information?
The problem is still happening. 
Do you think that a blindly reboot coud make it disappear?
Comment 12 Diep Pham Van 2014-05-09 09:36:48 UTC 
After a reboot, the problem is disappear.
I can not reproduce it anymore.