Bug 852869 - volume heal info shows mostly gfid not filenames
volume heal info shows mostly gfid not filenames
Status: CLOSED NOTABUG
Product: GlusterFS
Classification: Community
Component: replicate (Show other bugs)
3.3.0
Unspecified Unspecified
unspecified Severity low
: ---
: ---
Assigned To: Pranith Kumar K
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-29 15:51 EDT by Rob.Hendelman
Modified: 2013-10-29 19:03 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-01-11 02:20:05 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rob.Hendelman 2012-08-29 15:51:25 EDT
Description of problem:
volume heal info shows mostly gfids not filenames

Version-Release number of selected component (if applicable):
from git: 3.3.0 , commit e442b07f1d77e059e4b009adb4c9d0273850dcf5 from master branch

How reproducible:
Setup:
Volume Name: raid10
Type: Distributed-Replicate
Volume ID: 4e92f7ea-eed1-484c-967f-b2ccc4a832b7
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: evdevlxgl01:/mnt/gluster/brick1
Brick2: evdevlxgl02:/mnt/gluster/brick1
Brick3: evdevlxgl03:/mnt/gluster/brick1
Brick4: evdevlxgl04:/mnt/gluster/brick1
Options Reconfigured:
cluster.self-heal-window-size: 128
cluster.data-self-heal-algorithm: full
cluster.stripe-block-size: 524288
cluster.self-heal-daemon: on
diagnostics.brick-log-level: DEBUG
diagnostics.client-log-level: TRACE


Steps to Reproduce:
1. Setup as above
2. rsync files from nfs mount (not gluster nfs, ubuntu) to gluster mount
3. randomly reboot different nodes one at a time.
4. Expect heal to fix everything.
  
Actual results:
gluster> volume heal raid10 info
Gathering Heal info on volume raid10 has been successful

Brick evdevlxgl01:/mnt/gluster/brick1
Number of entries: 2
/d/M/s/A/evdev2
/d/M/s/A/evdev2/temp

Brick evdevlxgl02:/mnt/gluster/brick1
Number of entries: 59
<gfid:d0ddc89a-3542-4d6a-bdd8-7a9157361495>
<gfid:3f118b86-dbfc-49a9-88a5-0062be05911f>
<gfid:217c56c2-c9e1-46c9-bfde-e30f232fd634>
<gfid:2a2c3a95-c9a1-43c8-b5c5-f787fb93c684>
<gfid:766e6b7d-47b2-49dd-baa4-a0a45dce1249>
<gfid:df401d60-78ac-494b-95ba-b68641fa06fd>
<gfid:e2861c29-6285-4fc8-a3a9-a57767ed8734>
<gfid:8078f71f-6452-453e-b838-a8a9a0e8c8fa>
<gfid:5fe05283-bb98-4d8e-8c0b-80aaa8295698>
<gfid:faa65063-7554-495e-8478-f94472f27a10>
<gfid:3c61e9e3-4c66-49f8-8b4e-a9906d2ea98b>
<gfid:d4e67023-bb57-4df7-8814-b4135531dcc1>
<gfid:27b90a6f-c561-4f6c-88f7-d52a9fac0ab8>
<gfid:089734cf-b1ce-4bb9-af36-2fbf355eaf94>
<gfid:7f724507-6c25-45d4-a24b-e6f9d45aab60>
<gfid:d0a59f17-3a71-447b-805e-8a4bbcd7c57f>
/d/M/s/A/evdev2
/d/M/s/A/evdev2/temp
<gfid:fa68646a-ed2e-4267-aeb8-c9f9ab6975fb>
<gfid:680c9158-789a-432e-bba0-0328dd6b09f8>
<gfid:f0fd2b7f-ec12-4480-af5a-561cc45c6b3a>
<gfid:81c1477b-6c6f-4027-a499-ca5a5c57790f>
<gfid:7f71988f-012a-4efb-a728-be87b7fb9574>
<gfid:2cb8c612-c8c0-4a9c-b62d-d1a27a4ec992>
<gfid:48d1f10c-f741-4836-ba79-3e28b2f33d34>
<gfid:dd4a296f-049b-4e57-809f-afd1c9f6f752>
<gfid:0a5702d3-3a5e-4258-a849-aa1d14a7a5cc>
<gfid:f0c3b1e8-19f9-4840-a787-c5b0b902f269>
<gfid:2dc21f0a-6ecc-4339-99a9-08a708411ca5>
<gfid:ad33eecf-620c-481e-90a5-ed5312270375>
<gfid:09a35db7-2942-494a-acc8-c4aeb34b8fa5>
<gfid:36ebb133-e909-4f56-8720-7cb326558c3f>
<gfid:f1df0518-fa69-4efc-9681-a39602caf35d>
<gfid:91c0d94b-8a74-465e-9fe2-02c9af295606>
<gfid:4d17eb59-ec25-44dd-a461-a98ec7dcba70>
<gfid:8930060d-ac32-4654-b099-300e8cd84156>
<gfid:36820010-6410-41de-8cf6-89c255c1708f>
<gfid:31292473-ce88-4eeb-ba1f-513a7cf23b9e>
<gfid:6289d1dc-bfbf-43fa-94a7-63a017f8a9b9>
<gfid:4123865f-8ebb-4310-9721-e8d720505694>
<gfid:bf0dd4ff-6db9-4a52-98d7-8685ed9a221b>
<gfid:b130a8fc-a7a6-4d24-bf9c-92518ca93b38>
<gfid:a2e36141-473a-45d5-8211-b29ce01b8d66>
<gfid:346d2516-9786-42b5-9c4a-e4cfb78cfba0>
<gfid:0621fd80-0b04-40e1-bf77-2085be3b1478>
<gfid:1dfcf56c-1079-4de6-b18c-e1ba35b88485>
<gfid:bab2a792-f485-4c54-809d-483817fb9820>
<gfid:7801971a-c6f4-4995-8229-f8b021ade062>
<gfid:9982edca-1862-40ad-b175-c302bd78484b>
<gfid:42625a63-14f7-44f3-b249-7fe91dd95d60>
<gfid:4c285fe6-c433-426f-8c1f-b7b89aa59a0a>
<gfid:abfc94b4-ac45-4a88-9f3a-4709d49f0fbe>
<gfid:7cd38a1b-8056-47bf-b0ee-547c7f049b52>
<gfid:9e899002-0bec-497a-9079-eedb3684d2a8>
<gfid:47888ca7-82ac-49b8-82ac-a7409a1f6c83>
<gfid:d7a95e30-1b5d-402c-a3ba-0aebf5c4c0b0>
<gfid:8f58e5fb-780c-43e3-af66-de093da6499e>
<gfid:a6f4c199-aef8-4844-b166-50a7bbb95fb9>
<gfid:6c2e41f1-79b9-4a50-b4d3-75139b7a7b2f>

Brick evdevlxgl03:/mnt/gluster/brick1
Number of entries: 0

Brick evdevlxgl04:/mnt/gluster/brick1
Number of entries: 0

Expected results:
Healing/replicate to fix everything.  I'm uncertain what files are affected, although an rsync with -n option (dry run) shows filenames on the raw bricks, I'm uncertain as to why gluster can't heal these files.

Additional info:
It would be nice to have a nicer display than just the gfid above.
Comment 1 Pranith Kumar K 2012-08-30 05:02:25 EDT
Can you post the output of 'gluster volume heal raid10 split-brain' output?

Pranith.
Comment 2 Rob.Hendelman 2012-08-30 08:59:11 EDT
It looks like the same file(s over & over again.  The "files" (/d/M/s/A/evdev2 & /d/M/s/A/evdev2/temp) are actually directory entries.
================

Brick evdevlxgl01:/mnt/gluster/brick1
Number of entries: 210
at                    path on brick
-----------------------------------
2012-08-30 12:52:53 /d/M/s/A/evdev2/temp
2012-08-30 12:52:53 /d/M/s/A/evdev2
2012-08-30 12:42:53 /d/M/s/A/evdev2/temp
2012-08-30 12:42:53 /d/M/s/A/evdev2
2012-08-30 12:32:53 /d/M/s/A/evdev2/temp
2012-08-30 12:32:53 /d/M/s/A/evdev2
2012-08-30 12:22:53 /d/M/s/A/evdev2/temp
2012-08-30 12:22:53 /d/M/s/A/evdev2
2012-08-30 12:12:53 /d/M/s/A/evdev2/temp
2012-08-30 12:12:53 /d/M/s/A/evdev2
2012-08-30 12:02:53 /d/M/s/A/evdev2/temp
2012-08-30 12:02:53 /d/M/s/A/evdev2
2012-08-30 11:52:53 /d/M/s/A/evdev2/temp
2012-08-30 11:52:53 /d/M/s/A/evdev2
2012-08-30 11:42:53 /d/M/s/A/evdev2/temp
2012-08-30 11:42:53 /d/M/s/A/evdev2
2012-08-30 11:32:53 /d/M/s/A/evdev2/temp
2012-08-30 11:32:53 /d/M/s/A/evdev2
2012-08-30 11:22:53 /d/M/s/A/evdev2/temp
2012-08-30 11:22:53 /d/M/s/A/evdev2
2012-08-30 11:12:53 /d/M/s/A/evdev2/temp
2012-08-30 11:12:53 /d/M/s/A/evdev2
2012-08-30 11:02:52 /d/M/s/A/evdev2/temp
2012-08-30 11:02:52 /d/M/s/A/evdev2
2012-08-30 10:52:52 /d/M/s/A/evdev2/temp
2012-08-30 10:52:52 /d/M/s/A/evdev2
2012-08-30 10:42:52 /d/M/s/A/evdev2/temp
2012-08-30 10:42:52 /d/M/s/A/evdev2
2012-08-30 10:32:52 /d/M/s/A/evdev2/temp
2012-08-30 10:32:52 /d/M/s/A/evdev2
2012-08-30 10:22:52 /d/M/s/A/evdev2/temp
2012-08-30 10:22:52 /d/M/s/A/evdev2
2012-08-30 10:12:52 /d/M/s/A/evdev2/temp
2012-08-30 10:12:52 /d/M/s/A/evdev2
2012-08-30 10:02:52 /d/M/s/A/evdev2/temp
2012-08-30 10:02:52 /d/M/s/A/evdev2
2012-08-30 09:52:52 /d/M/s/A/evdev2/temp
2012-08-30 09:52:52 /d/M/s/A/evdev2
2012-08-30 09:42:52 /d/M/s/A/evdev2/temp
2012-08-30 09:42:52 /d/M/s/A/evdev2
2012-08-30 09:32:52 /d/M/s/A/evdev2/temp
2012-08-30 09:32:52 /d/M/s/A/evdev2
2012-08-30 09:22:52 /d/M/s/A/evdev2/temp
2012-08-30 09:22:52 /d/M/s/A/evdev2
2012-08-30 09:12:52 /d/M/s/A/evdev2/temp
2012-08-30 09:12:52 /d/M/s/A/evdev2
2012-08-30 09:02:52 /d/M/s/A/evdev2/temp
2012-08-30 09:02:52 /d/M/s/A/evdev2
2012-08-30 08:52:51 /d/M/s/A/evdev2/temp
2012-08-30 08:52:51 /d/M/s/A/evdev2
2012-08-30 08:42:51 /d/M/s/A/evdev2/temp
2012-08-30 08:42:51 /d/M/s/A/evdev2
2012-08-30 08:32:51 /d/M/s/A/evdev2/temp
2012-08-30 08:32:51 /d/M/s/A/evdev2
2012-08-30 08:22:51 /d/M/s/A/evdev2/temp
2012-08-30 08:22:51 /d/M/s/A/evdev2
2012-08-30 08:12:51 /d/M/s/A/evdev2/temp
2012-08-30 08:12:51 /d/M/s/A/evdev2
2012-08-30 08:02:51 /d/M/s/A/evdev2/temp
2012-08-30 08:02:51 /d/M/s/A/evdev2
2012-08-30 07:52:51 /d/M/s/A/evdev2/temp
2012-08-30 07:52:51 /d/M/s/A/evdev2
2012-08-30 07:42:51 /d/M/s/A/evdev2/temp
2012-08-30 07:42:51 /d/M/s/A/evdev2
2012-08-30 07:32:51 /d/M/s/A/evdev2/temp
2012-08-30 07:32:51 /d/M/s/A/evdev2
2012-08-30 07:22:51 /d/M/s/A/evdev2/temp
2012-08-30 07:22:51 /d/M/s/A/evdev2
2012-08-30 07:12:51 /d/M/s/A/evdev2/temp
2012-08-30 07:12:51 /d/M/s/A/evdev2
2012-08-30 07:02:51 /d/M/s/A/evdev2/temp
2012-08-30 07:02:51 /d/M/s/A/evdev2
2012-08-30 06:52:51 /d/M/s/A/evdev2/temp
2012-08-30 06:52:51 /d/M/s/A/evdev2
2012-08-30 06:42:50 /d/M/s/A/evdev2/temp
2012-08-30 06:42:50 /d/M/s/A/evdev2
2012-08-30 06:32:50 /d/M/s/A/evdev2/temp
2012-08-30 06:32:50 /d/M/s/A/evdev2
2012-08-30 06:22:50 /d/M/s/A/evdev2/temp
2012-08-30 06:22:50 /d/M/s/A/evdev2
2012-08-30 06:12:50 /d/M/s/A/evdev2/temp
2012-08-30 06:12:50 /d/M/s/A/evdev2
2012-08-30 06:02:50 /d/M/s/A/evdev2/temp
2012-08-30 06:02:50 /d/M/s/A/evdev2
2012-08-30 05:52:50 /d/M/s/A/evdev2/temp
2012-08-30 05:52:50 /d/M/s/A/evdev2
2012-08-30 05:42:50 /d/M/s/A/evdev2/temp
2012-08-30 05:42:50 /d/M/s/A/evdev2
2012-08-30 05:32:50 /d/M/s/A/evdev2/temp
2012-08-30 05:32:50 /d/M/s/A/evdev2
2012-08-30 05:22:50 /d/M/s/A/evdev2/temp
2012-08-30 05:22:50 /d/M/s/A/evdev2
2012-08-30 05:12:50 /d/M/s/A/evdev2/temp
2012-08-30 05:12:50 /d/M/s/A/evdev2
2012-08-30 05:02:50 /d/M/s/A/evdev2/temp
2012-08-30 05:02:50 /d/M/s/A/evdev2
2012-08-30 04:52:50 /d/M/s/A/evdev2/temp
2012-08-30 04:52:50 /d/M/s/A/evdev2
2012-08-30 04:42:50 /d/M/s/A/evdev2/temp
2012-08-30 04:42:50 /d/M/s/A/evdev2
2012-08-30 04:32:49 /d/M/s/A/evdev2/temp
2012-08-30 04:32:49 /d/M/s/A/evdev2
2012-08-30 04:22:49 /d/M/s/A/evdev2/temp
2012-08-30 04:22:49 /d/M/s/A/evdev2
2012-08-30 04:12:49 /d/M/s/A/evdev2/temp
2012-08-30 04:12:49 /d/M/s/A/evdev2
2012-08-30 04:02:49 /d/M/s/A/evdev2/temp
2012-08-30 04:02:49 /d/M/s/A/evdev2
2012-08-30 03:52:49 /d/M/s/A/evdev2/temp
2012-08-30 03:52:49 /d/M/s/A/evdev2
2012-08-30 03:42:49 /d/M/s/A/evdev2/temp
2012-08-30 03:42:49 /d/M/s/A/evdev2
2012-08-30 03:32:49 /d/M/s/A/evdev2/temp
2012-08-30 03:32:49 /d/M/s/A/evdev2
2012-08-30 03:22:49 /d/M/s/A/evdev2/temp
2012-08-30 03:22:49 /d/M/s/A/evdev2
2012-08-30 03:12:49 /d/M/s/A/evdev2/temp
2012-08-30 03:12:49 /d/M/s/A/evdev2
2012-08-30 03:02:49 /d/M/s/A/evdev2/temp
2012-08-30 03:02:49 /d/M/s/A/evdev2
2012-08-30 02:52:49 /d/M/s/A/evdev2/temp
2012-08-30 02:52:49 /d/M/s/A/evdev2
2012-08-30 02:42:49 /d/M/s/A/evdev2/temp
2012-08-30 02:42:49 /d/M/s/A/evdev2
2012-08-30 02:32:49 /d/M/s/A/evdev2/temp
2012-08-30 02:32:49 /d/M/s/A/evdev2
2012-08-30 02:22:49 /d/M/s/A/evdev2/temp
2012-08-30 02:22:49 /d/M/s/A/evdev2
2012-08-30 02:12:49 /d/M/s/A/evdev2/temp
2012-08-30 02:12:49 /d/M/s/A/evdev2
2012-08-30 02:02:49 /d/M/s/A/evdev2/temp
2012-08-30 02:02:49 /d/M/s/A/evdev2
2012-08-30 01:52:48 /d/M/s/A/evdev2/temp
2012-08-30 01:52:48 /d/M/s/A/evdev2
2012-08-30 01:42:48 /d/M/s/A/evdev2/temp
2012-08-30 01:42:48 /d/M/s/A/evdev2
2012-08-30 01:32:48 /d/M/s/A/evdev2/temp
2012-08-30 01:32:48 /d/M/s/A/evdev2
2012-08-30 01:22:48 /d/M/s/A/evdev2/temp
2012-08-30 01:22:48 /d/M/s/A/evdev2
2012-08-30 01:12:48 /d/M/s/A/evdev2/temp
2012-08-30 01:12:48 /d/M/s/A/evdev2
2012-08-30 01:02:48 /d/M/s/A/evdev2/temp
2012-08-30 01:02:48 /d/M/s/A/evdev2
2012-08-30 00:52:48 /d/M/s/A/evdev2/temp
2012-08-30 00:52:48 /d/M/s/A/evdev2
2012-08-30 00:42:48 /d/M/s/A/evdev2/temp
2012-08-30 00:42:48 /d/M/s/A/evdev2
2012-08-30 00:32:48 /d/M/s/A/evdev2/temp
2012-08-30 00:32:48 /d/M/s/A/evdev2
2012-08-30 00:22:48 /d/M/s/A/evdev2/temp
2012-08-30 00:22:48 /d/M/s/A/evdev2
2012-08-30 00:12:48 /d/M/s/A/evdev2/temp
2012-08-30 00:12:48 /d/M/s/A/evdev2
2012-08-30 00:02:48 /d/M/s/A/evdev2/temp
2012-08-30 00:02:48 /d/M/s/A/evdev2
2012-08-29 23:52:48 /d/M/s/A/evdev2/temp
2012-08-29 23:52:48 /d/M/s/A/evdev2
2012-08-29 23:42:47 /d/M/s/A/evdev2/temp
2012-08-29 23:42:47 /d/M/s/A/evdev2
2012-08-29 23:32:47 /d/M/s/A/evdev2/temp
2012-08-29 23:32:47 /d/M/s/A/evdev2
2012-08-29 23:22:47 /d/M/s/A/evdev2/temp
2012-08-29 23:22:47 /d/M/s/A/evdev2
2012-08-29 23:12:47 /d/M/s/A/evdev2/temp
2012-08-29 23:12:47 /d/M/s/A/evdev2
2012-08-29 23:02:47 /d/M/s/A/evdev2/temp
2012-08-29 23:02:47 /d/M/s/A/evdev2
2012-08-29 22:52:47 /d/M/s/A/evdev2/temp
2012-08-29 22:52:47 /d/M/s/A/evdev2
2012-08-29 22:42:47 /d/M/s/A/evdev2/temp
2012-08-29 22:42:47 /d/M/s/A/evdev2
2012-08-29 22:32:47 /d/M/s/A/evdev2/temp
2012-08-29 22:32:47 /d/M/s/A/evdev2
2012-08-29 22:22:47 /d/M/s/A/evdev2/temp
2012-08-29 22:22:47 /d/M/s/A/evdev2
2012-08-29 22:12:47 /d/M/s/A/evdev2/temp
2012-08-29 22:12:47 /d/M/s/A/evdev2
2012-08-29 22:02:47 /d/M/s/A/evdev2/temp
2012-08-29 22:02:47 /d/M/s/A/evdev2
2012-08-29 21:52:47 /d/M/s/A/evdev2/temp
2012-08-29 21:52:47 /d/M/s/A/evdev2
2012-08-29 21:42:47 /d/M/s/A/evdev2/temp
2012-08-29 21:42:47 /d/M/s/A/evdev2
2012-08-29 21:32:46 /d/M/s/A/evdev2/temp
2012-08-29 21:32:46 /d/M/s/A/evdev2
2012-08-29 21:22:46 /d/M/s/A/evdev2/temp
2012-08-29 21:22:46 /d/M/s/A/evdev2
2012-08-29 21:12:46 /d/M/s/A/evdev2/temp
2012-08-29 21:12:46 /d/M/s/A/evdev2
2012-08-29 21:02:46 /d/M/s/A/evdev2/temp
2012-08-29 21:02:46 /d/M/s/A/evdev2
2012-08-29 20:52:46 /d/M/s/A/evdev2/temp
2012-08-29 20:52:46 /d/M/s/A/evdev2
2012-08-29 20:42:46 /d/M/s/A/evdev2/temp
2012-08-29 20:42:46 /d/M/s/A/evdev2
2012-08-29 20:32:46 /d/M/s/A/evdev2/temp
2012-08-29 20:32:46 /d/M/s/A/evdev2
2012-08-29 20:22:46 /d/M/s/A/evdev2/temp
2012-08-29 20:22:46 /d/M/s/A/evdev2
2012-08-29 20:12:46 /d/M/s/A/evdev2/temp
2012-08-29 20:12:46 /d/M/s/A/evdev2
2012-08-29 20:02:46 /d/M/s/A/evdev2/temp
2012-08-29 20:02:46 /d/M/s/A/evdev2
2012-08-29 19:52:46 /d/M/s/A/evdev2/temp
2012-08-29 19:52:46 /d/M/s/A/evdev2
2012-08-29 19:42:46 /d/M/s/A/evdev2/temp
2012-08-29 19:42:46 /d/M/s/A/evdev2
2012-08-29 19:35:51 /d/M/s/A/evdev2/temp
2012-08-29 19:35:51 /d/M/s/A/evdev2

Brick evdevlxgl02:/mnt/gluster/brick1
Number of entries: 210
at                    path on brick
-----------------------------------
2012-08-30 12:52:58 /d/M/s/A/evdev2/temp
2012-08-30 12:52:58 /d/M/s/A/evdev2
2012-08-30 12:42:58 /d/M/s/A/evdev2/temp
2012-08-30 12:42:58 /d/M/s/A/evdev2
2012-08-30 12:32:58 /d/M/s/A/evdev2/temp
2012-08-30 12:32:58 /d/M/s/A/evdev2
2012-08-30 12:22:58 /d/M/s/A/evdev2/temp
2012-08-30 12:22:58 /d/M/s/A/evdev2
2012-08-30 12:12:58 /d/M/s/A/evdev2/temp
2012-08-30 12:12:58 /d/M/s/A/evdev2
2012-08-30 12:02:58 /d/M/s/A/evdev2/temp
2012-08-30 12:02:58 /d/M/s/A/evdev2
2012-08-30 11:52:58 /d/M/s/A/evdev2/temp
2012-08-30 11:52:58 /d/M/s/A/evdev2
2012-08-30 11:42:58 /d/M/s/A/evdev2/temp
2012-08-30 11:42:58 /d/M/s/A/evdev2
2012-08-30 11:32:58 /d/M/s/A/evdev2/temp
2012-08-30 11:32:58 /d/M/s/A/evdev2
2012-08-30 11:22:58 /d/M/s/A/evdev2/temp
2012-08-30 11:22:58 /d/M/s/A/evdev2
2012-08-30 11:12:57 /d/M/s/A/evdev2/temp
2012-08-30 11:12:57 /d/M/s/A/evdev2
2012-08-30 11:02:57 /d/M/s/A/evdev2/temp
2012-08-30 11:02:57 /d/M/s/A/evdev2
2012-08-30 10:52:57 /d/M/s/A/evdev2/temp
2012-08-30 10:52:57 /d/M/s/A/evdev2
2012-08-30 10:42:57 /d/M/s/A/evdev2/temp
2012-08-30 10:42:57 /d/M/s/A/evdev2
2012-08-30 10:32:57 /d/M/s/A/evdev2/temp
2012-08-30 10:32:57 /d/M/s/A/evdev2
2012-08-30 10:22:57 /d/M/s/A/evdev2/temp
2012-08-30 10:22:57 /d/M/s/A/evdev2
2012-08-30 10:12:57 /d/M/s/A/evdev2/temp
2012-08-30 10:12:57 /d/M/s/A/evdev2
2012-08-30 10:02:57 /d/M/s/A/evdev2/temp
2012-08-30 10:02:57 /d/M/s/A/evdev2
2012-08-30 09:52:57 /d/M/s/A/evdev2/temp
2012-08-30 09:52:57 /d/M/s/A/evdev2
2012-08-30 09:42:57 /d/M/s/A/evdev2/temp
2012-08-30 09:42:57 /d/M/s/A/evdev2
2012-08-30 09:32:57 /d/M/s/A/evdev2/temp
2012-08-30 09:32:57 /d/M/s/A/evdev2
2012-08-30 09:22:57 /d/M/s/A/evdev2/temp
2012-08-30 09:22:57 /d/M/s/A/evdev2
2012-08-30 09:12:56 /d/M/s/A/evdev2/temp
2012-08-30 09:12:56 /d/M/s/A/evdev2
2012-08-30 09:02:56 /d/M/s/A/evdev2/temp
2012-08-30 09:02:56 /d/M/s/A/evdev2
2012-08-30 08:52:56 /d/M/s/A/evdev2/temp
2012-08-30 08:52:56 /d/M/s/A/evdev2
2012-08-30 08:42:56 /d/M/s/A/evdev2/temp
2012-08-30 08:42:56 /d/M/s/A/evdev2
2012-08-30 08:32:56 /d/M/s/A/evdev2/temp
2012-08-30 08:32:56 /d/M/s/A/evdev2
2012-08-30 08:22:56 /d/M/s/A/evdev2/temp
2012-08-30 08:22:56 /d/M/s/A/evdev2
2012-08-30 08:12:56 /d/M/s/A/evdev2/temp
2012-08-30 08:12:56 /d/M/s/A/evdev2
2012-08-30 08:02:56 /d/M/s/A/evdev2/temp
2012-08-30 08:02:56 /d/M/s/A/evdev2
2012-08-30 07:52:56 /d/M/s/A/evdev2/temp
2012-08-30 07:52:56 /d/M/s/A/evdev2
2012-08-30 07:42:56 /d/M/s/A/evdev2/temp
2012-08-30 07:42:56 /d/M/s/A/evdev2
2012-08-30 07:32:56 /d/M/s/A/evdev2/temp
2012-08-30 07:32:56 /d/M/s/A/evdev2
2012-08-30 07:22:56 /d/M/s/A/evdev2/temp
2012-08-30 07:22:56 /d/M/s/A/evdev2
2012-08-30 07:12:56 /d/M/s/A/evdev2/temp
2012-08-30 07:12:56 /d/M/s/A/evdev2
2012-08-30 07:02:55 /d/M/s/A/evdev2/temp
2012-08-30 07:02:55 /d/M/s/A/evdev2
2012-08-30 06:52:55 /d/M/s/A/evdev2/temp
2012-08-30 06:52:55 /d/M/s/A/evdev2
2012-08-30 06:42:55 /d/M/s/A/evdev2/temp
2012-08-30 06:42:55 /d/M/s/A/evdev2
2012-08-30 06:32:55 /d/M/s/A/evdev2/temp
2012-08-30 06:32:55 /d/M/s/A/evdev2
2012-08-30 06:22:55 /d/M/s/A/evdev2/temp
2012-08-30 06:22:55 /d/M/s/A/evdev2
2012-08-30 06:12:55 /d/M/s/A/evdev2/temp
2012-08-30 06:12:55 /d/M/s/A/evdev2
2012-08-30 06:02:55 /d/M/s/A/evdev2/temp
2012-08-30 06:02:55 /d/M/s/A/evdev2
2012-08-30 05:52:55 /d/M/s/A/evdev2/temp
2012-08-30 05:52:55 /d/M/s/A/evdev2
2012-08-30 05:42:55 /d/M/s/A/evdev2/temp
2012-08-30 05:42:55 /d/M/s/A/evdev2
2012-08-30 05:32:55 /d/M/s/A/evdev2/temp
2012-08-30 05:32:55 /d/M/s/A/evdev2
2012-08-30 05:22:55 /d/M/s/A/evdev2/temp
2012-08-30 05:22:55 /d/M/s/A/evdev2
2012-08-30 05:12:55 /d/M/s/A/evdev2/temp
2012-08-30 05:12:55 /d/M/s/A/evdev2
2012-08-30 05:02:55 /d/M/s/A/evdev2/temp
2012-08-30 05:02:55 /d/M/s/A/evdev2
2012-08-30 04:52:54 /d/M/s/A/evdev2/temp
2012-08-30 04:52:54 /d/M/s/A/evdev2
2012-08-30 04:42:54 /d/M/s/A/evdev2/temp
2012-08-30 04:42:54 /d/M/s/A/evdev2
2012-08-30 04:32:54 /d/M/s/A/evdev2/temp
2012-08-30 04:32:54 /d/M/s/A/evdev2
2012-08-30 04:22:54 /d/M/s/A/evdev2/temp
2012-08-30 04:22:54 /d/M/s/A/evdev2
2012-08-30 04:12:54 /d/M/s/A/evdev2/temp
2012-08-30 04:12:54 /d/M/s/A/evdev2
2012-08-30 04:02:54 /d/M/s/A/evdev2/temp
2012-08-30 04:02:54 /d/M/s/A/evdev2
2012-08-30 03:52:54 /d/M/s/A/evdev2/temp
2012-08-30 03:52:54 /d/M/s/A/evdev2
2012-08-30 03:42:54 /d/M/s/A/evdev2/temp
2012-08-30 03:42:54 /d/M/s/A/evdev2
2012-08-30 03:32:54 /d/M/s/A/evdev2/temp
2012-08-30 03:32:54 /d/M/s/A/evdev2
2012-08-30 03:22:54 /d/M/s/A/evdev2/temp
2012-08-30 03:22:54 /d/M/s/A/evdev2
2012-08-30 03:12:53 /d/M/s/A/evdev2/temp
2012-08-30 03:12:53 /d/M/s/A/evdev2
2012-08-30 03:02:53 /d/M/s/A/evdev2/temp
2012-08-30 03:02:53 /d/M/s/A/evdev2
2012-08-30 02:52:53 /d/M/s/A/evdev2/temp
2012-08-30 02:52:53 /d/M/s/A/evdev2
2012-08-30 02:42:53 /d/M/s/A/evdev2/temp
2012-08-30 02:42:53 /d/M/s/A/evdev2
2012-08-30 02:32:53 /d/M/s/A/evdev2/temp
2012-08-30 02:32:53 /d/M/s/A/evdev2
2012-08-30 02:22:53 /d/M/s/A/evdev2/temp
2012-08-30 02:22:53 /d/M/s/A/evdev2
2012-08-30 02:12:53 /d/M/s/A/evdev2/temp
2012-08-30 02:12:53 /d/M/s/A/evdev2
2012-08-30 02:02:53 /d/M/s/A/evdev2/temp
2012-08-30 02:02:53 /d/M/s/A/evdev2
2012-08-30 01:52:53 /d/M/s/A/evdev2/temp
2012-08-30 01:52:53 /d/M/s/A/evdev2
2012-08-30 01:42:52 /d/M/s/A/evdev2/temp
2012-08-30 01:42:52 /d/M/s/A/evdev2
2012-08-30 01:32:52 /d/M/s/A/evdev2/temp
2012-08-30 01:32:52 /d/M/s/A/evdev2
2012-08-30 01:22:52 /d/M/s/A/evdev2/temp
2012-08-30 01:22:52 /d/M/s/A/evdev2
2012-08-30 01:12:52 /d/M/s/A/evdev2/temp
2012-08-30 01:12:52 /d/M/s/A/evdev2
2012-08-30 01:02:52 /d/M/s/A/evdev2/temp
2012-08-30 01:02:52 /d/M/s/A/evdev2
2012-08-30 00:52:52 /d/M/s/A/evdev2/temp
2012-08-30 00:52:52 /d/M/s/A/evdev2
2012-08-30 00:42:52 /d/M/s/A/evdev2/temp
2012-08-30 00:42:52 /d/M/s/A/evdev2
2012-08-30 00:32:52 /d/M/s/A/evdev2/temp
2012-08-30 00:32:52 /d/M/s/A/evdev2
2012-08-30 00:22:52 /d/M/s/A/evdev2/temp
2012-08-30 00:22:52 /d/M/s/A/evdev2
2012-08-30 00:12:52 /d/M/s/A/evdev2/temp
2012-08-30 00:12:52 /d/M/s/A/evdev2
2012-08-30 00:02:52 /d/M/s/A/evdev2/temp
2012-08-30 00:02:52 /d/M/s/A/evdev2
2012-08-29 23:52:52 /d/M/s/A/evdev2/temp
2012-08-29 23:52:52 /d/M/s/A/evdev2
2012-08-29 23:42:51 /d/M/s/A/evdev2/temp
2012-08-29 23:42:51 /d/M/s/A/evdev2
2012-08-29 23:32:51 /d/M/s/A/evdev2/temp
2012-08-29 23:32:51 /d/M/s/A/evdev2
2012-08-29 23:22:51 /d/M/s/A/evdev2/temp
2012-08-29 23:22:51 /d/M/s/A/evdev2
2012-08-29 23:12:51 /d/M/s/A/evdev2/temp
2012-08-29 23:12:51 /d/M/s/A/evdev2
2012-08-29 23:02:51 /d/M/s/A/evdev2/temp
2012-08-29 23:02:51 /d/M/s/A/evdev2
2012-08-29 22:52:51 /d/M/s/A/evdev2/temp
2012-08-29 22:52:51 /d/M/s/A/evdev2
2012-08-29 22:42:51 /d/M/s/A/evdev2/temp
2012-08-29 22:42:51 /d/M/s/A/evdev2
2012-08-29 22:32:51 /d/M/s/A/evdev2/temp
2012-08-29 22:32:51 /d/M/s/A/evdev2
2012-08-29 22:22:51 /d/M/s/A/evdev2/temp
2012-08-29 22:22:51 /d/M/s/A/evdev2
2012-08-29 22:12:51 /d/M/s/A/evdev2/temp
2012-08-29 22:12:51 /d/M/s/A/evdev2
2012-08-29 22:02:51 /d/M/s/A/evdev2/temp
2012-08-29 22:02:51 /d/M/s/A/evdev2
2012-08-29 21:52:50 /d/M/s/A/evdev2/temp
2012-08-29 21:52:50 /d/M/s/A/evdev2
2012-08-29 21:42:50 /d/M/s/A/evdev2/temp
2012-08-29 21:42:50 /d/M/s/A/evdev2
2012-08-29 21:32:50 /d/M/s/A/evdev2/temp
2012-08-29 21:32:50 /d/M/s/A/evdev2
2012-08-29 21:22:50 /d/M/s/A/evdev2/temp
2012-08-29 21:22:50 /d/M/s/A/evdev2
2012-08-29 21:12:50 /d/M/s/A/evdev2/temp
2012-08-29 21:12:50 /d/M/s/A/evdev2
2012-08-29 21:02:50 /d/M/s/A/evdev2/temp
2012-08-29 21:02:50 /d/M/s/A/evdev2
2012-08-29 20:52:50 /d/M/s/A/evdev2/temp
2012-08-29 20:52:50 /d/M/s/A/evdev2
2012-08-29 20:42:50 /d/M/s/A/evdev2/temp
2012-08-29 20:42:50 /d/M/s/A/evdev2
2012-08-29 20:32:50 /d/M/s/A/evdev2/temp
2012-08-29 20:32:50 /d/M/s/A/evdev2
2012-08-29 20:22:50 /d/M/s/A/evdev2/temp
2012-08-29 20:22:50 /d/M/s/A/evdev2
2012-08-29 20:12:50 /d/M/s/A/evdev2/temp
2012-08-29 20:12:50 /d/M/s/A/evdev2
2012-08-29 20:02:50 /d/M/s/A/evdev2/temp
2012-08-29 20:02:50 /d/M/s/A/evdev2
2012-08-29 19:52:49 /d/M/s/A/evdev2/temp
2012-08-29 19:52:49 /d/M/s/A/evdev2
2012-08-29 19:42:49 /d/M/s/A/evdev2/temp
2012-08-29 19:42:49 /d/M/s/A/evdev2
2012-08-29 19:32:49 <gfid:ed6ddd65-b702-486c-b3f7-08cf6fb1a952>
2012-08-29 19:32:49 <gfid:48fb419b-9c16-448d-98ac-f093c850e0f3>

Brick evdevlxgl03:/mnt/gluster/brick1
Number of entries: 0

Brick evdevlxgl04:/mnt/gluster/brick1
Number of entries: 0
Comment 3 Rob.Hendelman 2012-09-04 08:37:16 EDT
Any update(s) on this?  Do you need any assistance from my side? 

Let me know how I can assist.

Thanks,

Robert
Comment 4 Pranith Kumar K 2012-09-22 23:41:25 EDT
Rob,
  Extremely sorry for the delay in response.
  "volume heal <volname> *" commands work in best effort manner to show the file-paths. The information about what files need to be healed is stored in the form of gfids and not as path-names, because if it stores the path-names then It should also keep track of renames, delete, create of same file-path etc which makes things really complicated from a design point of view. That said, if the brick which keeps track of files that need to be healed is not re-started it will give the path corresponding to a gfid with VERY high probability from its inode structures. If you MUST see the file-paths, then you can always do a "find <mount-path> | xargs stat" to update these inode-structures and from then on you should be able to see the paths.

    For your issue of same entries repeating over and over in the info split-brain:
    Every-time 10 minutes self-heal-daemon tries to heal the volume and it seems every-time it is not able to completely heal the directory because of split-brain so you are observing the same entry over and over. Could you post the output of "getfattr -d -m . -e hex <path>" for both of these directories and every entry in that directory, not recursively, just its entries should suffice. I need to check if there are any entries with same name but different gfids in those directories to see if it went into an un-healable directory scenario.

Pranith.
Comment 5 Rob.Hendelman 2012-09-25 13:51:46 EDT
evdevlxgl01:

root@evdevlxgl01:~# getfattr -d -m . -e hex /mnt/gluster/brick1
getfattr: Removing leading '/' from absolute path names
# file: mnt/gluster/brick1
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.volume-id=0x4e92f7eaeed1484c967fb2ccc4a832b7

evdevlxgl02:

root@evdevlxgl02:/mnt/gluster/brick1# getfattr -d -m . -e hex /mnt/gluster/brick1
getfattr: Removing leading '/' from absolute path names
# file: mnt/gluster/brick1
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.volume-id=0x4e92f7eaeed1484c967fb2ccc4a832b7

evdevlxgl01:

root@evdevlxgl01:/mnt/gluster/brick1# find . -maxdepth 1 -exec getfattr -d -m . -e hex {} +
# file: .
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.volume-id=0x4e92f7eaeed1484c967fb2ccc4a832b7

# file: src.dd
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0xd208f16e7dbe4bff85c5a3dbf434b7de

# file: src
trusted.gfid=0xd97e6cb42176479597b169fad43dcf8d
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe

# file: hendero
trusted.gfid=0xfe621f460e864b56881c4b898a848264
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

# file: GearsSetup.exe
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0x511fb25158a545bf8e5c67b236617fd7

# file: lost+found
trusted.gfid=0x0e52f3d41c794269a3223231dc6bbea4
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe

# file: data
trusted.gfid=0xec2395fbe4ce49f08b2fc51215b8d428
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

# file: old-vm-gluster
trusted.gfid=0x6ba3934abfb14d73a9d55fd5d9986721
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

# file: sae_test
trusted.gfid=0x67152c40f3714fe982546c2815451c2f
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

evdevlxgl02:

root@evdevlxgl02:/mnt/gluster/brick1# find . -maxdepth 1 -exec getfattr -d -m . -e hex {} +
# file: .
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.volume-id=0x4e92f7eaeed1484c967fb2ccc4a832b7

# file: sae_test
trusted.gfid=0x67152c40f3714fe982546c2815451c2f
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

# file: data
trusted.gfid=0xec2395fbe4ce49f08b2fc51215b8d428
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

# file: src.dd
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0xd208f16e7dbe4bff85c5a3dbf434b7de

# file: GearsSetup.exe
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0x511fb25158a545bf8e5c67b236617fd7

# file: hendero
trusted.gfid=0xfe621f460e864b56881c4b898a848264
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

# file: src
trusted.gfid=0xd97e6cb42176479597b169fad43dcf8d
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe

# file: lost+found
trusted.gfid=0x0e52f3d41c794269a3223231dc6bbea4
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe

# file: old-vm-gluster
trusted.gfid=0x6ba3934abfb14d73a9d55fd5d9986721
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
Comment 6 Pranith Kumar K 2012-09-27 05:43:46 EDT
Rob,
   I was asking for the ouputs of directories
/d/M/s/A/evdev2/temp
/d/M/s/A/evdev2

It seems you have attached the outputs of the brick root directories. Could you post the output of the directories above.

Pranith
Comment 7 Rob.Hendelman 2012-09-27 09:44:12 EDT
on /d/M/s/A/evdev2/
=====================
devlxgl01 (pwd /d/M/s/A/evdev2)
=====================
getfattr -d -m . -e hex .
# file: .
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000300000001
trusted.gfid=0x48fb419b9c16448d98acf093c850e0f3
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
=====================
devlxgl02 (pwd /d/M/s/A/evdev2)
=====================
getfattr -d -m . -e hex .
# file: .
trusted.afr.raid10-client-0=0x0000000000000001000000ac
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0x48fb419b9c16448d98acf093c850e0f3
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe


=====================
on /d/M/s/A/evdev2/temp
=====================
devlxgl01 (pwd /d/M/s/A/evdev2/temp)
=====================
# file: .
trusted.afr.raid10-client-0=0x000000000000000000000000
trusted.afr.raid10-client-1=0x000000000000000400000000
trusted.gfid=0xed6ddd65b702486cb3f708cf6fb1a952
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
====================
devlxgl02 (pwd /d/M/s/A/evdev2/temp)
=====================
# file: .
trusted.afr.raid10-client-0=0x000000000000000200000000
trusted.afr.raid10-client-1=0x000000000000000000000000
trusted.gfid=0xed6ddd65b702486cb3f708cf6fb1a952
Comment 8 Pranith Kumar K 2012-12-17 07:03:22 EST
hi Rob,
Extremely sorry for the delay in response.
Entry self-heal for the directory is not happening because there is a meta-data split-brain on the directories. You need to fix the metadata split-brain manually, to trigger the self-heal.

Fix for the issue of meta-data split-brain blocking entry/data self-heal is already sent as part of the fix to 851451.
Comment 9 Rob.Hendelman 2012-12-17 12:22:47 EST
(In reply to comment #8)
> hi Rob,
> Extremely sorry for the delay in response.
> Entry self-heal for the directory is not happening because there is a
> meta-data split-brain on the directories. You need to fix the metadata
> split-brain manually, to trigger the self-heal.
> 
> Fix for the issue of meta-data split-brain blocking entry/data self-heal is
> already sent as part of the fix to 851451.

So is this fixed in a newer version/commit?  If so can you point me to it?  Looking at that bug looks like it isn't part of 3.3...

Thank you very much,

Robert
Comment 10 Pranith Kumar K 2013-01-11 02:20:05 EST
http://review.gluster.org/3853

Closing the bug.

Note You need to log in before you can comment on or make changes to this bug.