Bug 1090986 - DHT + Snapshot :- If snapshot is taken when Directory is created only on hashed sub-vol; On restoring that snapshot Directory is not listed on mount point and lookup on parent is not healing
Summary: DHT + Snapshot :- If snapshot is taken when Directory is created only on hash...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: rhgs-3.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: RHGS 3.0.0
Assignee: Susant Kumar Palai
QA Contact: amainkar
URL:
Whiteboard:
Depends On:
Blocks: 1092433 1092501 1139103 1139986
TreeView+ depends on / blocked
 
Reported: 2014-04-24 14:24 UTC by Rachana Patel
Modified: 2015-05-13 16:57 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.6.0.27-1
Doc Type: Bug Fix
Doc Text:
Previously the directory entries were added only from the first_up_subvol(brick online for the longest duration of time). If the directory was created on a hashed-brick and soon after if a snapshot was taken, the restored snapshot mount point did not list the directory entry. With this fix, the directory entries are filtered from the corresponding hashed subvolumes. Only in case of a hashed subvolume having a NULL value (either because of a layout anomaly or the hashed volume being offline), the entry is filtered from the first_up_subvol.
Clone Of:
: 1092433 (view as bug list)
Environment:
Last Closed: 2014-09-22 19:36:03 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2014:1278 0 normal SHIPPED_LIVE Red Hat Storage Server 3.0 bug fix and enhancement update 2014-09-22 23:26:55 UTC

Description Rachana Patel 2014-04-24 14:24:50 UTC
Description of problem:
=======================
Take Snapshot when Directory is created only on  hashed sub-volume and not on others.
On restoring that snapshot, Directory is not listed on mount point. Lookup on parent Directory is not healing Directory on non-hashed sub-volume.


Version-Release number of selected component (if applicable):
=============================================================
3.5qa2-0.340.gitc193996.el6_5.x86_64


How reproducible:
================
always



Steps to Reproduce:
1. create Distributed volume, start it and FUSE mount it.
2. create Directory from mount point and make sure you take a snap of volume when Directory is created only on hashed sub-volume and not on any other
3. stop volume and restore snap
4. mount volume again and list out content of parent Directory.
newly created Directory is not listed

[root@rhs-client18 new]# ls -l
total 0

5. verify Directory in backend.
Directory is present only on hashed sub-volume, not on other sub-volumes

server1:-
[root@rhs-client18 new]# ls -l
total 0

server2:-
[root@OVM5 brick3]# ls -l
total 0
drwxr-xr-x 2 root root 6 Apr 24 15:01 dir1

server3:-
[root@rhs-client18 new]# ls -l
total 0



Actual results:
===============
DHT self heal is not healing Directory entry and Directory is not visible on mount point

Expected results:
=================
Lookup on mount point should heal Directory entry on all up sub-volume

Additional info:

Comment 2 Rachana Patel 2014-04-29 09:26:27 UTC
Able to reproduce without taking snapshot. 
Steps:-
1) send Directory creation from one mount point.
2) When Directory is created on hashed and not not creates on other sub-volumes(non hashed), send lookup from other mount point

--> lookup is not healin Directory on other sub-volumes

Comment 3 Krutika Dhananjay 2014-05-22 07:38:54 UTC
*** Bug 1092501 has been marked as a duplicate of this bug. ***

Comment 5 Susant Kumar Palai 2014-06-03 06:13:06 UTC
Upstream patch : http://review.gluster.org/#/c/7599/

Comment 11 Nagaprasad Sathyanarayana 2014-06-17 10:48:26 UTC
https://code.engineering.redhat.com/gerrit/#/c/26144/

Comment 12 Rachana Patel 2014-06-26 12:13:11 UTC
verified with build 3.6.0.20-1.el6rhs.x86_64
works as per expectation hence moving to verified

Comment 13 Pavithra 2014-07-23 06:13:42 UTC
Hi Susant,

Please review the edited doc text for technical accuracy and sign off.

Comment 14 Pavithra 2014-07-23 06:13:57 UTC
Hi Susant,

Please review the edited doc text for technical accuracy and sign off.

Comment 15 Vijaykumar Koppad 2014-08-01 09:32:25 UTC
I was able to hit this issue on the build glusterfs-3.6.0.25-1.

1. create and start Distributed replicate master and slave volume, and create geo-rep relationship between master and slave (looks like geo-rep has nothing to do with this issue)
2. create Directory from mount point on master and make sure you take a snap of volume when Directory is created only on one of the sub-volume and not on any other.
3. restore snap (follow steps to restore snap)
4. mount volume again and list out content of parent Directory.
newly created Directory is not listed.

Comment 16 Kotresh HR 2014-08-01 11:55:19 UTC
I also could be able to hit this during snapshot restoration of geo-rep slave and master. A directory in restored master volume was only present in one brick and not on other. Gluster mount could not see the directory. Lookup on parent directory did not heal it but the lookup on the missing directory healed.

Comment 17 Susant Kumar Palai 2014-08-07 09:41:54 UTC
Root Cause of the issue:

Due to snapshot and mkdir race, snapshot took the entry from the non-hashed and not hashed. As dht_readdirp fop depends on the entry being present on the hashed subvolume to be filtered, the entry was not shown on the mount point and also not healed for the master volume.
[Hence, not a regression]

Relation to geo-rep:

The changlogs are captured at the brick level at the master and since one of the bricks has the directory entry, geo-rep syncs the directory on to the slave. Now the inconsistency being the slave has an entry which the master does not have from the gluster mount point.

 So closing this bug as it is not a regression and a new bug need to be created to track the geo-rep inconsistencies.

Comment 18 Susant Kumar Palai 2014-08-08 04:47:53 UTC
Moving this to ON_QA. If the snapshot has captured the entry from the mkdir on the hashed subvol and still the entry does not get listed on the mount point reopen this bug.

Comment 19 Vijaykumar Koppad 2014-08-08 12:50:59 UTC
Issue mentioned in the comment 17 is being tracked with the Bug 1128155

Comment 20 Rachana Patel 2014-09-18 08:47:07 UTC
verified with 3.6.0.28-1.el6rhs.x86_64, working as expected hence moving to verified

Comment 22 errata-xmlrpc 2014-09-22 19:36:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-1278.html


Note You need to log in before you can comment on or make changes to this bug.