Bug 1191413

Summary:	Geo-Replication : shows faulty with lots of error messages "ssh> tar: .gfid/<real GFID>: Cannot open: Structure needs cleaning"
Product:	[Community] GlusterFS	Reporter:	Kotresh HR <khiremat>
Component:	geo-replication	Assignee:	Kotresh HR <khiremat>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	mainline	CC:	aavati, asrivast, avishwan, bkunal, bugs, csaba, gluster-bugs, mmuir, nbalacha, nlevinki, nsathyan, rgowdapp, smanjara, storage-qa-internal, vbellur, vumrao
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.0beta1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1186705	Environment:
Last Closed:	2015-05-14 17:26:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1181410, 1186705
Bug Blocks:

Comment 1 Kotresh HR 2015-02-11 09:52:21 UTC

+++ This bug was initially created as a clone of Bug #1186705 +++

+++ This bug was initially created as a clone of Bug #1181410 +++

Description of problem:

Geo-Replication : shows faulty and not working 

with lots of error messages like :


[2015-01-09 14:27:02.747138] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/c19915e6-4570-4f36-94e3-6416a704a458: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.747344] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/2619de29-3fd6-4cf7-a4c9-a6eb91b3f5c1: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.747560] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/6621ece0-00a0-48f7-a747-f0080468965f: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.747853] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/7cd443a0-0067-49c9-9550-12573a12ddef: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.748079] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/a46e5995-9363-4bac-ab8d-521491a732de: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.748303] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/c6e10008-e60b-451a-b278-b70a0111d8c1: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.748524] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/946c71ed-7880-420f-903c-afb53fd36090: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.748773] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/93f05242-9dc3-423f-8229-f21c4ff60382: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.748979] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/d69e49e9-3e6f-4525-95a7-f736f587487d: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.749191] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/05bf19af-bed9-421a-bf26-0164881aef08: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.749413] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/006ed177-7af9-41bc-a25d-775644f6727a: Cannot open: Structure needs cleaning
[2015-01-09 14:27:02.749617] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: .gfid/66b9ac53-129c-4a61-918f-afa1466a19c9: Cannot open: Structure needs cleaning


==========================================================================

tar+ssh start hybrid crawl but because of above given errors going faulty.

[2015-01-09 14:27:02.750931] E [resource(/data/master/dp-vol):207:logerr] Popen: ssh> tar: Exiting with failure status due to previous errors
[2015-01-09 14:27:02.751357] I [syncdutils(/data/master/dp-vol):159:finalize] <top>: exiting.
[2015-01-09 14:27:02.764004] I [monitor(monitor):81:set_state] Monitor: new state: faulty

Comment 2 Anand Avati 2015-02-11 10:06:48 UTC

REVIEW: http://review.gluster.org/9634 (feature/gfid-access: Send a named lookup before trying to create a file.) posted (#1) for review on master by Kotresh HR (khiremat)

Comment 3 Anand Avati 2015-02-12 06:30:23 UTC

REVIEW: http://review.gluster.org/9634 (feature/gfid-access: Send a named lookup before trying to create a file.) posted (#2) for review on master by Kotresh HR (khiremat)

Comment 4 Anand Avati 2015-02-16 06:07:31 UTC

REVIEW: http://review.gluster.org/9634 (feature/gfid-access: Send a named lookup before trying to create a file.) posted (#3) for review on master by Kotresh HR (khiremat)

Comment 5 Anand Avati 2015-02-16 15:48:17 UTC

COMMIT: http://review.gluster.org/9634 committed in master by Raghavendra G (rgowdapp) 
------
commit e04039092d28dde70a9a394010cd04cf0a12726a
Author: Kotresh HR <khiremat>
Date:   Tue Feb 10 16:43:50 2015 +0530

    feature/gfid-access: Send a named lookup before trying to create a file.
    
    Normally a named-lookup is done by the kernel on an entry before it
    issues a dentry creation fop like create/mknod etc. This will enable
    cluster translators like dht to maintain internal consistency like
    deleting a linkto file if no corresponding datafile is present etc.
    While handling file creation on auxiliary gfid mounts, we issue dentry
    creation fop without issuing a lookup. If there are stale-linkto files,
    creation would fail with EEXIST, however access would fail since there
    is no datafile. A named lookup would cleanup the linkto file allowing
    create to succeed.
    
    Change-Id: I2932107296adac710dd179df7d0946b8697a497a
    BUG: 1191413
    Signed-off-by: Kotresh HR <khiremat>
    Reviewed-on: http://review.gluster.org/9634
    Reviewed-by: Raghavendra G <rgowdapp>
    Tested-by: Raghavendra G <rgowdapp>

Comment 6 Niels de Vos 2015-05-14 17:26:28 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 7 Niels de Vos 2015-05-14 17:28:23 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 8 Niels de Vos 2015-05-14 17:35:16 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user