Bug 1272036

Summary: Data Tiering:getting failed to fsync on germany-hot-dht (Structure needs cleaning) warning
Product: [Community] GlusterFS Reporter: Nag Pavan Chilakam <nchilaka>
Component: tieringAssignee: Nithya Balachandran <nbalacha>
Status: CLOSED CURRENTRELEASE QA Contact: bugs <bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.7.5CC: bugs, vagarwal
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1273372 1274595 (view as bug list) Environment:
Last Closed: 2015-11-17 06:00:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1273372, 1274595, 1275914    

Description Nag Pavan Chilakam 2015-10-15 10:36:42 UTC 
Description of problem:
======================

I am seeing getting this error when there is a failure to migrate file to hot tier


In my case following is the way I probably hit it.

I had the following setup:

a hot tier 2x2 setup with each 1GB bricks
over a cold tier 2x2 of 500GB space

I then created files f1 of 700MB and then after f1 got demoted i created f2 of 700MB. When even f2 got demoted, I touched both files to get them promoted to hot brick, and see what happens as it can accomodate only 1GB data.

I then saw  logs and found this error.





Version-Release number of selected component (if applicable):
==============================================================
glusterfs-server-3.7.5-0.18.gitb8ba012.el7.centos.x86_64




[2015-10-14 12:12:04.867032] W [dht-rebalance.c:1247:dht_migrate_file] 0-germany-tier-dht: /berlin.7: failed to fsync on germany-hot-dht (Structure needs cleaning)
[2015-10-14 13:14:08.219263] W [dht-rebalance.c:1247:dht_migrate_file] 0-greece-tier-dht: /athens.10: failed to fsync on greece-hot-dht (Structure needs cleaning)
[2015-10-14 13:28:25.074476] W [dht-rebalance.c:1247:dht_migrate_file] 0-greece-tier-dht: /athens.2: failed to fsync on greece-hot-dht (Structure needs cleaning)
[2015-10-14 13:30:26.091045] W [dht-rebalance.c:1247:dht_migrate_file] 0-greece-tier-dht: /athens.10: failed to fsync on greece-hot-dht (Structure needs cleaning)
[2015-10-14 13:34:26.703553] W [dht-rebalance.c:1247:dht_migrate_file] 0-greece-tier-dht: /athens.2: failed to fsync on greece-hot-dht (Structure needs cleaning)
[2015-10-14 13:34:51.579771] W [dht-rebalance.c:1247:dht_migrate_file] 0-greece-tier-dht: /athens.3: failed to fsync on greece-hot-dht (Structure needs cleaning)
[2015-10-14 13:38:26.134830] W [dht-rebalance.c:1247:dht_migrate_file] 0-greece-tier-dht: /athens.4: failed to fsync on greece-hot-dht (Structure needs cleaning)
[2015-10-14 13:44:24.459860] W [dht-rebalance.c:1247:dht_migrate_file] 0-greece-tier-dht: /athens.3: failed to fsync on greece-hot-dht (Structure needs cleaning)



attaching sos report:
Kindly refer to "greece" volume tier logs
Comment 1 Nag Pavan Chilakam 2015-10-15 10:43:00 UTC 
sosreprots

rhsqe-repo.lab.eng.blr.redhat.com:/home/repo/sosreports/bug.1272036
[root@rhsqe-repo sosreports]#
Comment 2 Vijay Bellur 2015-10-21 11:19:52 UTC 
REVIEW: http://review.gluster.org/12408 (cluster/dht : op_ret not set correctly in dht_fsync_cbk) posted (#1) for review on release-3.7 by N Balachandran (nbalacha)
Comment 3 Vijay Bellur 2015-10-21 11:21:37 UTC 
REVIEW: http://review.gluster.org/12409 (cluster/dht : op_ret not set correctly in dht_fsync_cbk) posted (#1) for review on release-3.7 by N Balachandran (nbalacha)
Comment 4 Vijay Bellur 2015-10-21 15:17:55 UTC 
REVIEW: http://review.gluster.org/12409 (cluster/dht : op_ret not set correctly in dht_fsync_cbk) posted (#2) for review on release-3.7 by N Balachandran (nbalacha)
Comment 5 Vijay Bellur 2015-10-21 16:58:06 UTC 
REVIEW: http://review.gluster.org/12409 (cluster/dht : op_ret not set correctly in dht_fsync_cbk) posted (#3) for review on release-3.7 by N Balachandran (nbalacha)
Comment 6 Vijay Bellur 2015-10-24 16:22:51 UTC 
REVIEW: http://review.gluster.org/12409 (cluster/dht : op_ret not set correctly in dht_fsync_cbk) posted (#4) for review on release-3.7 by N Balachandran (nbalacha)
Comment 7 Vijay Bellur 2015-10-25 13:41:35 UTC 
COMMIT: http://review.gluster.org/12409 committed in release-3.7 by Dan Lambright (dlambrig) 
------
commit fca25fbd0d24b6eb944415174a67d1774b5d2479
Author: N Balachandran <nbalacha>
Date:   Wed Oct 21 16:50:41 2015 +0530

    cluster/dht : op_ret not set correctly in dht_fsync_cbk
    
    local->op_ret was not set correctly in dht_fsync_cbk in case
    the file was being migrated
    
    > Change-Id: If73ae04368ea0c7f6868c8704dfc2deb2faee753
    > Signed-off-by: N Balachandran <nbalacha>
    > Reviewed-on: http://review.gluster.org/12401
    > Tested-by: Gluster Build System <jenkins.com>
    > Reviewed-by: Raghavendra G <rgowdapp>
    > Reviewed-by: Jeff Darcy <jdarcy>
    (cherry picked from commit 9710f58e5874bccb4b328abef80ea226ccf9c798)
    
    Change-Id: I2addb86083c1d8305cf91e0b0385deeb227216c8
    BUG: 1272036
    Signed-off-by: N Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/12409
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Dan Lambright <dlambrig>
Comment 8 Raghavendra Talur 2015-11-17 06:00:20 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.6, please open a new bug report.

glusterfs-3.7.6 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-November/024359.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user