Bug 1303817

Summary: [Tiering]: Detach tier failed on two nodes as a result of fix layout failure
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: krishnaram Karthick <kramdoss>
Component: distributeAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED WONTFIX QA Contact: storage-qa-internal <storage-qa-internal>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: amukherj, dlambrig, nbalacha
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: tier-attach-detach
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-22 08:08:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description krishnaram Karthick 2016-02-02 05:34:01 UTC 
Description of problem:

On a 16 node system, when detach tier failed on two nodes as a result of fix layout fail. Fix layout could have failed as a result of rm -rf on a directory with files and sub directories. 

Failure of fix layout shouldn't fail rebalance.

[2016-02-02 04:20:00.214345] E [MSGID: 109016] [dht-rebalance.c:3120:gf_defrag_fix_layout] 0-krk-vol-tier-dht: Fix layout failed for /untar/linux-4.4/arch/mips/pnx833x
[2016-02-02 04:20:00.215683] E [MSGID: 109016] [dht-rebalance.c:3120:gf_defrag_fix_layout] 0-krk-vol-tier-dht: Fix layout failed for /untar/linux-4.4/arch/mips
[2016-02-02 04:20:00.217183] E [MSGID: 109016] [dht-rebalance.c:3120:gf_defrag_fix_layout] 0-krk-vol-tier-dht: Fix layout failed for /untar/linux-4.4/arch
[2016-02-02 04:20:00.218029] E [MSGID: 109016] [dht-rebalance.c:3120:gf_defrag_fix_layout] 0-krk-vol-tier-dht: Fix layout failed for /untar/linux-4.4
[2016-02-02 04:20:00.219747] E [MSGID: 109016] [dht-rebalance.c:3120:gf_defrag_fix_layout] 0-krk-vol-tier-dht: Fix layout failed for /untar
[2016-02-02 04:20:00.221389] I [dht-rebalance.c:2109:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 3
[2016-02-02 04:20:00.222006] I [dht-rebalance.c:2109:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 4
[2016-02-02 04:20:00.222412] I [MSGID: 109028] [dht-rebalance.c:3619:gf_defrag_status_get] 0-krk-vol-tier-dht: Rebalance is failed. Time taken is 195.00 secs


Version-Release number of selected component (if applicable):
glusterfs-3.7.5-17.el7rhgs.x86_64

How reproducible:
Yet to be determined

Steps to Reproduce:
1. On a tiered volume, have a directory with multiple subdirectories - (kernel untar for ex)
2. Issue rm -rf on this directory and start detach tier process
3. Wait for detach tier process to complete

Actual results:
Detach tier failed on two nodes and couple of files were not mobed from hot tier to cold tier

Expected results:
Detach tier should succeed on all nodes and fix layout failure should not cause rebalance to fail

Additional info:
sosreport shall be atatched