Bug 1234295

Summary: tiering: gluster v status <vol-name> gives error "commit failed on localhost"
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Anil Shah <ashah>
Component: tierAssignee: Joseph Elwin Fernandes <josferna>
Status: CLOSED CURRENTRELEASE QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: rhs-bugs, sankarshan, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-30 12:39:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1223636    

Description Anil Shah 2015-06-22 10:09:55 UTC
Description of problem:

On Tiered volume, gluster volume status shows error "Commit failed failed on localhost" 

glusterd logs
=========================================

[2015-06-22 09:05:53.710783] I [glusterd-handler.c:4205:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vol0
[2015-06-22 09:05:53.721641] E [MSGID: 106062] [glusterd-op-sm.c:2816:_add_remove_bricks_to_dict] 0-management: Unable to get brick3
[2015-06-22 09:05:53.721691] E [MSGID: 106397] [glusterd-op-sm.c:2863:_add_task_to_dict] 0-management: Failed to add remove bricks to dict
[2015-06-22 09:05:53.721715] E [MSGID: 106060] [glusterd-op-sm.c:2934:glusterd_aggregate_task_status] 0-management: Failed to add task details to dict
[2015-06-22 09:05:53.721740] E [MSGID: 106123] [glusterd-syncop.c:1413:gd_commit_op_phase] 0-management: Commit of operation 'Volume Status' failed on localhost    
[2015-06-22 09:05:53.721740] E [MSGID: 106123] [glusterd-syncop.c:1413:gd_commit_op_phase] 0-management: Commit of operation 'Volume Status' failed on localhost  
Version-Release number of selected component (if applicable):


How reproducible:

1/1

Steps to Reproduce:
1. Create 2*2 distribute-replicate volume
2. add 1*2 replica tier
3. set vol0 cluster.tier-demote-frequency 30 
4. set vol0 features.ctr-enabled 1 
5. Enable uss
6. detach tier 
7 Didn't commit
8 Create snapshot 
9 run gluster v status <volname>

Actual results:

gluster v status command  fails with error "Commit failed on localhost"

glusterd logs:
========================================

[2015-06-22 09:05:53.710783] I [glusterd-handler.c:4205:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vol0
[2015-06-22 09:05:53.721641] E [MSGID: 106062] [glusterd-op-sm.c:2816:_add_remove_bricks_to_dict] 0-management: Unable to get brick3
[2015-06-22 09:05:53.721691] E [MSGID: 106397] [glusterd-op-sm.c:2863:_add_task_to_dict] 0-management: Failed to add remove bricks to dict
[2015-06-22 09:05:53.721715] E [MSGID: 106060] [glusterd-op-sm.c:2934:glusterd_aggregate_task_status] 0-management: Failed to add task details to dict
[2015-06-22 09:05:53.721740] E [MSGID: 106123] [glusterd-syncop.c:1413:gd_commit_op_phase] 0-management: Commit of operation 'Volume Status' failed on localhost    
[2015-06-22 09:05:53.721740] E [MSGID: 106123] [glusterd-syncop.c:1413:gd_commit_op_phase] 0-management: Commit of operation 'Volume Status' failed on localhost  

Expected results:


Additional info:

[root@darkknight ~]# gluster v info vol0
 
Volume Name: vol0
Type: Tier
Volume ID: b9b586fa-4cf7-49e4-aace-cd09a805281f
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 2 = 2
Brick1: 10.70.33.219:/rhs/brick1/SS2
Brick2: 10.70.33.214:/rhs/brick1/SS1
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.33.214:/rhs/brick1/b1
Brick4: 10.70.33.219:/rhs/brick1/b2
Brick5: 10.70.33.225:/rhs/brick1/b3
Brick6: 10.70.44.13:/rhs/brick1/b4
Options Reconfigured:
cluster.entry-self-heal: off
cluster.metadata-self-heal: off
cluster.data-self-heal: off
features.barrier: disable
server.allow-insecure: on
features.show-snapshot-directory: enable
features.ctr-enabled: 1
cluster.tier-demote-frequency: 10
features.uss: enable

Comment 2 Anil Shah 2015-06-22 10:30:29 UTC
logs uploaded @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1234295/