Bug 1559452 - Volume status inode is broken with brickmux
Summary: Volume status inode is broken with brickmux
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: rhgs-3.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: RHGS 3.4.0
Assignee: hari gowtham
QA Contact: Bala Konda Reddy M
URL:
Whiteboard:
Depends On: 1566067 1569336 1569346 1579769
Blocks: 1503137
TreeView+ depends on / blocked
 
Reported: 2018-03-22 15:19 UTC by Rajesh Madaka
Modified: 2018-09-04 06:46 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.12.2-13
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1566067 (view as bug list)
Environment:
Last Closed: 2018-09-04 06:45:40 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 None None None 2018-09-04 06:46:54 UTC

Description Rajesh Madaka 2018-03-22 15:19:17 UTC
Description of problem:

Gluster volume status inode command failing on subsequent volumes.


Version-Release number of selected component (if applicable):

3.8.4-54-2

How reproducible:

Every time

Steps to Reproduce:
1.Create 3 node cluster(n1, n2, n3).
2.Create 2 replica-3 volumes (v1, v2).
3.Mount 2 volumes on two different clients(c1, c2).
4.Start running I/O parallel on two mount points.
5.While running I/O's , start executing 'Gluster volume status v1 inode' and
'Gluster volume status v1 fd' frequently with some time gap
6.In sameway run volume status inode command for v2 also
7.Then create new volume  v3 (distirubted_replicated)
8. Then perform "gluster volume status v3 inode" and "gluster volume status v3 fd" on node n1
9. 'Gluster volume status inode' and 'gluster volume status fd' commands are failing for newly created volume.
10. node n1 bricks of volume v3  went to offline 

Actual results:

root@dhcp37-113 home]# gluster vol status rp1 fd
Error : Request timed out
[root@dhcp37-113 home]# gluster vol status drp1 inode
Error : Request timed out

gluster vol status drp1
Status of volume: drp1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.37.113:/bricks/brick1/drp1      N/A       N/A        N       N/A  
Brick 10.70.37.157:/bricks/brick1/drp1      49152     0          Y       2125 
Brick 10.70.37.174:/bricks/brick1/drp1      49152     0          Y       2306 
Brick 10.70.37.113:/bricks/brick2/drp1      N/A       N/A        N       N/A  
Brick 10.70.37.157:/bricks/brick2/drp1      49152     0          Y       2125 
Brick 10.70.37.174:/bricks/brick2/drp1      49152     0          Y       2306 
Self-heal Daemon on localhost               N/A       N/A        Y       4507 
Self-heal Daemon on 10.70.37.157            N/A       N/A        Y       4006 
Self-heal Daemon on 10.70.37.174            N/A       N/A        Y       4111 
 
Task Status of Volume drp1



Expected results:

Bricks should not go to offline and gluster volume status inode and fd commands should get executed successfully


Additional info:

Comment 2 Rajesh Madaka 2018-03-22 15:24:15 UTC
Changed to correct version-release number

Version-Release number of selected component (if applicable):

3.8.4-54-3

Comment 9 Rajesh Madaka 2018-05-04 06:22:42 UTC
I have followed the steps from description, volume status inode command failing on 3rd volume. changing state to assigned.

Below error message i am getting for 3rd volume

# gluster vol status dstrbrep1 inode

Commit failed on dhcp37-174. Please check log file for details.

Build version:

glusterfs-3.12.2-8

logs pasted below:
-----------------

[2018-05-03 12:44:36.320186] I [MSGID: 106499] [glusterd-handler.c:4383:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dstrbrep1

[2018-05-03 12:44:36.336651] E [MSGID: 106062] [glusterd-utils.c:10024:glusterd_volume_status_copy_to_op_ctx_dict] 0-management: Failed to get other count from rsp_dict

[2018-05-03 12:44:36.336723] E [MSGID: 106108] [glusterd-syncop.c:1146:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from  node/brick

[2018-05-03 12:44:36.336750] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Commit failed on dhcp37-174. Please check log file for details

Comment 10 hari gowtham 2018-05-17 11:51:35 UTC
From the machine I can see that the commit fails only on the remote node and not on the originator. other-count will be set in the commit as commit failed other count wasn't set and it caused the aggregation failure.

The commit phase on the remote node is a combination of brickop and commitop as it works on op-sm framework. So the brickop failure can cause a commitop failure.

On debugging the set up i saw a glusterd_op_ac_brick_op_failed which shows that the brickop didnt complete successfully. 

So from the glusterd_brick_op_cbk it was found that the rsp was not set properly.

So the brick process was debugged and from there its found that the lock was failing because it was held already.

[2018-05-03 12:43:18.903779] W [MSGID: 101014] [client_t.c:857:gf_client_dump_inodes_to_dict] 0-client_t: Unable to acquire lock

The issue happens with distributed-replicated volumes, and not with replica volumes.The glusterd_brick_op sends request one after the other for each brick. This further confirms that the earlier request was being processed and before it was done processing the second request for the second brick tries to get the lock and fails as its held by the previous brick request. 

As the failure is based on the duration of the lock being held, the spurious reproduction of the commit failure is explained.

Comment 11 hari gowtham 2018-05-18 09:58:09 UTC
the patch on master is https://review.gluster.org/#/c/20035/

Comment 14 Bala Konda Reddy M 2018-07-17 14:02:31 UTC
Build: 3.12.2-13

Followed the steps in the description. 
On a three node cluster
created two replica 3 volumes and performed gluster volume status <volum> inode and fd continuously while io is in progress
Created a new volume(distributed replica) and started it, On the new volume all the bricks online.

Marking it to verified

Comment 16 errata-xmlrpc 2018-09-04 06:45:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607


Note You need to log in before you can comment on or make changes to this bug.