Description of problem: Gluster volume status inode command failing on subsequent volumes. Version-Release number of selected component (if applicable): 3.8.4-54-2 How reproducible: Every time Steps to Reproduce: 1.Create 3 node cluster(n1, n2, n3). 2.Create 2 replica-3 volumes (v1, v2). 3.Mount 2 volumes on two different clients(c1, c2). 4.Start running I/O parallel on two mount points. 5.While running I/O's , start executing 'Gluster volume status v1 inode' and 'Gluster volume status v1 fd' frequently with some time gap 6.In sameway run volume status inode command for v2 also 7.Then create new volume v3 (distirubted_replicated) 8. Then perform "gluster volume status v3 inode" and "gluster volume status v3 fd" on node n1 9. 'Gluster volume status inode' and 'gluster volume status fd' commands are failing for newly created volume. 10. node n1 bricks of volume v3 went to offline Actual results: root@dhcp37-113 home]# gluster vol status rp1 fd Error : Request timed out [root@dhcp37-113 home]# gluster vol status drp1 inode Error : Request timed out gluster vol status drp1 Status of volume: drp1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.37.113:/bricks/brick1/drp1 N/A N/A N N/A Brick 10.70.37.157:/bricks/brick1/drp1 49152 0 Y 2125 Brick 10.70.37.174:/bricks/brick1/drp1 49152 0 Y 2306 Brick 10.70.37.113:/bricks/brick2/drp1 N/A N/A N N/A Brick 10.70.37.157:/bricks/brick2/drp1 49152 0 Y 2125 Brick 10.70.37.174:/bricks/brick2/drp1 49152 0 Y 2306 Self-heal Daemon on localhost N/A N/A Y 4507 Self-heal Daemon on 10.70.37.157 N/A N/A Y 4006 Self-heal Daemon on 10.70.37.174 N/A N/A Y 4111 Task Status of Volume drp1 Expected results: Bricks should not go to offline and gluster volume status inode and fd commands should get executed successfully Additional info:
Changed to correct version-release number Version-Release number of selected component (if applicable): 3.8.4-54-3
I have followed the steps from description, volume status inode command failing on 3rd volume. changing state to assigned. Below error message i am getting for 3rd volume # gluster vol status dstrbrep1 inode Commit failed on dhcp37-174. Please check log file for details. Build version: glusterfs-3.12.2-8 logs pasted below: ----------------- [2018-05-03 12:44:36.320186] I [MSGID: 106499] [glusterd-handler.c:4383:__glusterd_handle_status_volume] 0-management: Received status volume req for volume dstrbrep1 [2018-05-03 12:44:36.336651] E [MSGID: 106062] [glusterd-utils.c:10024:glusterd_volume_status_copy_to_op_ctx_dict] 0-management: Failed to get other count from rsp_dict [2018-05-03 12:44:36.336723] E [MSGID: 106108] [glusterd-syncop.c:1146:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick [2018-05-03 12:44:36.336750] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Commit failed on dhcp37-174. Please check log file for details
From the machine I can see that the commit fails only on the remote node and not on the originator. other-count will be set in the commit as commit failed other count wasn't set and it caused the aggregation failure. The commit phase on the remote node is a combination of brickop and commitop as it works on op-sm framework. So the brickop failure can cause a commitop failure. On debugging the set up i saw a glusterd_op_ac_brick_op_failed which shows that the brickop didnt complete successfully. So from the glusterd_brick_op_cbk it was found that the rsp was not set properly. So the brick process was debugged and from there its found that the lock was failing because it was held already. [2018-05-03 12:43:18.903779] W [MSGID: 101014] [client_t.c:857:gf_client_dump_inodes_to_dict] 0-client_t: Unable to acquire lock The issue happens with distributed-replicated volumes, and not with replica volumes.The glusterd_brick_op sends request one after the other for each brick. This further confirms that the earlier request was being processed and before it was done processing the second request for the second brick tries to get the lock and fails as its held by the previous brick request. As the failure is based on the duration of the lock being held, the spurious reproduction of the commit failure is explained.
the patch on master is https://review.gluster.org/#/c/20035/
Build: 3.12.2-13 Followed the steps in the description. On a three node cluster created two replica 3 volumes and performed gluster volume status <volum> inode and fd continuously while io is in progress Created a new volume(distributed replica) and started it, On the new volume all the bricks online. Marking it to verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607