Bug 1247274

Summary: Unable to change the auth.allow setting on volumes due to spurious client version issue
Product: [Community] GlusterFS Reporter: Alastair Neil <aneil2>
Component: cliAssignee: bugs <bugs>
Status: CLOSED NOTABUG QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.4CC: amukherj, aneil2, bugs, gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-05 19:35:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alastair Neil 2015-07-27 17:03:21 UTC 
Description of problem:
I have a replica 3 volume homes with numerous clients, I need to change the auth.allow to enable mounting by clients on a new subnet.  any attempt to change the setting using:

gluster volume set homes auth.allow 129.174.126.*,129.174.120.*,129.174.121.*,129.174.122.*,129.174.123.*

generates:

volume set: failed: Staging failed on 129.174.126.56. Error: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again

To my knowledge this should never be possible as auth.allow has existed in every version of Gluster.

Version-Release number of selected component (if applicable):
3.6.4

How reproducible:
always

Steps to Reproduce:
1.run gluster volume set volname aut.allow <list>
2.
3.

Actual results:
auth.allow parameters unchanged


Expected results:
auth.allow parameters set to new value


Additional info:

This cluster was originally installed with 3.3, and hase been progressively upgraded.
Comment 1 Atin Mukherjee 2015-07-28 04:00:42 UTC 
Please attach glusterd logs, with out the log files its difficult to analyze the problem.
Comment 2 Alastair Neil 2015-07-28 13:35:46 UTC 
let me know what if any other logs you need.

Node  Gluster0: etc-glusterfs-glusterd.vol.log:

[2015-07-28 13:23:55.157052] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:23:55.157832] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:23:55.159092] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.453804] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Staging failed on 129.174.126.56. Error: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again
[2015-07-28 13:24:00.681844] I [glusterd-handler.c:1241:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-07-28 13:24:00.740495] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.741785] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.743125] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.744396] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.745617] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.746575] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.747446] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.748670] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

Node Gluster1: etc-glusterfs-glusterd.vol.log:

[2015-07-28 13:25:10.295707] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: tune2fs exited with non-zero exit status
[2015-07-28 13:25:10.295758] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:10.719401] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:10.719445] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:11.571048] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:11.571097] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:11.957819] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:11.957861] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:12.345546] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:12.345587] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:12.835625] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:12.835689] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:13.344794] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:13.344836] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:36.304764] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : bbb60160-66e9-44b9-a778-aa3d45755075
[2015-07-28 13:25:47.242369] E [glusterd-op-sm.c:671:glusterd_check_client_op_version_support] 0-management: One or more clients don't support the required op-version
[2015-07-28 13:25:47.242421] E [glusterd-op-sm.c:4585:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Set', Status : -1


Node Gluster-2: etc-glusterfs-glusterd.vol.log:

[2015-07-28 13:25:10.295758] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:10.719401] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:10.719445] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:11.571048] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:11.571097] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:11.957819] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:11.957861] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:12.345546] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:12.345587] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:12.835625] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:12.835689] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:13.344794] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:13.344836] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:36.304764] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : bbb60160-66e9-44b9-a778-aa3d45755075
[2015-07-28 13:25:47.242369] E [glusterd-op-sm.c:671:glusterd_check_client_op_version_support] 0-management: One or more clients don't support the required op-version
[2015-07-28 13:25:47.242421] E [glusterd-op-sm.c:4585:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Set', Status : -1
[2015-07-28 13:26:33.168980] E [glusterd-op-sm.c:671:glusterd_check_client_op_version_support] 0-management: One or more clients don't support the required op-version
[2015-07-28 13:26:33.169032] E [glusterd-op-sm.c:4585:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Set', Status : -1
[2015-07-28 13:26:36.502079] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : 7c4b3693-77f0-4087-a236-76f4f41edb2f
[2015-07-28 13:27:36.685167] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : 2588f9c9-2e33-47fc-9566-c95ab381dfd7
[2015-07-28 13:28:36.834453] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : d90dc97f-f6bc-40d7-bef9-85f5f73fcb14
Comment 3 Niels de Vos 2015-08-04 12:35:12 UTC 
From the log: One or more clients don't support the required op-version

This suggests that there is a client that does not support the options for the volume that needs this auth.allow change.

Could you check if all the clients that connect to the volume have the same version?
Comment 4 Alastair Neil 2015-08-04 19:17:32 UTC 
Not an easy thing to do if you have hundreds of clients. I have been able to eventually track down the client that was causing the issue.  It was an ubuntu 14.04 client, I'm not sure what gluster version it was using as I don't have access to the system. 

This would have been much easier to track down if the ip addresses of the problem clients were logged with the error.

"One or more clients don't support the required op-version"

useless!

Alternatively, have "gluster volume status volname clients" report some information about the clients other than just ip address, client version springs to mind.
Comment 5 Atin Mukherjee 2015-08-05 10:35:16 UTC 
(In reply to Alastair Neil from comment #4)
> Not an easy thing to do if you have hundreds of clients. I have been able to
> eventually track down the client that was causing the issue.  It was an
> ubuntu 14.04 client, I'm not sure what gluster version it was using as I
> don't have access to the system. 
> 
> This would have been much easier to track down if the ip addresses of the
> problem clients were logged with the error.
> 
> "One or more clients don't support the required op-version"
I agree that there is a scope of improvement in logging. I've sent a patch http://review.gluster.org/11831 to address it. Hopefully I will try to get it pushed for next 3.7 release.
> 
> useless!
> 
> Alternatively, have "gluster volume status volname clients" report some
> information about the clients other than just ip address, client version
> springs to mind.