Bug 1247274 - Unable to change the auth.allow setting on volumes due to spurious client version issue
Unable to change the auth.allow setting on volumes due to spurious client ver...
Status: CLOSED NOTABUG
Product: GlusterFS
Classification: Community
Component: cli (Show other bugs)
3.6.4
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: bugs@gluster.org
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-07-27 13:03 EDT by Alastair Neil
Modified: 2015-08-05 15:35 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-08-05 15:35:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alastair Neil 2015-07-27 13:03:21 EDT
Description of problem:
I have a replica 3 volume homes with numerous clients, I need to change the auth.allow to enable mounting by clients on a new subnet.  any attempt to change the setting using:

gluster volume set homes auth.allow 129.174.126.*,129.174.120.*,129.174.121.*,129.174.122.*,129.174.123.*

generates:

volume set: failed: Staging failed on 129.174.126.56. Error: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again

To my knowledge this should never be possible as auth.allow has existed in every version of Gluster.

Version-Release number of selected component (if applicable):
3.6.4

How reproducible:
always

Steps to Reproduce:
1.run gluster volume set volname aut.allow <list>
2.
3.

Actual results:
auth.allow parameters unchanged


Expected results:
auth.allow parameters set to new value


Additional info:

This cluster was originally installed with 3.3, and hase been progressively upgraded.
Comment 1 Atin Mukherjee 2015-07-28 00:00:42 EDT
Please attach glusterd logs, with out the log files its difficult to analyze the problem.
Comment 2 Alastair Neil 2015-07-28 09:35:46 EDT
let me know what if any other logs you need.

Node  Gluster0: etc-glusterfs-glusterd.vol.log:

[2015-07-28 13:23:55.157052] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:23:55.157832] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:23:55.159092] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.453804] E [glusterd-syncop.c:105:gd_collate_errors] 0-: Staging failed on 129.174.126.56. Error: One or more connected clients cannot support the feature being set. These clients need to be upgraded or disconnected before running this command again
[2015-07-28 13:24:00.681844] I [glusterd-handler.c:1241:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-07-28 13:24:00.740495] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.741785] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.743125] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.744396] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.745617] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.746575] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.747446] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-07-28 13:24:00.748670] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req

Node Gluster1: etc-glusterfs-glusterd.vol.log:

[2015-07-28 13:25:10.295707] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: tune2fs exited with non-zero exit status
[2015-07-28 13:25:10.295758] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:10.719401] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:10.719445] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:11.571048] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:11.571097] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:11.957819] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:11.957861] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:12.345546] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:12.345587] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:12.835625] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:12.835689] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:13.344794] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:13.344836] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:36.304764] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : bbb60160-66e9-44b9-a778-aa3d45755075
[2015-07-28 13:25:47.242369] E [glusterd-op-sm.c:671:glusterd_check_client_op_version_support] 0-management: One or more clients don't support the required op-version
[2015-07-28 13:25:47.242421] E [glusterd-op-sm.c:4585:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Set', Status : -1


Node Gluster-2: etc-glusterfs-glusterd.vol.log:

[2015-07-28 13:25:10.295758] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:10.719401] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:10.719445] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:11.571048] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:11.571097] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:11.957819] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:11.957861] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:12.345546] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:12.345587] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:12.835625] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:12.835689] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:13.344794] E [glusterd-utils.c:7410:glusterd_add_inode_size_to_dict] 0-management: xfs_info exited with non-zero exit status
[2015-07-28 13:25:13.344836] E [glusterd-utils.c:7436:glusterd_add_inode_size_to_dict] 0-management: failed to get inode size
[2015-07-28 13:25:36.304764] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : bbb60160-66e9-44b9-a778-aa3d45755075
[2015-07-28 13:25:47.242369] E [glusterd-op-sm.c:671:glusterd_check_client_op_version_support] 0-management: One or more clients don't support the required op-version
[2015-07-28 13:25:47.242421] E [glusterd-op-sm.c:4585:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Set', Status : -1
[2015-07-28 13:26:33.168980] E [glusterd-op-sm.c:671:glusterd_check_client_op_version_support] 0-management: One or more clients don't support the required op-version
[2015-07-28 13:26:33.169032] E [glusterd-op-sm.c:4585:glusterd_op_ac_stage_op] 0-management: Stage failed on operation 'Volume Set', Status : -1
[2015-07-28 13:26:36.502079] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : 7c4b3693-77f0-4087-a236-76f4f41edb2f
[2015-07-28 13:27:36.685167] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : 2588f9c9-2e33-47fc-9566-c95ab381dfd7
[2015-07-28 13:28:36.834453] E [glusterd-op-sm.c:207:glusterd_get_txn_opinfo] 0-: Unable to get transaction opinfo for transaction ID : d90dc97f-f6bc-40d7-bef9-85f5f73fcb14
Comment 3 Niels de Vos 2015-08-04 08:35:12 EDT
From the log: One or more clients don't support the required op-version

This suggests that there is a client that does not support the options for the volume that needs this auth.allow change.

Could you check if all the clients that connect to the volume have the same version?
Comment 4 Alastair Neil 2015-08-04 15:17:32 EDT
Not an easy thing to do if you have hundreds of clients. I have been able to eventually track down the client that was causing the issue.  It was an ubuntu 14.04 client, I'm not sure what gluster version it was using as I don't have access to the system. 

This would have been much easier to track down if the ip addresses of the problem clients were logged with the error.

"One or more clients don't support the required op-version"

useless!

Alternatively, have "gluster volume status volname clients" report some information about the clients other than just ip address, client version springs to mind.
Comment 5 Atin Mukherjee 2015-08-05 06:35:16 EDT
(In reply to Alastair Neil from comment #4)
> Not an easy thing to do if you have hundreds of clients. I have been able to
> eventually track down the client that was causing the issue.  It was an
> ubuntu 14.04 client, I'm not sure what gluster version it was using as I
> don't have access to the system. 
> 
> This would have been much easier to track down if the ip addresses of the
> problem clients were logged with the error.
> 
> "One or more clients don't support the required op-version"
I agree that there is a scope of improvement in logging. I've sent a patch http://review.gluster.org/11831 to address it. Hopefully I will try to get it pushed for next 3.7 release.
> 
> useless!
> 
> Alternatively, have "gluster volume status volname clients" report some
> information about the clients other than just ip address, client version
> springs to mind.

Note You need to log in before you can comment on or make changes to this bug.