Bug 1213893 – rebalance stuck at 0 byte when auth.allow is set

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1213893 - rebalance stuck at 0 byte when auth.allow is set

Summary:	rebalance stuck at 0 byte when auth.allow is set

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Gluster Storage
Classification:	Red Hat
Component:	distribute (Show other bugs)
Sub Component:
Version:	3.0
Hardware:	All Linux

Priority	medium Severity high
Target Milestone:	---
Target Release:	RHGS 3.1.1
Assigned To:	Nithya Balachandran
QA Contact:	RajeshReddy
Docs Contact:

URL:
Whiteboard:
Keywords:	ZStream

Depends On:
Blocks:	1248415 1251815 1253542
	Show dependency tree / graph

Reported:	2015-04-21 09:52 EDT by Cedric Buissart
Modified:	2016-09-17 10:54 EDT (History)
CC List:	10 users (show)

See Also:
Fixed In Version:	glusterfs-3.7.1-12
Doc Type:	Bug Fix
Doc Text:	Previously, the brick processes did not consider rebalance processes to be trusted clients. As a consequence, if the auth.allow option was set for a volume, connections from the rebalance processes for that volume were rejected by the brick processes, causing rebalance to hang. With this fix, the rebalance process is treated as a trusted client by the brick processes. Now, the rebalance works even if the auth.allow option is set for a volume.
Story Points:	---
Clone Of:
Clones:	1248415 (view as bug list)
Environment:
Last Closed:	2015-10-05 03:08:14 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	1422103	None	None	None	Never
Red Hat Product Errata	RHSA-2015:1845	normal	SHIPPED_LIVE	Moderate: Red Hat Gluster Storage 3.1 update	2015-10-05 07:06:22 EDT

Groups: None (edit)

Description Cedric Buissart 2015-04-21 09:52:26 EDT

Description of problem:

When setting auth.allow, rebalance will get stuck unless the IPs of the gluster nodes themselves are included.
The rebalance will be kept as 'in progress', but will be kept at 0 Byte.



                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress               0.00
                             rhs30-node3                0        0Bytes             0             0             0          in progress               0.00
                             rhs30-node4                0        0Bytes             0             0             0          in progress               0.00
                         192.168.100.206                0        0Bytes             0             0             0          in progress               0.00
volume rebalance: thingluster: success:

On the bricks logs, we can see the authentication being prevented :
[2015-04-21 13:43:03.131329] E [server-handshake.c:589:server_setvolume] 0-thingluster-server: Cannot authenticate client from cbuissar-rhs30-node1-8521-2015/04/21-13:42:58:108057-thingluster-client-0-0-0 3.6.0.53
[2015-04-21 13:43:08.419405] E [authenticate.c:239:gf_authenticate] 0-auth: no authentication module is interested in accepting remote-client (null)


Version-Release number of selected component (if applicable): tested on 3.0u3 and 3.0u4


How reproducible: 100%/easy


Steps to Reproduce:
1. set auth.allow to some client IP
2. mount and move files
3. start rebalance

Actual results:
rebalance is hung, authentication errors are shown in the brick logs

Expected results:
Rebalance should still work if we restrict auth.allow.


Additional info:
Workaround : add all the IPs of the gluster nodes in auth.allow.

Comment 1 Cedric Buissart 2015-04-21 09:55:45 EDT

And the rebalance-<volume>.log :

[2015-04-21 13:43:08.412805] W [client-handshake.c:1108:client_setvolume_cbk] 0-thingluster-client-3: failed to set the volume (Permission denied)
[2015-04-21 13:43:08.412821] W [client-handshake.c:1134:client_setvolume_cbk] 0-thingluster-client-3: failed to get 'process-uuid' from reply dict
[2015-04-21 13:43:08.412828] E [client-handshake.c:1140:client_setvolume_cbk] 0-thingluster-client-3: SETVOLUME on remote-host failed: Authentication failed
[2015-04-21 13:43:08.412834] I [client-handshake.c:1225:client_setvolume_cbk] 0-thingluster-client-3: sending AUTH_FAILED even

Comment 6 RajeshReddy 2015-09-04 04:35:45 EDT

Tested with build "glusterfs-3.7.1-12" and after setting auth.allow and nfs.rpc-auth-allow able to run re balance job without any problem so marking this bug as verified

Comment 8 errata-xmlrpc 2015-10-05 03:08:14 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html

Note You need to log in before you can comment on or make changes to this bug.