Bug 1248415 - rebalance stuck at 0 byte when auth.allow is set
rebalance stuck at 0 byte when auth.allow is set
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: glusterd (Show other bugs)
mainline
All Linux
medium Severity high
: ---
: ---
Assigned To: Nithya Balachandran
: Reopened
Depends On: 1213893
Blocks: 1253542
  Show dependency treegraph
 
Reported: 2015-07-30 05:01 EDT by Nithya Balachandran
Modified: 2016-06-16 09:28 EDT (History)
8 users (show)

See Also:
Fixed In Version: glusterfs-3.8rc2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1213893
: 1253542 (view as bug list)
Environment:
Last Closed: 2016-06-16 09:28:07 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nithya Balachandran 2015-07-30 05:01:35 EDT
+++ This bug was initially created as a clone of Bug #1213893 +++

Description of problem:

When setting auth.allow, rebalance will get stuck unless the IPs of the gluster nodes themselves are included.
The rebalance will be kept as 'in progress', but will be kept at 0 Byte.



                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress               0.00
                             rhs30-node3                0        0Bytes             0             0             0          in progress               0.00
                             rhs30-node4                0        0Bytes             0             0             0          in progress               0.00
                         192.168.100.206                0        0Bytes             0             0             0          in progress               0.00
volume rebalance: thingluster: success:

On the bricks logs, we can see the authentication being prevented :
[2015-04-21 13:43:03.131329] E [server-handshake.c:589:server_setvolume] 0-thingluster-server: Cannot authenticate client from cbuissar-rhs30-node1-8521-2015/04/21-13:42:58:108057-thingluster-client-0-0-0 3.6.0.53
[2015-04-21 13:43:08.419405] E [authenticate.c:239:gf_authenticate] 0-auth: no authentication module is interested in accepting remote-client (null)


Version-Release number of selected component (if applicable): tested on 3.0u3 and 3.0u4


How reproducible: 100%/easy


Steps to Reproduce:
1. set auth.allow to some client IP
2. mount and move files
3. start rebalance

Actual results:
rebalance is hung, authentication errors are shown in the brick logs

Expected results:
Rebalance should still work if we restrict auth.allow.


Additional info:
Workaround : add all the IPs of the gluster nodes in auth.allow.

--- Additional comment from Cedric Buissart on 2015-04-21 09:55:45 EDT ---

And the rebalance-<volume>.log :

[2015-04-21 13:43:08.412805] W [client-handshake.c:1108:client_setvolume_cbk] 0-thingluster-client-3: failed to set the volume (Permission denied)
[2015-04-21 13:43:08.412821] W [client-handshake.c:1134:client_setvolume_cbk] 0-thingluster-client-3: failed to get 'process-uuid' from reply dict
[2015-04-21 13:43:08.412828] E [client-handshake.c:1140:client_setvolume_cbk] 0-thingluster-client-3: SETVOLUME on remote-host failed: Authentication failed
[2015-04-21 13:43:08.412834] I [client-handshake.c:1225:client_setvolume_cbk] 0-thingluster-client-3: sending AUTH_FAILED even
Comment 1 Anand Avati 2015-07-30 05:04:14 EDT
REVIEW: http://review.gluster.org/11803 (glusterd/rebalance: create rebalance volfile) posted (#1) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 2 Anand Avati 2015-08-03 04:29:38 EDT
REVIEW: http://review.gluster.org/11819 (glusterd/rebalance: trusted rebalance volfile) posted (#1) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 3 Anand Avati 2015-08-03 04:35:27 EDT
REVIEW: http://review.gluster.org/11819 (glusterd/rebalance: trusted rebalance volfile) posted (#2) for review on master by N Balachandran (nbalacha@redhat.com)
Comment 4 Anand Avati 2015-08-11 05:59:59 EDT
COMMIT: http://review.gluster.org/11819 committed in master by Atin Mukherjee (amukherj@redhat.com) 
------
commit 8603ac80218d6161416fd0f3cca511ba09aef2c8
Author: N Balachandran <nbalacha@redhat.com>
Date:   Mon Aug 3 13:57:37 2015 +0530

    glusterd/rebalance: trusted rebalance volfile
    
    Creating the client volfiles with GF_CLIENT_OTHER
    overwrites the trusted rebalance volfile and causes rebalance
    to fail if auth.allow is set.
    Now, we always set the value of trusted-client to GF_CLIENT_TRUSTED
    for rebalance volfiles.
    
    Change-Id: I95eb510256d18dfa9048f96a1aeb71cca4811811
    BUG: 1248415
    Signed-off-by: N Balachandran <nbalacha@redhat.com>
    Reviewed-on: http://review.gluster.org/11819
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Avra Sengupta <asengupt@redhat.com>
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
Comment 5 Nagaprasad Sathyanarayana 2015-10-25 11:20:10 EDT
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.
Comment 6 Niels de Vos 2016-06-16 09:28:07 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.