Bug 1429117 - auth failure after upgrade to GlusterFS 3.10
Summary: auth failure after upgrade to GlusterFS 3.10
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: core
Version: 3.10
Hardware: Unspecified
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: glusterfs-3.10.1 1433815 1437332
TreeView+ depends on / blocked
 
Reported: 2017-03-04 16:10 UTC by Flexadot
Modified: 2017-04-05 00:01 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.10.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1433815 (view as bug list)
Environment:
Last Closed: 2017-04-05 00:01:42 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Flexadot 2017-03-04 16:10:22 UTC
Description of problem:
We enabled the IP based auth feature with
gluster volume set store_temp auth.allow xxx.xxx.xxx...
This worked fine up to GlusterFS 3.9. After upgrading to 3.10, we noticed that we cannot mount any volume from a remove client anymore.
Looking at the brick logs we found:

[2017-03-04 15:56:17.469490] I [MSGID: 115091] [server-handshake.c:659:server_setvolume] 0-store_temp-server: Failed to get client opversion
[2017-03-04 15:56:17.469520] E [MSGID: 115004] [authenticate.c:224:gf_authenticate] 0-auth: no authentication module is interested in accepting remote-client (null)
[2017-03-04 15:56:17.469602] E [MSGID: 115001] [server-handshake.c:718:server_setvolume] 0-store_temp-server: Cannot authenticate client from backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0 3.9.1 [Permission denied]
[2017-03-04 15:56:28.472405] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-store_temp-server: disconnecting connection from backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0
[2017-03-04 15:56:28.472518] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-store_temp-server: Shutting down connection backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0

This problem exists even when creating completely new volumes. We already restarted and even rebooted all GlusterFS peers and the clients as well. All peers and all clients have been upgraded to 3.10


Version-Release number of selected component (if applicable):
3.10

How reproducible:
-Create a new volume 
-enable auth.allow based on IPs

Steps to Reproduce:
1. gluster volume create store_temp disperse 3 redundancy 1 ...
2. gluster volume set store_temp auth.allow xxx.xxx.xxx.xxx
3. gluster volume start store_temp
4. gluster mount ... (on a client)

Actual results:
-error message at clients "failed to set the volume [Permission denied]"
-error message at server: "no authentication module is interested in accepting remote-client (null)"

Expected results:
successful mount

Additional info:
Ubuntu 16.04

Comment 1 Jiffin 2017-03-07 12:29:04 UTC
Can you provide entire logs including bricks,glusterd and glusterfs client.
Also it will be easier if can take the tcdump from server and client

Comment 2 Jonathan Michalon 2017-03-07 14:11:19 UTC
I am stumbling on the same problem.
Setting log level to DEBUG (gluster volume set volname diagnostics.brick-log-level DEBUG) I got this first interesting stuff:
  allowed = "192.168.122.186", received addr = "R"
Then some time afterwards:
  allowed = "192.168.122.186", received addr = "m"

So it was looking like we were reading some random memory. And indeed looking into source code, between 3.9 and 3.10 the big switch/case filling peer_addr disappeared in /xlators/protocol/auth/addr/src/addr.c 
I think this is enough to tell that there is some problem here :)

Comment 3 Atin Mukherjee 2017-03-13 06:00:47 UTC
auth failures need not be in glusterd, moving this to core component.

Comment 4 Yong 2017-03-19 07:35:18 UTC
I have the same issue, I think this is critical, please help

Comment 5 haloween 2017-03-27 15:04:28 UTC
On a fresh install of 3.10 i have the same symptoms. 
I'm running a single node install and it's impossible to mount a share over local network.

Comment 6 Worker Ant 2017-03-30 05:59:10 UTC
REVIEW: https://review.gluster.org/16967 (protocol : fix auth-allow regression) posted (#1) for review on release-3.10 by Atin Mukherjee (amukherj)

Comment 7 Worker Ant 2017-03-30 14:06:34 UTC
COMMIT: https://review.gluster.org/16967 committed in release-3.10 by Shyamsundar Ranganathan (srangana) 
------
commit bbf83e34d78e064befe816edf71a9ee5c2c5c209
Author: Atin Mukherjee <amukherj>
Date:   Mon Mar 20 05:15:25 2017 +0530

    protocol : fix auth-allow regression
    
    One of the brick multiplexing patches (commit 1a95fc3) had some changes
    in gf_auth () & server_setvolume () functions which caused auth-allow
    feature to be broken. mount doesn't succeed even if it's part of the
    auth-allow list. This fix does the following:
    
    1. Reintroduce the peer-info data back in gf_auth () so that fnmatch has
    valid input and it can decide on the result.
    
    2. config-params dict should capture key values pairs for all the bricks
    in case brick multiplexing is on. In case brick multiplexing isn't
    enabled, then config-params should carry attributes from protocol/server
    such that all rpc auth related attributes stay in tact in the
    dictionary.
    
    >Reviewed-on: https://review.gluster.org/16920
    >Tested-by: Jeff Darcy <jeff.us>
    >Smoke: Gluster Build System <jenkins.org>
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.org>
    >Reviewed-by: Jeff Darcy <jeff.us>
    >Reviewed-by: MOHIT AGRAWAL <moagrawa>
    >(cherry picked from commit 0bd58241143e91b683a3e5c4335aabf9eed537fe)
    
    Change-Id: I007c4c6d78620a896b8858a29459a77de8b52412
    BUG: 1429117
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: https://review.gluster.org/16967
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana>

Comment 8 Shyamsundar 2017-04-05 00:01:42 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.1, please open a new bug report.

glusterfs-3.10.1 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-April/030494.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.