Bug 1433815

+++ This bug was initially created as a clone of Bug #1429117 +++

Description of problem:
We enabled the IP based auth feature with
gluster volume set store_temp auth.allow xxx.xxx.xxx...
This worked fine up to GlusterFS 3.9. After upgrading to 3.10, we noticed that we cannot mount any volume from a remove client anymore.
Looking at the brick logs we found:

[2017-03-04 15:56:17.469490] I [MSGID: 115091] [server-handshake.c:659:server_setvolume] 0-store_temp-server: Failed to get client opversion
[2017-03-04 15:56:17.469520] E [MSGID: 115004] [authenticate.c:224:gf_authenticate] 0-auth: no authentication module is interested in accepting remote-client (null)
[2017-03-04 15:56:17.469602] E [MSGID: 115001] [server-handshake.c:718:server_setvolume] 0-store_temp-server: Cannot authenticate client from backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0 3.9.1 [Permission denied]
[2017-03-04 15:56:28.472405] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-store_temp-server: disconnecting connection from backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0
[2017-03-04 15:56:28.472518] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-store_temp-server: Shutting down connection backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0

This problem exists even when creating completely new volumes. We already restarted and even rebooted all GlusterFS peers and the clients as well. All peers and all clients have been upgraded to 3.10


Version-Release number of selected component (if applicable):
3.10

How reproducible:
-Create a new volume 
-enable auth.allow based on IPs

Steps to Reproduce:
1. gluster volume create store_temp disperse 3 redundancy 1 ...
2. gluster volume set store_temp auth.allow xxx.xxx.xxx.xxx
3. gluster volume start store_temp
4. gluster mount ... (on a client)

Actual results:
-error message at clients "failed to set the volume [Permission denied]"
-error message at server: "no authentication module is interested in accepting remote-client (null)"

Expected results:
successful mount

Additional info:
Ubuntu 16.04

--- Additional comment from Jiffin on 2017-03-07 07:29:04 EST ---

Can you provide entire logs including bricks,glusterd and glusterfs client.
Also it will be easier if can take the tcdump from server and client

--- Additional comment from Jonathan Michalon on 2017-03-07 09:11:19 EST ---

I am stumbling on the same problem.
Setting log level to DEBUG (gluster volume set volname diagnostics.brick-log-level DEBUG) I got this first interesting stuff:
  allowed = "192.168.122.186", received addr = "R"
Then some time afterwards:
  allowed = "192.168.122.186", received addr = "m"

So it was looking like we were reading some random memory. And indeed looking into source code, between 3.9 and 3.10 the big switch/case filling peer_addr disappeared in /xlators/protocol/auth/addr/src/addr.c 
I think this is enough to tell that there is some problem here :)

--- Additional comment from Atin Mukherjee on 2017-03-13 02:00:47 EDT ---

auth failures need not be in glusterd, moving this to core component.

--- Additional comment from Yong on 2017-03-19 03:35:18 EDT ---

I have the same issue, I think this is critical, please help