Description of problem: When have enabled ssl/tls command "gluster v heal <VOLUME> info" return "VOLNAME: Not able to fetch volfile from glusterd Volume heal failed." Version-Release number of selected component (if applicable): 3.7.3-ubuntu1~trusty1 How reproducible: Enable ssl/tls. Steps to Reproduce: 1. Setup installation from 2 replica. 2. Enable ssl/tls: Generate a private key for each system. openssl genrsa -out /etc/ssl/glusterfs.key 2048 Use the generated private key to create a signed certificate by running the following command: openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=COMMONNAME" -out /etc/ssl/glusterfs.pem Concatinate all glusterfs.pem to /etc/ssl/glusterfs.ca and copy on all node. Umount mount-point Enable encrypting managment traffic: touch /var/lib/glusterd/secure-access Stop VOLUME gluster volume stop VOLNAME Setup list of allow servers and clients gluster volume set VOLNAME auth.ssl-allow 'server1,server2,server3,client1,client2,client3' Enable variables: gluster volume set VOLNAME client.ssl on gluster volume set VOLNAME server.ssl on Stop all glusterfs services: /etc/init.d/glusterfs-server stop pkill glusterd pkill glusterfs pkill glusterfsd Start glusterfs service /etc/init.d/glusterfs-server start Start VOLUME: gluster volume start VOLNAME If we want mount our share. 3. Try get "gluster v heal <VOLUME> info" Actual results: VOLNAME: Not able to fetch volfile from glusterd Volume heal failed. Expected results: List of files need healing Additional info: OS: Ubuntu 14.04.3 LTS In glfsheal-VOLNAME.log: [2015-09-01 14:02:03.666757] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-09-01 14:02:03.670743] W [socket.c:642:__socket_rwv] 0-gfapi: readv on 127.0.0.1:24007 failed (No data available) [2015-09-01 14:02:03.671053] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7feb297ebf46] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7feb27fad54e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7feb27fad65e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7feb27faef1c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7feb27faf6b8] ))))) 0-gfapi: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2015-09-01 14:02:03.670590 (xid=0x1) [2015-09-01 14:02:03.671076] E [MSGID: 104007] [glfs-mgmt.c:637:glfs_mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch volume file (key:repofiles) [Invalid argument] [2015-09-01 14:02:03.671102] E [MSGID: 104024] [glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with remote-host: localhost (No data available) [No data available] [2015-09-01 14:02:03.671115] I [MSGID: 104025] [glfs-mgmt.c:744:mgmt_rpc_notify] 0-glfs-mgmt: Exhausted all volfile servers [Transport endpoint is not connected] In cli.log: [2015-09-01 14:02:03.596437] I [cli.c:720:main] 0-cli: Started running gluster with version 3.7.3 [2015-09-01 14:02:03.599274] I [socket.c:3971:socket_init] 0-glusterfs: SSL support for glusterd is ENABLED [2015-09-01 14:02:03.600027] I [socket.c:3971:socket_init] 0-glusterfs: SSL support for glusterd is ENABLED [2015-09-01 14:02:03.659342] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-09-01 14:02:03.659426] I [socket.c:2409:socket_event_handler] 0-transport: disconnecting now [2015-09-01 14:02:03.672269] I [input.c:36:cli_batch] 0-: Exiting with: 255 In etc-glusterfs-glusterd.vol.log every 3 sec next errors: [2015-09-01 14:22:01.690274] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-09-01 14:22:04.690595] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-09-01 14:22:07.690876] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-09-01 14:22:10.691273] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-09-01 14:22:13.691728] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-09-01 14:22:16.692098] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-09-01 14:22:19.692517] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-09-01 14:22:22.692831] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument) [2015-09-01 14:22:25.693201] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
Experiencing the exact same issue: Gluster 3.7.4 Ubuntu 14.04.3 The bug does not occur when the secure-access file is absent.
Need to check if http://review.gluster.org/12772 fixes this bug as well. Pranith
This bug is fixed as part of 1321514. Closing it. Fix is available in 3.7.10