Bug 1258931 - gluster v heal volname info does not work with enabled ssl/tls
Summary: gluster v heal volname info does not work with enabled ssl/tls
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: cli
Version: 3.7.3
Hardware: Unspecified
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Pranith Kumar K
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1279628 1320388 1321514 1369170
TreeView+ depends on / blocked
 
Reported: 2015-09-01 14:28 UTC by Igor Biryulin
Modified: 2016-08-22 15:36 UTC (History)
6 users (show)

Fixed In Version:
Clone Of:
: 1279628 (view as bug list)
Environment:
Last Closed: 2016-08-22 15:36:30 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Igor Biryulin 2015-09-01 14:28:22 UTC
Description of problem:
When have enabled ssl/tls command "gluster v heal <VOLUME> info" return
"VOLNAME: Not able to fetch volfile from glusterd
Volume heal failed."


Version-Release number of selected component (if applicable):
3.7.3-ubuntu1~trusty1


How reproducible:
Enable ssl/tls.

Steps to Reproduce:
1. Setup installation from 2 replica.
2. Enable ssl/tls:
Generate a private key for each system.
openssl genrsa -out /etc/ssl/glusterfs.key 2048

Use the generated private key to create a signed certificate by running the following command:
openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=COMMONNAME" -out /etc/ssl/glusterfs.pem

Concatinate all glusterfs.pem to /etc/ssl/glusterfs.ca and copy on all node.

Umount mount-point

Enable encrypting managment traffic:
touch /var/lib/glusterd/secure-access

Stop VOLUME
gluster volume stop VOLNAME

Setup list of allow servers and clients
gluster volume set VOLNAME auth.ssl-allow 'server1,server2,server3,client1,client2,client3'

Enable variables:
gluster volume set VOLNAME client.ssl on
gluster volume set VOLNAME server.ssl on

Stop all glusterfs services:
/etc/init.d/glusterfs-server stop
pkill glusterd
pkill glusterfs 
pkill glusterfsd

Start glusterfs service
/etc/init.d/glusterfs-server start

Start VOLUME:
gluster volume start VOLNAME

If we want mount our share.

3. Try get "gluster v heal <VOLUME> info"

Actual results:
VOLNAME: Not able to fetch volfile from glusterd
Volume heal failed.

Expected results:
List of files need healing


Additional info:
OS: Ubuntu 14.04.3 LTS

In glfsheal-VOLNAME.log:
[2015-09-01 14:02:03.666757] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-09-01 14:02:03.670743] W [socket.c:642:__socket_rwv] 0-gfapi: readv on 127.0.0.1:24007 failed (No data available)
[2015-09-01 14:02:03.671053] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7feb297ebf46] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7feb27fad54e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7feb27fad65e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7feb27faef1c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7feb27faf6b8] ))))) 0-gfapi: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2015-09-01 14:02:03.670590 (xid=0x1)
[2015-09-01 14:02:03.671076] E [MSGID: 104007] [glfs-mgmt.c:637:glfs_mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch volume file (key:repofiles) [Invalid argument]
[2015-09-01 14:02:03.671102] E [MSGID: 104024] [glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with remote-host: localhost (No data available) [No data available]
[2015-09-01 14:02:03.671115] I [MSGID: 104025] [glfs-mgmt.c:744:mgmt_rpc_notify] 0-glfs-mgmt: Exhausted all volfile servers [Transport endpoint is not connected]

In cli.log:
[2015-09-01 14:02:03.596437] I [cli.c:720:main] 0-cli: Started running gluster with version 3.7.3
[2015-09-01 14:02:03.599274] I [socket.c:3971:socket_init] 0-glusterfs: SSL support for glusterd is ENABLED
[2015-09-01 14:02:03.600027] I [socket.c:3971:socket_init] 0-glusterfs: SSL support for glusterd is ENABLED
[2015-09-01 14:02:03.659342] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-09-01 14:02:03.659426] I [socket.c:2409:socket_event_handler] 0-transport: disconnecting now
[2015-09-01 14:02:03.672269] I [input.c:36:cli_batch] 0-: Exiting with: 255

In etc-glusterfs-glusterd.vol.log every 3 sec next errors:
[2015-09-01 14:22:01.690274] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:04.690595] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:07.690876] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:10.691273] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:13.691728] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:16.692098] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:19.692517] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:22.692831] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:25.693201] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)

Comment 1 JWeir 2015-10-07 16:09:03 UTC
Experiencing the exact same issue:  

Gluster 3.7.4
Ubuntu 14.04.3

The bug does not occur when the secure-access file is absent.

Comment 2 Pranith Kumar K 2015-12-17 04:21:11 UTC
Need to check if http://review.gluster.org/12772 fixes this bug as well.

Pranith

Comment 5 Pranith Kumar K 2016-08-22 15:36:30 UTC
This bug is fixed as part of 1321514. Closing it. Fix is available in 3.7.10


Note You need to log in before you can comment on or make changes to this bug.