Bug 1258931 - gluster v heal volname info does not work with enabled ssl/tls
gluster v heal volname info does not work with enabled ssl/tls
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: cli (Show other bugs)
3.7.3
Unspecified Linux
medium Severity medium
: ---
: ---
Assigned To: Pranith Kumar K
: Triaged
Depends On:
Blocks: 1279628 1320388 1321514 1369170
  Show dependency treegraph
 
Reported: 2015-09-01 10:28 EDT by Igor Biryulin
Modified: 2016-08-22 11:36 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1279628 (view as bug list)
Environment:
Last Closed: 2016-08-22 11:36:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Igor Biryulin 2015-09-01 10:28:22 EDT
Description of problem:
When have enabled ssl/tls command "gluster v heal <VOLUME> info" return
"VOLNAME: Not able to fetch volfile from glusterd
Volume heal failed."


Version-Release number of selected component (if applicable):
3.7.3-ubuntu1~trusty1


How reproducible:
Enable ssl/tls.

Steps to Reproduce:
1. Setup installation from 2 replica.
2. Enable ssl/tls:
Generate a private key for each system.
openssl genrsa -out /etc/ssl/glusterfs.key 2048

Use the generated private key to create a signed certificate by running the following command:
openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=COMMONNAME" -out /etc/ssl/glusterfs.pem

Concatinate all glusterfs.pem to /etc/ssl/glusterfs.ca and copy on all node.

Umount mount-point

Enable encrypting managment traffic:
touch /var/lib/glusterd/secure-access

Stop VOLUME
gluster volume stop VOLNAME

Setup list of allow servers and clients
gluster volume set VOLNAME auth.ssl-allow 'server1,server2,server3,client1,client2,client3'

Enable variables:
gluster volume set VOLNAME client.ssl on
gluster volume set VOLNAME server.ssl on

Stop all glusterfs services:
/etc/init.d/glusterfs-server stop
pkill glusterd
pkill glusterfs 
pkill glusterfsd

Start glusterfs service
/etc/init.d/glusterfs-server start

Start VOLUME:
gluster volume start VOLNAME

If we want mount our share.

3. Try get "gluster v heal <VOLUME> info"

Actual results:
VOLNAME: Not able to fetch volfile from glusterd
Volume heal failed.

Expected results:
List of files need healing


Additional info:
OS: Ubuntu 14.04.3 LTS

In glfsheal-VOLNAME.log:
[2015-09-01 14:02:03.666757] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-09-01 14:02:03.670743] W [socket.c:642:__socket_rwv] 0-gfapi: readv on 127.0.0.1:24007 failed (No data available)
[2015-09-01 14:02:03.671053] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7feb297ebf46] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7feb27fad54e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7feb27fad65e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x9c)[0x7feb27faef1c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7feb27faf6b8] ))))) 0-gfapi: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2015-09-01 14:02:03.670590 (xid=0x1)
[2015-09-01 14:02:03.671076] E [MSGID: 104007] [glfs-mgmt.c:637:glfs_mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch volume file (key:repofiles) [Invalid argument]
[2015-09-01 14:02:03.671102] E [MSGID: 104024] [glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with remote-host: localhost (No data available) [No data available]
[2015-09-01 14:02:03.671115] I [MSGID: 104025] [glfs-mgmt.c:744:mgmt_rpc_notify] 0-glfs-mgmt: Exhausted all volfile servers [Transport endpoint is not connected]

In cli.log:
[2015-09-01 14:02:03.596437] I [cli.c:720:main] 0-cli: Started running gluster with version 3.7.3
[2015-09-01 14:02:03.599274] I [socket.c:3971:socket_init] 0-glusterfs: SSL support for glusterd is ENABLED
[2015-09-01 14:02:03.600027] I [socket.c:3971:socket_init] 0-glusterfs: SSL support for glusterd is ENABLED
[2015-09-01 14:02:03.659342] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-09-01 14:02:03.659426] I [socket.c:2409:socket_event_handler] 0-transport: disconnecting now
[2015-09-01 14:02:03.672269] I [input.c:36:cli_batch] 0-: Exiting with: 255

In etc-glusterfs-glusterd.vol.log every 3 sec next errors:
[2015-09-01 14:22:01.690274] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:04.690595] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:07.690876] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:10.691273] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:13.691728] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:16.692098] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:19.692517] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:22.692831] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-09-01 14:22:25.693201] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
Comment 1 JWeir 2015-10-07 12:09:03 EDT
Experiencing the exact same issue:  

Gluster 3.7.4
Ubuntu 14.04.3

The bug does not occur when the secure-access file is absent.
Comment 2 Pranith Kumar K 2015-12-16 23:21:11 EST
Need to check if http://review.gluster.org/12772 fixes this bug as well.

Pranith
Comment 5 Pranith Kumar K 2016-08-22 11:36:30 EDT
This bug is fixed as part of 1321514. Closing it. Fix is available in 3.7.10

Note You need to log in before you can comment on or make changes to this bug.