Bug 1243722

Summary: glusterd crashed when a client which doesn't support SSL tries to mount a SSL enabled gluster volume
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: M S Vishwanath Bhat <vbhat>
Component: glusterdAssignee: Kaushal <kaushal>
Status: CLOSED ERRATA QA Contact: M S Vishwanath Bhat <vbhat>
Severity: high Docs Contact:
Priority: urgent    
Version: rhgs-3.1CC: amukherj, asriram, divya, kaushal, mlawrenc, mzywusko, nlevinki, rcyriac, rgowdapp, sasundar, vagarwal, vbellur
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.1   
Hardware: x86_64   
OS: Linux   
Whiteboard: GlusterD
Fixed In Version: glusterfs-3.7.1-12 Doc Type: Bug Fix
Doc Text:
Previously, the glusterd was not fully initializing its transports when using management encryption. As a consequence, an unencrypted incoming connection would cause glusterd to crash.As a fix, the transports are now fully initialized and additional checks have been added to handle unencrypted incoming connections.Now, glusterd no longer crashes on incoming unencrypted connections when using management encryption.
Story Points: ---
Clone Of:
: 1243774 (view as bug list) Environment:
Last Closed: 2015-10-05 07:20:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1243774    
Bug Blocks: 1216951, 1244415, 1251815, 1314702    

Description M S Vishwanath Bhat 2015-07-16 07:21:05 UTC
Description of problem:
I was running rebalance automation and I had a gluster volume with SSL network encryption enabled in both data and management path. But when a client with a gluster version which does not support SSL tries to mount the this gluster volume, the glusterd in the volfile-server crashed.

Version-Release number of selected component (if applicable):
gluster server: glusterfs-3.7.1-10.el6rhs.x86_64
gluster client: glusterfs-3.6.0.53-1.el6.x86_64

How reproducible:
Hit only once in as many tries

Steps to Reproduce:
1. Run rebalance regression tests with SSL enabled with old client version

Actual results:
glusterd crashed with below backtrace

#0  list_del (rpc=<value optimized out>, xl=0x7fb94bfe1050, event=<value optimized out>, data=0x7fb92c000be0) at ../../../../libglusterfs/src/list.h:76
#1  glusterd_rpcsvc_notify (rpc=<value optimized out>, xl=0x7fb94bfe1050, event=<value optimized out>, data=0x7fb92c000be0) at glusterd.c:347
#2  0x00007fb94a9df665 in rpcsvc_handle_disconnect (svc=0x7fb94bfea380, trans=0x7fb92c000be0) at rpcsvc.c:754
#3  0x00007fb94a9e11c0 in rpcsvc_notify (trans=0x7fb92c000be0, mydata=<value optimized out>, event=<value optimized out>, data=0x7fb92c000be0) at rpcsvc.c:792
#4  0x00007fb94a9e2ad8 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:543
#5  0x00007fb93dca9ba3 in socket_poller (ctx=0x7fb92c000be0) at socket.c:2582
#6  0x00007fb949d02a51 in start_thread () from /lib64/libpthread.so.0
#7  0x00007fb94966c96d in clone () from /lib64/libc.so.6


Expected results:
There should be no glusterd crash even if a client which does not support the SSL tries to mount the SSL enabled volume.

Additional info:


I will upload the sosreport from the crashed machine.

Comment 2 Kaushal 2015-07-16 09:55:30 UTC
Upstream fix posted at http://review.gluster.org/11692

Comment 3 Kaushal 2015-07-16 09:57:41 UTC
Downstream fix posted at https://code.engineering.redhat.com/gerrit/53141

Comment 6 monti lawrence 2015-07-24 15:01:23 UTC
Doc text is edited. Please sign off to be included in Known Issues.

Comment 7 Kaushal 2015-07-27 05:01:00 UTC
The doc-text looks fine.

Comment 10 M S Vishwanath Bhat 2015-09-09 09:54:08 UTC
I just tried mounting the volume from a client where ssl is not enabled and I don't see any crash.

I did the following things.

Create a 2*2 dist-rep volume, after setting up /etc/ssl/glusterfs.ca as per the setup doc. Encryption was enabled in both I/O path and management path (/var/lib/glusterd/secure-access)

Now I tried to mount the volume from a client where ssl was not configured (secure-access and /etc/ssl/glusterfs.ca not present). The mounting failed, but no glusterd crashed. Also in glusterd log it was mentioned as SSL connection error.

[2015-09-09 09:50:48.321206] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c3f618f4216c677800277657ab6d7389.socket failed (Invalid argument)
[2015-09-09 09:50:49.129991] E [socket.c:406:ssl_setup_connection] 0-socket.management: SSL connect error
[2015-09-09 09:50:49.130136] E [socket.c:260:ssl_dump_error_stack] 0-socket.management:   error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
[2015-09-09 09:50:49.130181] E [socket.c:2442:socket_poller] 0-socket.management: server setup failed
[2015-09-09 09:50:51.322009] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c3f618f4216c677800277657ab6d7389.socket failed (Invalid argument)


I setup ssl in client now. Created secure-access and created /etc/ssl/glusterfs.ca. Tried to mount and it was mounted.


Moving the bug to verified. Please reopen if hit again.

Comment 12 errata-xmlrpc 2015-10-05 07:20:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html