Bug 1449495

Created attachment 1277525 [details]
rdma.patch

Description of problem:

In 3.10.1, glfsheal with disperse volume is always crushed in RDMA environment.

Version-Release number of selected component (if applicable): v3.10.1

How reproducible:

Steps to Reproduce:

1. install Mellanox OFED packages(librdmacm, libibverbs, etc.)
2. gluster volume create <vol> disperse 3 server{1..4}:<vol> transport rdma
3. run glfsheal

Actual results:

[root@server-1 ~]# glfsheal IBTEST                                                                                      Segmentation fault (core dumped)  

Expected results:

[root@server-1 ~]# glfsheal IBTEST
Brick 10.10.1.220:/volume/IBTEST
<gfid:d338c46e-bff6-4da0-b962-590ef3a19102> - Is in split-brain

...

Additional info:

- coredump with gdb

Core was generated by `/usr/sbin/glfsheal IBTEST'.                                                                     
Program terminated with signal 11, Segmentation fault.                                                                                                                                               
#0  0x00007efc1467b56c in __gf_rdma_teardown (this=0x7efc08032740) at rdma.c:3255
3255            if (peer->cm_id->qp != NULL) {                                                                                                                                   
(gdb) bt                                                                                                                                                                                                   
#0  0x00007efc1467b56c in __gf_rdma_teardown (this=0x7efc08032740) at rdma.c:3255                                                                        
#1  0x00007efc1467b6b0 in gf_rdma_teardown (this=0x7efc08032740, port=<value optimized out>) at rdma.c:3287    
#2  gf_rdma_connect (this=0x7efc08032740, port=<value optimized out>) at rdma.c:4769                                           
#3  0x00007efc23df24e9 in rpc_clnt_reconnect (conn_ptr=0x7efc080325d0) at rpc-clnt.c:422                                                                   
#4  0x00007efc23df25d6 in rpc_clnt_start (rpc=0x7efc080325a0) at rpc-clnt.c:1210                                 
#5  0x00007efc16b55c53 in notify (this=0x7efc0801cec0, event=1, data=0x7efc0801e930) at client.c:2354                                                                                    
#6  0x00007efc2423f592 in xlator_notify (xl=0x7efc0801cec0, event=1, data=0x7efc0801e930) at xlator.c:566                                                                  
#7  0x00007efc242c07c7 in default_notify (this=0x7efc0801e930, event=1, data=0x7efc08020270) at defaults.c:3090
#8  0x00007efc155d5d18 in notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at snapview-client.c:2393
#9  0x00007efc2423f592 in xlator_notify (xl=0x7efc0801e930, event=1, data=0x7efc08020270) at xlator.c:566          
#10 0x00007efc242c07c7 in default_notify (this=0x7efc08020270, event=1, data=0x7efc08021ea0) at defaults.c:3090                                                                
#11 0x00007efc153ba69e in notify (this=0x7efc08020270, event=<value optimized out>, data=0x7efc08021ea0) at io-stats.c:3991                                                                  
#12 0x00007efc2423f592 in xlator_notify (xl=0x7efc08020270, event=1, data=0x7efc08021ea0) at xlator.c:566              
#13 0x00007efc242c07c7 in default_notify (this=0x7efc08021ea0, event=1, data=0x7efc08021ea0) at defaults.c:3090                
#14 0x00007efc2423f592 in xlator_notify (xl=0x7efc08021ea0, event=1, data=0x7efc08021ea0) at xlator.c:566                                                                            
#15 0x00007efc24279bae in glusterfs_graph_parent_up (graph=<value optimized out>) at graph.c:442        
#16 0x00007efc24279fb2 in glusterfs_graph_activate (graph=0x7efc08003990, ctx=0x7eb1a0) at graph.c:711                           
#17 0x00007efc23bc9f91 in glfs_process_volfp (fs=<value optimized out>, fp=0x7efc08003710) at glfs-mgmt.c:79                                                                     
#18 0x00007efc23bca3aa in glfs_mgmt_getspec_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7efc080012d0) at glfs-mgmt.c:665
#19 0x00007efc23df2ad5 in rpc_clnt_handle_reply (clnt=0x876540, pollin=0x7efc080028c0) at rpc-clnt.c:793                                                                                                                                      #20 0x00007efc23df3c85 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x876570, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7efc080028c0) at rpc-clnt.c:986              
#21 0x00007efc23deed68 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:538
#22 0x00007efc16fd19bd in socket_event_poll_in (this=0x876740) at socket.c:2268                                          
#23 0x00007efc16fd2cbe in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0x876740, poll_in=1, poll_out=0, poll_err=0) at socket.c:2398                  
#24 0x00007efc242a0716 in event_dispatch_epoll_handler (data=0x7efc10000920) at event-epoll.c:572
#25 event_dispatch_epoll_worker (data=0x7efc10000920) at event-epoll.c:675                                                 
#26 0x00007efc2351daa1 in start_thread (arg=0x7efc17be0700) at pthread_create.c:301                                                                                                
#27 0x00007efc2326abcd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
#0  0x00007f5f9a28956c in __gf_rdma_teardown (this=0x7f5f94032780) at rdma.c:3255
(gdb) list
3250            gf_rdma_peer_t    *peer = NULL;
3251
3252            priv = this->private;
3253            peer = &priv->peer;
3254
3255            if (peer->cm_id->qp != NULL) {
3256                    __gf_rdma_destroy_qp (this);
3257            }
3258
3259            if (!list_empty (&priv->peer.ioq)) {
(gdb) print peer->cm_id
$4 = (struct rdma_cm_id *) 0x0

In my opinion, It is caused by wrong exception handling at __gf_rdma_teardown() (https://github.com/gluster/glusterfs/blob/master/rpc/rpc-transport/rdma/src/rdma.c#L3256).

If an error has occured with rdma_create_id() in gf_rdma_connect(), process will jump to the 'unlock' label and then call gf_rdma_teardown() which call __gf_rdma_teardown().

Presently, __gf_rdma_teardown() checks InifiniBand QP with peer->cm_id->qp!

Unfortunately, cm_id is not allocated and will be crushed in this situation :)

I attach ugly patch for resolving this issue!