Hide Forgot
Create a volume with tcp,rdma transport type. mounted via nfs client in another machine and started running the dbench. Now nfs server crashed with the folowing back trace Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff57c75000 Core was generated by `/usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/'. Program terminated with signal 11, Segmentation fault. #0 0x00002aaaaad0f12b in afr_changelog_post_op_cbk (frame=0x2acb959d7990, cookie=0x2acb95733028, this=0x10e2e2b0, op_ret=0, op_errno=22, xattr=0x136cd670) at ../../../../../xlators/cluster/afr/src/afr-transaction.c:347 347 call_count = --local->call_count; (gdb) bt #0 0x00002aaaaad0f12b in afr_changelog_post_op_cbk (frame=0x2acb959d7990, cookie=0x2acb95733028, this=0x10e2e2b0, op_ret=0, op_errno=22, xattr=0x136cd670) at ../../../../../xlators/cluster/afr/src/afr-transaction.c:347 #1 0x00002aaaaaac501a in client3_1_xattrop_cbk (req=0x2aaab1750730, iov=0x2aaab1750770, count=1, myframe=0x2acb95733028) at ../../../../../xlators/protocol/client/src/client3_1-fops.c:1425 #2 0x00002acb94a8e25e in rpc_clnt_handle_reply (clnt=0x124a9cd0, pollin=0x143b95e0) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:789 #3 0x00002acb94a8e586 in rpc_clnt_notify (trans=0x124a57f0, mydata=0x124a9d00, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x143b95e0) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:902 #4 0x00002acb94a8a9f3 in rpc_transport_notify (this=0x124a57f0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x143b95e0) at ../../../../rpc/rpc-lib/src/rpc-transport.c:498 #5 0x00002aaaade75044 in rdma_pollin_notify (peer=0x124a5c18, post=0x12011dc0) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:3085 #6 0x00002aaaade7538e in rdma_recv_reply (peer=0x124a5c18, post=0x12011dc0) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:3172 #7 0x00002aaaade756ab in rdma_process_recv (peer=0x124a5c18, wc=0x4421e0d0) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:3262 #8 0x00002aaaade7593e in rdma_recv_completion_proc (data=0x10e345e0) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:3347 #9 0x000000328420673d in start_thread () from /lib64/libpthread.so.0 #10 0x0000003283ad40cd in clone () from /lib64/libc.so.6 (gdb) f 1 #1 0x00002aaaaaac501a in client3_1_xattrop_cbk (req=0x2aaab1750730, iov=0x2aaab1750770, count=1, myframe=0x2acb95733028) at ../../../../../xlators/protocol/client/src/client3_1-fops.c:1425 1425 STACK_UNWIND_STRICT (xattrop, frame, op_ret, (gdb) f 2 #2 0x00002acb94a8e25e in rpc_clnt_handle_reply (clnt=0x124a9cd0, pollin=0x143b95e0) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:789 789 req->cbkfn (req, req->rsp, req->rspcnt, saved_frame->frame); I have attached the client log. core file is too big to attach. I have archived it.
What is the version of the QA release for the crash?.
(In reply to comment #1) > What is the version of the QA release for the crash?. It's 3.3.0qa14
(In reply to comment #2) > (In reply to comment #1) > > What is the version of the QA release for the crash?. > > It's 3.3.0qa14 Vishwa, Did you get a chance to verify if it is due to the rdma corruption bug? Pranith
(In reply to comment #3) > (In reply to comment #2) > > (In reply to comment #1) > > > What is the version of the QA release for the crash?. > > > > It's 3.3.0qa14 > > Vishwa, > why is here op_ret set to '-1' for an EINVAL? doesn't it means this can be happening from an incorrectly sent return value?.
(In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #2) > > > (In reply to comment #1) > > > > What is the version of the QA release for the crash?. > > > > > > It's 3.3.0qa14 > > > > Vishwa, > > > why is here op_ret set to '-1' for an EINVAL? doesn't it means this can be > happening from an incorrectly sent return value?. Where do you see the op_ret -1?. We should not look at the op_errno unless op_ret is set to -1. Pranith
(In reply to comment #3) > (In reply to comment #2) > > (In reply to comment #1) > > > What is the version of the QA release for the crash?. > > > > It's 3.3.0qa14 > > Vishwa, > Did you get a chance to verify if it is due to the rdma corruption bug? > > Pranith Vishwa, I am moving the bug to INFO-REQUESTED Pranith
*** This bug has been marked as a duplicate of bug 766603 ***