Observations:
1. The crash happens after a call bail out:
[2010-04-21 09:55:04] E [client-protocol.c:313:call_bail]
ec2-75-101-197-117.compute-1.amazonaws.com-1: bailing out frame FINODELK(31

2. Within client_finodelk_cbk the core points to the STACK_UNWIND.

3. I've checked the sanity of various variables that are inited or use inside STACK_UNWIND.

(gdb) p *frame
$10 = {root = 0x2aaab012c910, parent = 0x2aaab0170d20, next = 0x2aaab0170d20, prev = 0x2aaab012c998, local = 0x0, this = 0x6142a0, 
  ret = 0x2aaaac189992 <afr_lock_cbk>, ref_count = 0, lock = 1, cookie = 0x0, complete = _gf_true, op = GF_FOP_STAT, begin = {tv_sec = 0, 
    tv_usec = 0}, end = {tv_sec = 0, tv_usec = 0}}

The following variables are being used from frame:
                fn = frame->ret;                                        \
                _parent = frame->parent;                                \
                _parent->ref_count--;                                   \
                old_THIS = THIS;                                        \
                THIS = _parent->this;                                   \
                frame->complete = _gf_true;                             \
                                                                        \
                if (((xlator_t *) old_THIS)->ctx->measure_latency) {    \
                        gettimeofday (&frame->end, NULL);               \
                }                                                       \
                                                           

Printing each of these in order:

(gdb) p fn
$15 = (ret_fn_t) 0x2aaaac189992 <afr_lock_cbk>


(gdb) p _parent
$16 = (call_frame_t *) 0x2aaab0170d20


(gdb) p *_parent
$17 = {root = 0x2aaab012c910, parent = 0x2aaab02b52c0, next = 0x2aaab02b52c0, prev = 0x2aaab0117b60, local = 0x2aaab010fcc0, this = 0x616210, 
  ret = 0x2aaaac3cffdc <dht_writev_cbk>, ref_count = 0, lock = 1, cookie = 0x2aaab0170d20, complete = _gf_false, op = GF_FOP_STAT, begin = {
    tv_sec = 0, tv_usec = 0}, end = {tv_sec = 0, tv_usec = 0}}


(gdb) p *_parent->this
$18 = {name = 0x6161f0 "mirror-1", type = 0x616b70 "cluster/replicate", next = 0x6150b0, prev = 0x616d20, parents = 0x617de0, 
  children = 0x616c60, options = 0x615a30, fops = 0x2aaaac3b30a0, mops = 0x2aaaac3b34e0, cbks = 0x2aaaac3b3360, dumpops = 0x2aaaac3b3340, 
  volume_options = {next = 0x616be0, prev = 0x616be0}, fini = 0x2aaaac16a1de <fini>, init = 0x2aaaac1694fc <init>, 
  notify = 0x2aaaac169175 <notify>, latencies = {{min = 0, max = 0, total = 0, std = 0, mean = 0, count = 0} <repeats 45 times>}, 
  ctx = 0x60b010, itable = 0x0, ready = 1 '\001', init_succeeded = 1 '\001', private = 0x620000}


(gdb) p old_THIS
$19 = (xlator_t *) 0x2aaaaaf34640


(gdb) p *old_THIS
$20 = {name = 0x2aaaaad16a51 "glusterfs", type = 0x2aaaaad16a5b "global", next = 0x0, prev = 0x0, parents = 0x0, children = 0x0, 
  options = 0x0, fops = 0x0, mops = 0x0, cbks = 0x0, dumpops = 0x0, volume_options = {next = 0x0, prev = 0x0}, fini = 0, init = 0, 
  notify = 0, latencies = {{min = 0, max = 0, total = 0, std = 0, mean = 0, count = 0} <repeats 45 times>}, ctx = 0x0, itable = 0x0, 
  ready = 0 '\0', init_succeeded = 0 '\0', private = 0x0}

old_THIS->ctx is NULL and that is referenced next in:

if (((xlator_t *) old_THIS)->ctx->measure_latency) {    \
                        gettimeofday (&frame->end, NULL);               \
                }                                                       \
                           

This may be a hint of the problem. Continuing to investigate.

This results in a crash in STACK_UNWIND after a call bail out. Since the call bail does not occur in the context of an xlator, the old_THIS in STACK_UNWIND is handed a global_xlator which did not have its ctx set, resulting in a NULL de-reference.

PATCH: http://patches.gluster.com/patch/3155 in master (core: Set ctx in global_xlator to prevent NULL ptr reference)

Regression Test:
The crash happens due to dereferencing of a call-bail out. We can simulate a call bailout by setting the frame-timeout in proto/client to a very low value so that even a slight amount of IO load can result in a bail-out. Do not use io-threads. It increases concurrency and reduces wait times so it'll be hard to produce a bail-out with that in the graph.

Test Case:


1. Before mounting on the NFS client, run the following command:
 $ echo "128" > /proc/sys/sunrpc/tcp_slot_table_entries

This will increase the amount of load generated by the NFS client.

2. Mount the nfs server. You'll obviously have to have  a network link between proto/client and server so that  the network delay also makes it easier to simulate call bail-out. Feel free to use a simple posix config minus io-threads under the protocol/server.

3. On the NFS mount. Run:
$ iozone -t 16 -s 1g -r 1m -F <mountpoint>/f1 <mountpoint>/f2......so on

We're starting 16 threads to generate as much load as possible on the disks at the server, so that the increased IO response time combined with the low frame-timeout will cause a call bail-out.

When the call bailout happens you'll observe either that the dd fails, or you'll have to manually inspect the log file to confirm, in case the NFS client retransmits the write request.

Verified with nfs beta rc7 ,it's fixed now.