Bug 762899 (GLUSTER-1167)

Isn't this the same bugs as 1054? 

http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1054

Will check it out but looks unlikely because that bugs contains a call to open
after mknod and our workload consists of just an open as the file already
exists.

#16 0x000000398c2d3c2d in clone () from /lib64/libc.so.6
No symbol table info available.
(gdb) 
(gdb) 
(gdb) 
(gdb) bt
#0  0x00002b999147cb56 in ioc_open_cbk (frame=0x14f68120, cookie=0x14f681a0, this=0x14eb63f0, op_ret=0, op_errno=0, fd=0x14f67e00) at io-cache.c:533
#1  0x00002b999126c41c in iot_open_cbk (frame=0x14f681a0, cookie=0x14f67760, this=0x14eb5490, op_ret=0, op_errno=0, fd=0x14f67e00) at io-threads.c:679
#2  0x00002b9990374da6 in default_open_cbk (frame=0x14f67760, cookie=0x14f67b40, this=0x14eb41e0, op_ret=0, op_errno=0, fd=0x14f67e00) at defaults.c:495
#3  0x00002b9990e48c37 in posix_open (frame=0x14f67b40, this=0x14eb2e50, loc=0x14f67810, flags=2, fd=0x14f67e00, wbflags=2) at posix.c:2258
#4  0x00002b999105fe94 in ac_open_resume (frame=0x14f67760, this=0x14eb41e0, loc=0x14f67810, flags=2, fd=0x14f67e00, wbflags=2) at access-control.c:1242
#5  0x00002b999038b512 in call_resume_wind (stub=0x14f677e0) at call-stub.c:2169
#6  0x00002b999039179c in call_resume (stub=0x14f677e0) at call-stub.c:3953
#7  0x00002b999105ff55 in ac_open_create_stat_cbk (frame=0x14f67760, cookie=0x14f67a30, this=0x14eb41e0, op_ret=0, op_errno=0, buf=0x41ca3e90) at access-control.c:1266
#8  0x00002b9990e40e34 in posix_stat (frame=0x14f67a30, this=0x14eb2e50, loc=0x41ca3f50) at posix.c:576
#9  0x00002b99910602be in ac_open_create (stub=0x14f677e0) at access-control.c:1293
#10 0x00002b99910607e7 in ac_open (frame=0x14f67760, this=0x14eb41e0, loc=0x14f68360, flags=2, fd=0x14f67e00, wbflags=2) at access-control.c:1376
#11 0x00002b999126c636 in iot_open_wrapper (frame=0x14f681a0, this=0x14eb5490, loc=0x14f68360, flags=2, fd=0x14f67e00, wbflags=2) at io-threads.c:688
#12 0x00002b999038b512 in call_resume_wind (stub=0x14f68330) at call-stub.c:2169
#13 0x00002b999039179c in call_resume (stub=0x14f68330) at call-stub.c:3953
#14 0x00002b9991269190 in iot_worker (data=0x14eb96d0) at io-threads.c:118
#15 0x000000398ca064a7 in start_thread () from /lib64/libpthread.so.0
#16 0x000000398c2d3c2d in clone () from /lib64/libc.so.6
(gdb) list
528	
529		if (op_ret != -1) {
530	                inode_ctx_get (fd->inode, this, &tmp_ioc_inode);
531	                ioc_inode = (ioc_inode_t *)(long)tmp_ioc_inode;
532	                        
533	                ioc_table_lock (ioc_inode->table);
534	                {
535	                        list_move_tail (&ioc_inode->inode_lru,
536	                                        &table->inode_lru[ioc_inode->weight]);
537	                }
(gdb) p ioc_inode
$1 = (ioc_inode_t *) 0x0

Created attachment 255 [details]
bugfix -- this one works, the previous was wrong

commit id :  0f37f5771ed1425c5822ae76857b44787ef564b0

When trying to stop fio -- gnfs server crashed- core is in brick8

log file-
----------
[2010-07-19 04:12:27] D [nfs3-helpers.c:2174:nfs3_log_rw_call] nfs-nfsv3: XID: a17c4ab3, READ: args: FH: hashcount 1, xlid 0, gen 5495510538991435789, ino 7758436, offset: 712597504,  count: 4096
[2010-07-19 04:12:27] T [nfs3.c:1501:nfs3_read] nfs-nfsv3: FH to Volume: brick1
[2010-07-19 04:12:27] T [nfs3-helpers.c:2799:nfs3_fh_resolve_inode] nfs-nfsv3: FH needs inode resolution
[2010-07-19 04:12:27] T [nfs3-helpers.c:2732:nfs3_fh_resolve_inode_hard] nfs-nfsv3: FH hard resolution: ino: 7758436, gen: 5495510538991435789, hashidx: 1
[2010-07-19 04:12:27] T [nfs3-helpers.c:2737:nfs3_fh_resolve_inode_hard] nfs-nfsv3: Dir will be opened: /
[2010-07-19 04:12:27] T [nfs-fops.c:394:nfs_fop_opendir] nfs: Opendir: /
[2010-07-19 04:12:27] T [access-control.c:205:ac_test_access] access-control: Root has access
[2010-07-19 04:12:27] T [nfs3-helpers.c:2530:nfs3_fh_resolve_opendir_cbk] nfs-nfsv3: Reading directory: /
[2010-07-19 04:12:27] T [nfs-fops.c:483:nfs_fop_readdirp] nfs: readdir
[2010-07-19 04:12:27] T [nfs3-helpers.c:2671:nfs3_fh_resolve_search_dir] nfs-nfsv3: Candidate: random-read.1.0, ino: 7758436, gen: 5495510538991435789
[2010-07-19 04:12:27] T [nfs3-helpers.c:2337:nfs3_fh_resolve_check_entry] nfs-nfsv3: Found entry: gen: 5495510538991435789 ino: 7758436, name: random-read.1.0
[2010-07-19 04:12:27] T [nfs3-helpers.c:2500:nfs3_fh_resolve_found] nfs-nfsv3: Candidate entry was found
[2010-07-19 04:12:27] T [nfs3-helpers.c:2423:nfs3_fh_resolve_found_entry] nfs-nfsv3: Entry got from itable
[2010-07-19 04:12:27] T [nfs3-helpers.c:2018:nfs3_file_open_and_resume] nfs-nfsv3: Opening: /random-read.1.0
[2010-07-19 04:12:27] T [nfs3-helpers.c:1946:__nfs3_queue_call_state] nfs-nfsv3: Initing inode queue
[2010-07-19 04:12:27] T [nfs3-helpers.c:1952:__nfs3_queue_call_state] nfs-nfsv3: First call in queue
[2010-07-19 04:12:27] T [nfs3-helpers.c:1957:__nfs3_queue_call_state] nfs-nfsv3: Queueing call state
[2010-07-19 04:12:27] T [nfs3-helpers.c:1998:__nfs3_file_open_and_resume] nfs-nfsv3: Opening uncached fd
[2010-07-19 04:12:27] T [nfs-fops.c:1064:nfs_fop_open] nfs: Open: /random-read.1.0
[2010-07-19 04:12:27] T [access-control.c:205:ac_test_access] access-control: Root has access
pending frames:

patchset: git://git.sv.gnu.org/gluster.git
signal received: 11
time of crash: 2010-07-19 04:12:27
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.0.0git
/lib64/libc.so.6[0x398c2302d0]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/performance/io-cache.so(ioc_open_cbk+0x240)[0x2ba4518e5890]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/performance/io-threads.so(iot_open_cbk+0x90)[0x2ba4516d73f0]
/root/laks/nfs-test//lib/libglusterfs.so.0(default_open_cbk+0x92)[0x2ba4507ff1c2]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/storage/posix.so(posix_open+0x243)[0x2ba4512c1313]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/features/access-control.so(ac_open_resume+0x128)[0x2ba4514cb868]
/root/laks/nfs-test//lib/libglusterfs.so.0(call_resume+0xd7b)[0x2ba4508083db]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/features/access-control.so(ac_open_create_stat_cbk+0x7b)[0x2ba4514ccf9b]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/storage/posix.so(posix_stat+0x153)[0x2ba4512c0873]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/features/access-control.so(ac_open_create+0x14a)[0x2ba4514ceffa]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/features/access-control.so(ac_open+0x4b)[0x2ba4514cf0ab]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/performance/io-threads.so(iot_open_wrapper+0x128)[0x2ba4516d9e48]
/root/laks/nfs-test//lib/libglusterfs.so.0(call_resume+0xd7b)[0x2ba4508083db]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/performance/io-threads.so(iot_worker+0xd6)[0x2ba4516de026]
/lib64/libpthread.so.0[0x398ca064a7]
/lib64/libc.so.6(clone+0x6d)[0x398c2d3c2d]
---------

(gdb) bt full
#0  ioc_open_cbk (frame=0xc9a9810, cookie=<value optimized out>, this=0xc8f73f0, op_ret=0, op_errno=0, fd=0xc9b9dc0) at io-cache.c:533
	tmp_ioc_inode = 0
	local = (ioc_local_t *) 0xc9a9890
	ioc_inode = (ioc_inode_t *) 0x0
	__FUNCTION__ = "ioc_open_cbk"
#1  0x00002ba4516d73f0 in iot_open_cbk (frame=0xc9c2320, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0, fd=0xc9b9dc0)
    at io-threads.c:679
	_parent = (call_frame_t *) 0xc9a9810
	old_THIS = (xlator_t *) 0xc8f6490
#2  0x00002ba4507ff1c2 in default_open_cbk (frame=0xc9a8850, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0, fd=0xc9b9dc0)
    at defaults.c:495
	fn = (ret_fn_t) 0x2ba4516d7360 <iot_open_cbk>
	_parent = (call_frame_t *) 0xc9c2320
	old_THIS = (xlator_t *) 0xc8f51e0
#3  0x00002ba4512c1313 in posix_open (frame=0xc9a9540, this=0xc8f3e50, loc=<value optimized out>, flags=2, fd=0xc9b9dc0, wbflags=2) at posix.c:2258
	_parent = (call_frame_t *) 0xc9a8850
	old_THIS = (xlator_t *) 0xc8f3e50
	op_ret = 0
	op_errno = 0
	real_path = <value optimized out>
	_fd = <value optimized out>
	pfd = <value optimized out>
	gid = 0
	stbuf = {ia_ino = 7758436, ia_gen = 5495510538991435789, ia_dev = 2051, ia_type = IA_IFREG, ia_prot = {suid = 0 '\0', sgid = 0 '\0', sticky = 0 '\0', owner = {
      read = 1 '\001', write = 1 '\001', exec = 0 '\0'}, group = {read = 1 '\001', write = 0 '\0', exec = 0 '\0'}, other = {read = 1 '\001', write = 0 '\0', 
      exec = 0 '\0'}}, ia_nlink = 1, ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 1073741824, ia_blksize = 4096, ia_blocks = 2099216, ia_atime = 1279537754, 
  ia_atime_nsec = 0, ia_mtime = 1279527532, ia_mtime_nsec = 0, ia_ctime = 1279527532, ia_ctime_nsec = 0}
	__FUNCTION__ = "posix_open"
#4  0x00002ba4514cb868 in ac_open_resume (frame=<value optimized out>, this=0xc8f51e0, loc=0xc9c1f70, flags=2, fd=0xc9b9dc0, wbflags=2) at access-control.c:1242
	_new = (call_frame_t *) 0xc9a95d8
	old_THIS = (xlator_t *) 0xc8f51e0
#5  0x00002ba4508083db in call_resume (stub=0xc9c1f40) at call-stub.c:2169
	old_THIS = (xlator_t *) 0xc8f51e0
	__FUNCTION__ = "call_resume"
#6  0x00002ba4514ccf9b in ac_open_create_stat_cbk (frame=0xc9a8850, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0, buf=0x429cdef0)
    at access-control.c:1266
	stub = (call_stub_t *) 0xc9c1f40
---Type <return> to continue, or q <return> to quit--- 
#7  0x00002ba4512c0873 in posix_stat (frame=0xc9a94c0, this=<value optimized out>, loc=0x429cdfa0) at posix.c:576
	_parent = (call_frame_t *) 0xc9a8850
	old_THIS = (xlator_t *) 0xc8f3e50
	buf = {ia_ino = 7758434, ia_gen = 5495510538991435777, ia_dev = 2051, ia_type = IA_IFDIR, ia_prot = {suid = 0 '\0', sgid = 0 '\0', sticky = 0 '\0', owner = {
      read = 1 '\001', write = 1 '\001', exec = 1 '\001'}, group = {read = 1 '\001', write = 0 '\0', exec = 1 '\001'}, other = {read = 1 '\001', write = 0 '\0', 
      exec = 1 '\001'}}, ia_nlink = 2, ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 4096, ia_blksize = 4096, ia_blocks = 16, ia_atime = 1279537947, 
  ia_atime_nsec = 0, ia_mtime = 1279527464, ia_mtime_nsec = 0, ia_ctime = 1279537882, ia_ctime_nsec = 0}
	real_path = <value optimized out>
	op_ret = 0
	op_errno = 0
	__FUNCTION__ = "posix_stat"
#8  0x00002ba4514ceffa in ac_open_create (stub=<value optimized out>) at access-control.c:1293
	_new = (call_frame_t *) 0xc9a95d8
	old_THIS = (xlator_t *) 0xc8f51e0
	ret = <value optimized out>
	parentloc = {path = 0xc91a6d0 "/", name = 0xc91a6d1 "", ino = 1, inode = 0xc8fa2a0, parent = 0x0}
	this = (xlator_t *) 0xc8f51e0
#9  0x00002ba4514cf0ab in ac_open (frame=0xc9a8850, this=0xc8f51e0, loc=<value optimized out>, flags=0, fd=<value optimized out>, wbflags=<value optimized out>)
    at access-control.c:1376
	stub = (call_stub_t *) 0x2ba450a392d8
	ret = <value optimized out>
	__FUNCTION__ = "ac_open"
#10 0x00002ba4516d9e48 in iot_open_wrapper (frame=<value optimized out>, this=0xc8f6490, loc=0xc9c23d0, flags=2, fd=0xc9b9dc0, wbflags=2) at io-threads.c:688
	_new = (call_frame_t *) 0xc9a95d8
	old_THIS = (xlator_t *) 0xc8f6490
#11 0x00002ba4508083db in call_resume (stub=0xc9c23a0) at call-stub.c:2169
	old_THIS = (xlator_t *) 0xc8f6490
	__FUNCTION__ = "call_resume"
#12 0x00002ba4516de026 in iot_worker (data=0xc8fa6c0) at io-threads.c:118
	conf = <value optimized out>
	this = <value optimized out>
	stub = (call_stub_t *) 0xc9c23a0
	sleep_till = {tv_sec = 1279538067, tv_nsec = 0}
	ret = <value optimized out>
	timeout = 0 '\0'
	bye = 0 '\0'

Created attachment 257 [details]
cat /proc/cpuinfo

This crash is not on nfs-beta-rc10 but on a months old version.

Created attachment 258 [details]
full strace -p performed (from near startup till end of thrashing ... about 1 minute of clock time)

The latest log says:
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/features/access-control.so(ac_open_create+0x265)[0x2b36d9c122be]
/root/laks/nfs-test//lib/glusterfs/3.0.0git/xlator/features/access-control.so(ac_open+0xc7)[0x2b36d9c127e7]

BUT with nfs, ac_open must never call ac_open_create because nfs never calls open with the O_CREATE flag. There is a bug in here somewhere.

Info from the latest core:

(gdb) list
528	
529		if (op_ret != -1) {
530	                inode_ctx_get (fd->inode, this, &tmp_ioc_inode);
531	                ioc_inode = (ioc_inode_t *)(long)tmp_ioc_inode;
532	                        
533	                ioc_table_lock (ioc_inode->table);
534	                {
535	                        list_move_tail (&ioc_inode->inode_lru,
536	                                        &table->inode_lru[ioc_inode->weight]);
537	                }
(gdb) p ioc_inode
$2 = (ioc_inode_t *) 0x0
(gdb) bt
#0  0x00002ad2484bfb56 in ioc_open_cbk (frame=0x2aaaac011810, cookie=0x2aaaac022070, this=0x1b867490, op_ret=0, op_errno=0, fd=0x2aaaac011510) at io-cache.c:533
#1  0x00002ad2475cada6 in default_open_cbk (frame=0x2aaaac022070, cookie=0x2aaaac0223c0, this=0x1b8661e0, op_ret=0, op_errno=0, fd=0x2aaaac011510) at defaults.c:495
#2  0x00002ad24809ec37 in posix_open (frame=0x2aaaac0223c0, this=0x1b864e50, loc=0x2aaaac022120, flags=2, fd=0x2aaaac011510, wbflags=2) at posix.c:2258
#3  0x00002ad2482b5e94 in ac_open_resume (frame=0x2aaaac022070, this=0x1b8661e0, loc=0x2aaaac022120, flags=2, fd=0x2aaaac011510, wbflags=2) at access-control.c:1242
#4  0x00002ad2475e1512 in call_resume_wind (stub=0x2aaaac0220f0) at call-stub.c:2169
#5  0x00002ad2475e779c in call_resume (stub=0x2aaaac0220f0) at call-stub.c:3953
#6  0x00002ad2482b5f55 in ac_open_create_stat_cbk (frame=0x2aaaac022070, cookie=0x2aaaac022340, this=0x1b8661e0, op_ret=0, op_errno=0, buf=0x41c978d0)
    at access-control.c:1266
#7  0x00002ad248096e34 in posix_stat (frame=0x2aaaac022340, this=0x1b864e50, loc=0x41c97990) at posix.c:576
#8  0x00002ad2482b62be in ac_open_create (stub=0x2aaaac0220f0) at access-control.c:1293
#9  0x00002ad2482b67e7 in ac_open (frame=0x2aaaac022070, this=0x1b8661e0, loc=0x2aaaaac9a408, flags=2, fd=0x2aaaac011510, wbflags=2) at access-control.c:1376
#10 0x00002ad2484c0565 in ioc_open (frame=0x2aaaac011810, this=0x1b867490, loc=0x2aaaaac9a408, flags=2, fd=0x2aaaac011510, wbflags=2) at io-cache.c:680
#11 0x00002ad2486dfb7c in nfs_fop_open (nfsx=0x1b8685d0, xl=0x1b867490, nfu=0x41c97c70, loc=0x2aaaaac9a408, flags=2, fd=0x2aaaac011510, wbflags=2, 
    cbk=0x2ad2486e17bb <nfs_inode_open_cbk>, local=0x2aaaaab06020) at nfs-fops.c:1068
#12 0x00002ad2486e1a0d in nfs_inode_open (nfsx=0x1b8685d0, xl=0x1b867490, nfu=0x41c97c70, loc=0x2aaaaac9a408, flags=2, wbflags=2, 
    cbk=0x2ad2486f9a99 <nfs3_file_open_cbk>, local=0x2aaaaac9a020) at nfs-inodes.c:215
#13 0x00002ad2486e33db in nfs_open (nfsx=0x1b8685d0, xl=0x1b867490, nfu=0x41c97c70, pathloc=0x2aaaaac9a408, flags=2, cbk=0x2ad2486f9a99 <nfs3_file_open_cbk>, 
    local=0x2aaaaac9a020) at nfs-generics.c:177
#14 0x00002ad2486f9eb5 in __nfs3_file_open_and_resume (cs=0x2aaaaac9a020) at nfs3-helpers.c:1999
#15 0x00002ad2486fa03a in nfs3_file_open_and_resume (cs=0x2aaaaac9a020, resume=0x2ad2486ea056 <nfs3_read_fd_resume>) at nfs3-helpers.c:2033
#16 0x00002ad2486ea229 in nfs3_read_resume (carg=0x2aaaaac9a020) at nfs3.c:1467
#17 0x00002ad2486fb4f2 in nfs3_fh_resolve_found_entry (cs=0x2aaaaac9a020, candidate=0x2aaaac019de0) at nfs3-helpers.c:2424
#18 0x00002ad2486fb829 in nfs3_fh_resolve_found (cs=0x2aaaaac9a020, candidate=0x2aaaac019de0) at nfs3-helpers.c:2501
#19 0x00002ad2486fbff2 in nfs3_fh_resolve_check_response (cs=0x2aaaaac9a020, candidate=0x2aaaac019de0, response=1, last_offt=581493067) at nfs3-helpers.c:2643
#20 0x00002ad2486fc196 in nfs3_fh_resolve_search_dir (cs=0x2aaaaac9a020, entries=0x41c982d0) at nfs3-helpers.c:2679
#21 0x00002ad2486fc289 in nfs3_fh_resolve_readdir_cbk (frame=0x2aaaac009278, cookie=0x1b8685d0, this=0x1b867490, op_ret=4, op_errno=2, entries=0x41c982d0)
    at nfs3-helpers.c:2701
#22 0x00002ad2486dbb8d in nfs_fop_readdirp_cbk (frame=0x2aaaac009278, cookie=0x1b8685d0, this=0x1b867490, op_ret=4, op_errno=2, entries=0x41c982d0) at nfs-fops.c:463
#23 0x00002ad2475cfc1c in default_readdirp_cbk (frame=0x2aaaac019c60, cookie=0x2aaaac019ce0, this=0x1b867490, op_ret=4, op_errno=2, entries=0x41c982d0)
    at defaults.c:1247
#24 0x00002ad2475cfc1c in default_readdirp_cbk (frame=0x2aaaac019ce0, cookie=0x2aaaac019d60, this=0x1b8661e0, op_ret=4, op_errno=2, entries=0x41c982d0)
    at defaults.c:1247
#25 0x00002ad2480a65fe in posix_do_readdir (frame=0x2aaaac019d60, this=0x1b864e50, fd=0x2aaaac019f40, size=65536, off=0, whichop=44) at posix.c:4070
#26 0x00002ad2480a66c3 in posix_readdirp (frame=0x2aaaac019d60, this=0x1b864e50, fd=0x2aaaac019f40, size=65536, off=0) at posix.c:4091
#27 0x00002ad2475d003a in default_readdirp (frame=0x2aaaac019ce0, this=0x1b8661e0, fd=0x2aaaac019f40, size=65536, off=0) at defaults.c:1274
#28 0x00002ad2475d003a in default_readdirp (frame=0x2aaaac019c60, this=0x1b867490, fd=0x2aaaac019f40, size=65536, off=0) at defaults.c:1274
#29 0x00002ad2486dbf05 in nfs_fop_readdirp (nfsx=0x1b8685d0, xl=0x1b867490, nfu=0x41c98630, dirfd=0x2aaaac019f40, bufsize=65536, offset=0, 
    cbk=0x2ad2486fc1a1 <nfs3_fh_resolve_readdir_cbk>, local=0x2aaaaac9a020) at nfs-fops.c:487
#30 0x00002ad2486e2f9f in nfs_readdirp (nfsx=0x1b8685d0, xl=0x1b867490, nfu=0x41c98630, dirfd=0x2aaaac019f40, bufsize=65536, offset=0, 
    cbk=0x2ad2486fc1a1 <nfs3_fh_resolve_readdir_cbk>, local=0x2aaaaac9a020) at nfs-generics.c:72
#31 0x00002ad2486fba5d in nfs3_fh_resolve_opendir_cbk (frame=0x2aaaac000ef8, cookie=0x1b8685d0, this=0x1b867490, op_ret=0, op_errno=22, fd=0x2aaaac019f40)
    at nfs3-helpers.c:2541
#32 0x00002ad2486e292d in nfs_inode_opendir_cbk (frame=0x2aaaac000ef8, cookie=0x1b8685d0, this=0x1b867490, op_ret=0, op_errno=22, fd=0x2aaaac019f40)
    at nfs-inodes.c:539
#33 0x00002ad2486db322 in nfs_fop_opendir_cbk (frame=0x2aaaac000ef8, cookie=0x1b8685d0, this=0x1b867490, op_ret=0, op_errno=22, fd=0x2aaaac019f40) at nfs-fops.c:377
#34 0x00002ad2475cc07c in default_opendir_cbk (frame=0x2aaaac000f80, cookie=0x2aaaac019fb0, this=0x1b867490, op_ret=0, op_errno=22, fd=0x2aaaac019f40)
    at defaults.c:689
#35 0x00002ad2475cc07c in default_opendir_cbk (frame=0x2aaaac019fb0, cookie=0x2aaaac009120, this=0x1b8661e0, op_ret=0, op_errno=22, fd=0x2aaaac019f40)
    at defaults.c:689
#36 0x00002ad248098790 in posix_opendir (frame=0x2aaaac009120, this=0x1b864e50, loc=0x2aaaac019a40, fd=0x2aaaac019f40) at posix.c:954
#37 0x00002ad2482b79eb in ac_opendir_resume (frame=0x2aaaac019fb0, this=0x1b8661e0, loc=0x2aaaac019a40, fd=0x2aaaac019f40) at access-control.c:1537
#38 0x00002ad2475e19de in call_resume_wind (stub=0x2aaaac019a10) at call-stub.c:2368
#39 0x00002ad2475e779c in call_resume (stub=0x2aaaac019a10) at call-stub.c:3953
#40 0x00002ad2482b7aa3 in ac_opendir_stat_cbk (frame=0x2aaaac019fb0, cookie=0x2aaaac000ce0, this=0x1b8661e0, op_ret=0, op_errno=0, buf=0x41c98a80)
    at access-control.c:1559
#41 0x00002ad248096e34 in posix_stat (frame=0x2aaaac000ce0, this=0x1b864e50, loc=0x2aaaaac9a408) at posix.c:576
#42 0x00002ad2482b7e0e in ac_opendir (frame=0x2aaaac019fb0, this=0x1b8661e0, loc=0x2aaaaac9a408, fd=0x2aaaac019f40) at access-control.c:1585
#43 0x00002ad2475cc280 in default_opendir (frame=0x2aaaac000f80, this=0x1b867490, loc=0x2aaaaac9a408, fd=0x2aaaac019f40) at defaults.c:701
#44 0x00002ad2486db6a4 in nfs_fop_opendir (nfsx=0x1b8685d0, xl=0x1b867490, nfu=0x41c98d60, pathloc=0x2aaaaac9a408, dirfd=0x2aaaac019f40, 
    cbk=0x2ad2486e286f <nfs_inode_opendir_cbk>, local=0x2aaaaab06020) at nfs-fops.c:398
#45 0x00002ad2486e2aba in nfs_inode_opendir (nfsx=0x1b8685d0, xl=0x1b867490, nfu=0x41c98d60, loc=0x2aaaaac9a408, cbk=0x2ad2486fb888 <nfs3_fh_resolve_opendir_cbk>, 
    local=0x2aaaaac9a020) at nfs-inodes.c:563
#46 0x00002ad2486e3963 in nfs_opendir (nfsx=0x1b8685d0, fopxl=0x1b867490, nfu=0x41c98d60, pathloc=0x2aaaaac9a408, cbk=0x2ad2486fb888 <nfs3_fh_resolve_opendir_cbk>, 
    local=0x2aaaaac9a020) at nfs-generics.c:318
#47 0x00002ad2486fc4e7 in nfs3_fh_resolve_inode_hard (cs=0x2aaaaac9a020) at nfs3-helpers.c:2738
#48 0x00002ad2486fc889 in nfs3_fh_resolve_inode (cs=0x2aaaaac9a020) at nfs3-helpers.c:2803
#49 0x00002ad2486fc9f8 in nfs3_fh_resolve_and_resume (cs=0x2aaaaac9a020, fh=0x41c98ed0, entry=0x0, resum_fn=0x2ad2486ea1ba <nfs3_read_resume>) at nfs3-helpers.c:2842
#50 0x00002ad2486ea54c in nfs3_read (req=0x1b918240, fh=0x41c98ed0, offset=712597504, count=4096) at nfs3.c:1506
#51 0x00002ad2486ea691 in nfs3svc_read (req=0x1b918240) at nfs3.c:1540
#52 0x00002ad24891d852 in rpcsvc_handle_rpc_call (conn=0x1b88a9e0) at rpcsvc.c:1849
#53 0x00002ad24891e651 in rpcsvc_record_update_state (conn=0x1b88a9e0, dataread=0) at rpcsvc.c:2329
#54 0x00002ad24891e7b9 in rpcsvc_conn_data_poll_in (conn=0x1b88a9e0) at rpcsvc.c:2372
#55 0x00002ad24891ec15 in rpcsvc_conn_data_handler (fd=11, idx=3, data=0x1b88a9e0, poll_in=1, poll_out=0, poll_err=0) at rpcsvc.c:2501
#56 0x00002ad2475ec2b9 in event_dispatch_epoll_handler (event_pool=0x1b869f70, events=0x1b8754e0, i=0) at event.c:804
(gdb) fr 14
#14 0x00002ad2486f9eb5 in __nfs3_file_open_and_resume (cs=0x2aaaaac9a020) at nfs3-helpers.c:1999
1999	        ret = nfs_open (cs->nfsx, cs->vol, &nfu, &cs->resolvedloc, O_RDWR,
(gdb)

Created attachment 259 [details]
Bugfixed version HU file

This is not the exact crash seen in 855 but the fixes submitted for 855 fix this bug too. Verified by git bisect. No need for separate regression test.

There is however one bug in access control that I will classify as normal.

        /* If we are not supposed to create the file then there is no need to
         * check the parent dir permissions. */
        if (!(flags & O_CREAT))
                ret = ac_open_create (stub);
        else
                ret = ac_open_only (stub);

This should be:

        if (flags & O_CREAT)
                ret = ac_open_create (stub);
        else
                ret = ac_open_only (stub);

But it does not cause any problems in the open because it only results in one extra stat on the parent directory. It also does not modify the flags variable, so the posix_open behaviour or ioc_open behaviour does not change because of this bug.

*** This bug has been marked as a duplicate of bug 855 ***

PATCH: http://patches.gluster.com/patch/3923 in master (access-control: Fix O_CREATE determination check)