Bug 1218422

Related link:

https://github.com/nfs-ganesha/nfs-ganesha/issues/66

Description of problem:

Cache inode refcount is not handled correctly when there is an error; which results into one thread freeing up memory for cache_inode_entry while another thread is still accessing it. Problem is the below section of the code.

src/include/cache_inode_hash.h#cih_remove_checked()

code (with fix - adding inavl check).

    if (entry->fh_hk.inavl)
    {
        PTHREAD_RWLOCK_wrlock(&cp->lock);
        if (entry->fh_hk.inavl)
        {
            node = cih_fhcache_inline_lookup(&cp->t, &entry->fh_hk.node_k);
            if (node)
            {
                avltree_remove(node, &cp->t);
                cp->cache[cih_cache_offsetof(&cih_fhcache,
                                            entry->fh_hk.key.hk)] = NULL;
                entry->fh_hk.inavl = false;
                /* return sentinel ref */
                cache_inode_lru_unref(entry, LRU_FLAG_NONE);
            }
        }
        PTHREAD_RWLOCK_unlock(&cp->lock);
    }

Version-Release number of selected component (if applicable):
v2.0

How reproducible:
very easily. Below are the steps to reproduce it.

Steps to Reproduce:
1. We need to alter fsal code in the following way to reproduce this issue.
2. Make sure that ops->getattr and ops->write returns EIO for 25% of the calls. something like this.

// sample code..
uint64_t op_cnt = 1;

bool raise_io()
{
    uint64_t cnt = atomic_inc_uint64_t(&op_cnt);
    if (cnt < 200)
    {
        return false;
    }
    return !(cnt % 4);
}

3. Now once this is setup. Make sure that you mount the export locally and copy huge file. Something like 650MB ISO file (you can generate it with "dd" too). File contents doesn't matter. We are making sure that size of the file is large such that you get lot of parallel writes for the same file.
4) Try to repeat this operation multiple times and you should get a coredump. 

Actual results:

Cache_inode_refcount drops to 0 while one of the threads still have access to it.

Expected results:

Cache inode should get released correctly from cache. Which also means once cache_inode_entry refcount drops to 0 then you no thread should get access to it.

Additional info:

This is one of the symptoms. In which case thread may access the cache_inode_entry while it is freed up.

related stack trace
#0 0x00000000004cb43e in cache_inode_lock_trust_attrs (entry=0x7fa70be5bc80, need_wr_lock=false) at /home/ovj/nfs-ganesha-hdvg/src/cache_inode/cache_inode_misc.c:899
#1 0x000000000047e47d in cache_entry_to_nfs3_Fattr (entry=0x7fa70be5bc80, Fattr=0x7fa6a0991f70) at /home/ovj/nfs-ganesha-hdvg/src/Protocols/NFS/nfs_proto_tools.c:3567
#2 0x0000000000479a4a in nfs_SetPostOpAttr (entry=0x7fa70be5bc80, attr=0x7fa6a0991f68) at /home/ovj/nfs-ganesha-hdvg/src/Protocols/NFS/nfs_proto_tools.c:77
#3 0x0000000000479c3c in nfs_SetWccData (before_attr=0x0, entry=0x7fa70be5bc80, wcc_data=0x7fa6a0991f48) at /home/ovj/nfs-ganesha-hdvg/src/Protocols/NFS/nfs_proto_tools.c:130
#4 0x0000000000456477 in nfs3_write (arg=0x7fa69047f740, worker=0x7fa70bc70180, req=0x7fa69047f688, res=0x7fa6a0991f40) at /home/ovj/nfs-ganesha-hdvg/src/Protocols/NFS/nfs3_write.c:269
#5 0x000000000044c919 in nfs_rpc_execute (req=0x7fa69204c280, worker_data=0x7fa70bc70180) at /home/ovj/nfs-ganesha-hdvg/src/MainNFSD/nfs_worker_thread.c:1257
#6 0x000000000044d2a5 in worker_run (ctx=0x7fa7157e2300) at /home/ovj/nfs-ganesha-hdvg/src/MainNFSD/nfs_worker_thread.c:1509