Hide Forgot
Here is the time taken for 'ls -l' on a directory containing 100,000 files to complete. with stat-prefetch: =================== [root@brick5 stat-prefetch]# time LD_PRELOAD=/opt/glusterfs/2.0.7rc5/lib/glusterfs/glusterfs-booster.so ls -l /mnt/test/ > /dev/null real 5m9.580s user 3m54.671s sys 0m9.469s without stat-prefetch: ====================== [root@brick5 stat-prefetch]# time LD_PRELOAD=/opt/glusterfs/2.0.7rc5/lib/glusterfs/glusterfs-booster.so ls -l /mnt/test/ > /dev/null real 1m18.921s user 0m2.360s sys 0m2.532s The callgrind report shows ls spending most of its time in sp_cache_get_entry, which has to traverse a list of atmost 100,000 entries. The procedure occupying second slot in consumption of time is strcmp used in sp_cache_get_entry to compare the name being looked upon to the entry already cached.
After migrating stat-prefetch to use rbtree based hash table to store dentry cache, performance increased and here are the times for 'ls -l' to complete on a directory containing 375,000 4k files. without stat-prefetch: ====================== [root@brick5 stat-prefetch]# time LD_PRELOAD=/opt/glusterfs/raghu.git/lib/glusterfs/glusterfs-booster.so ls -l /mnt/test/ > /dev/null real 5m19.846s user 0m6.100s sys 0m4.556s with stat-prefetch: =================== [root@brick5 stat-prefetch]# time LD_PRELOAD=/opt/glusterfs/raghu.git/lib/glusterfs/glusterfs-booster.so ls -l /mnt/test/ > /dev/null real 4m32.724s user 0m12.097s sys 0m9.761s
Created attachment 76 [details] patch to fix lpr-0.50-1
Changes done till now: 1. migration of cache from list to rbtree based hash table 2. bug-fix to sp_lookup to fetch entry from cache even when xattr_req is not NULL, but is empty. 3. fetching cache from inode when cache is not present in parent for directories, so that we can look for entry "." in the cache. 4. memory leak fixes (bug: 257) with these changes following are the time taken for 'ls -l' to complete on a directory containing 375,000 4k files, without stat-prefetch: ====================== [root@brick5 stat-prefetch]# time ls -l /mnt/raghu/test >/dev/null real 3m41.187s user 0m2.876s sys 0m3.488s with stat-prefetch: =================== [root@brick5 stat-prefetch]# time ls -l /mnt/raghu/test >/dev/null real 1m23.090s user 0m3.236s sys 0m6.572s 5. with lookup-behind removed from sp_lookup (but will be done on-demand basis in ops like open, chmod etc which are acting on a path), [root@brick5 stat-prefetch]# time ls -l /mnt/raghu/test >/dev/null real 0m52.442s user 0m3.532s sys 0m7.356s 6. instead of storing the entire dentry, a custom structure containing only name and stat structure can be stored in cache. And this structure should be allocated using mem-pool. while changes 1-4 are complete and patches submitted, 5 and 6 are still to be implemented.
PATCH: http://patches.gluster.com/patch/1583 in release-2.0 (mem-pool: add mem_pool_destroy.)
PATCH: http://patches.gluster.com/patch/1720 in release-2.0 (performance/stat-prefetch: change the cache to use rbtree based hashtable instead of list for caching dentries.)
PATCH: http://patches.gluster.com/patch/1721 in release-2.0 (performance/stat-prefetch: change behaviour of sp_lookup when xattr_req is not NULL.)
PATCH: http://patches.gluster.com/patch/1722 in release-2.0 (performance/stat-prefetch: optimizations to sp_lookup.)
PATCH: http://patches.gluster.com/patch/1723 in release-2.0 (performance/stat-prefetch: Add accounting of cache hits and misses.)
PATCH: http://patches.gluster.com/patch/1759 in release-2.0 (performance/stat-prefetch: remove lookup-behind in sp_lookup.)
PATCH: http://patches.gluster.com/patch/1760 in release-2.0 (performance/stat-prefetch: Add support code to implement lookup-behind.)
PATCH: http://patches.gluster.com/patch/1761 in release-2.0 (performance/stat-prefetch: use op_errno instead of errno to store error code.)
PATCH: http://patches.gluster.com/patch/1762 in release-2.0 (performance/stat-prefetch: implement procedure sp_process_inode_ctx.)
PATCH: http://patches.gluster.com/patch/1763 in release-2.0 (performance/stat-prefetch: lookup the path in open if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1764 in release-2.0 (performance/stat-prefetch: do lookup on the path in checksum if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1765 in release-2.0 (performance/stat-prefetch: lookup the path in stat if the it is not already looked-up.)
PATCH: http://patches.gluster.com/patch/1766 in release-2.0 (performance/stat-prefetch: lookup the path in chmod if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1767 in release-2.0 (performance/stat-prefetch: lookup the path in chown if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1768 in release-2.0 (performance/stat-prefetch: do lookup in truncate if the path is not already looked-up)
PATCH: http://patches.gluster.com/patch/1769 in release-2.0 (performance/stat-prefetch: do lookup in utimens if the path is not already looked up)
PATCH: http://patches.gluster.com/patch/1770 in release-2.0 (performance/stat-prefetch: implement sp_access.)
PATCH: http://patches.gluster.com/patch/1771 in release-2.0 (performance/stat-prefetch: do lookup in mknod if the path is not already looked up.)
PATCH: http://patches.gluster.com/patch/1772 in release-2.0 (performance/stat-prefetch: do lookup in readlink if the path is not already looked up.)
PATCH: http://patches.gluster.com/patch/1773 in release-2.0 (performance/stat-prefetch: do lookup in unlink if the path has not already been looked up.)
PATCH: http://patches.gluster.com/patch/1774 in release-2.0 (performance/stat-prefetch: lookup the path in mkdir if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1775 in release-2.0 (performance/stat-prefetch: do lookup in rmdir if the path has not already been looked up.)
PATCH: http://patches.gluster.com/patch/1776 in release-2.0 (performance/stat-prefetch: do lookup of path in symlink if it is not already looked-up.)
PATCH: http://patches.gluster.com/patch/1777 in release-2.0 (performance/stat-prefetch: lookup oldpath and newpath in rename if they've not already been looked up.)
PATCH: http://patches.gluster.com/patch/1778 in release-2.0 (performance/stat-prefetch: lookup oldloc->path in link if it has not already been looked up.)
PATCH: http://patches.gluster.com/patch/1779 in release-2.0 (performance/stat-prefetch: remove stat corresponding to oldloc->path from cache in sp_link.)
PATCH: http://patches.gluster.com/patch/1780 in release-2.0 (performance/stat-prefetch: lookup path in create if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1781 in release-2.0 (performance/stat-prefetch: lookup path in opendir if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1782 in release-2.0 (performance/stat-prefetch: lookup path in setxattr if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1783 in release-2.0 (performance/stat-prefetch: implement sp_getxattr.)
PATCH: http://patches.gluster.com/patch/1784 in release-2.0 (performance/stat-prefetch: do lookup on path in removexattr if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1785 in release-2.0 (performance/stat-prefetch: implement sp_inodelk.)
PATCH: http://patches.gluster.com/patch/1786 in release-2.0 (performance/stat-prefetch: implement sp_entrylk.)
PATCH: http://patches.gluster.com/patch/1787 in release-2.0 (performance/stat-prefetch: lookup the path in xattrop if it is not already looked up.)
PATCH: http://patches.gluster.com/patch/1788 in release-2.0 (performance/stat-prefetch: checking for cache and creation if not present is made atomic.)
Just a reminder. Following still needs to be completed. 6. instead of storing the entire dentry, a custom structure containing only name and stat structure can be stored in cache. And this structure should be allocated using mem-pool.
PATCH: http://patches.gluster.com/patch/2145 in release-2.0 (performance/stat-prefetch: allocate and init local only if cache-miss happens in lookup.)