Bug 731585

+++ This bug was initially created as a clone of Bug #729261 +++

Description of problem:

We copied small files on FhGFS file systems and noticed unusual high CPU load of the fhgfs-meta server (6 x E5520  @ 2.27GHz cores saturated). Application and kernel profiling showed that is due to the ext3/ext4 mbcache usage.

CPU: Intel Core/i7, speed 2266.81 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        app name                 symbol name
836662   48.7378  vmlinux                  mb_cache_entry_insert
368013   21.4377  vmlinux                  mb_cache_entry_free
103116    6.0068  vmlinux                  mb_cache_entry_release
93262     5.4328  vmlinux                  mb_cache_entry_get
72118     4.2011  vmlinux                  mb_cache_entry_find_first
45358     2.6422  vmlinux                  mb_cache_shrink_fn

The issue was also already exposed in the past by the Lustre file system and the Lustre file system simply added a patch to disable the mbcache:

https://bugzilla.lustre.org/show_bug.cgi?id=22771


Unlike Lustre the FhGFS file system does not need kernel patches at all and so our users depend on upstream kernels. We therefore would like to see either the patch attached to the Lustre bugzilla in RHEL kernel version or at least commit 3a48ee8a4ad26c3a538b6fc11a86a8f80c3dce18 (mbcache: Limit the maximum number of cache entries) landed in upstream linux, which should "At least partially solves https://bugzilla.lustre.org/show_bug.cgi?id=22771".


Version-Release number of selected component (if applicable):


How reproducible:

Easily with FhGFS or Lustre and with an inode size of 128 Bytes of the mete server.

Install FhGFS, the meta server should be on ext3 or ext4 and with enabled XATTR usage. Furthermore, in order to reproduce, the inode size should be limited to 128 Bytes, 


Steps to Reproduce:
1. Install FhGFS
2. meta server on ext3 or ext4 with 128 Bytes inode size, enable XATTR usage
3. Fill the filesystem with small files and introduce a high memory usage (server has at least 12 GB memory).
  
Actual results:

Top will show a high CPU usage of fhgfs-meta and Oprofile profiling will show that is due to mbcache.

Expected results:

Low CPU usage of fhgfs-meta.

Additional info:

--- Additional comment from rwheeler on 2011-08-09 05:46:48 EDT ---

Do you see the same issue in RHEL6?

Thanks!

--- Additional comment from bernd.schubert.de on 2011-08-09 07:35:29 EDT ---

I have not tested it yet, but I guess so, as commit 3a48ee8a4ad26c3a538b6fc11a86a8f80c3dce18 is not included in 2.6.32-131.6.1.el6 yet. I will test with that kernel version as soon as possible and then will report results here.

Cheers,
Bernd

--- Additional comment from bernd.schubert.de on 2011-08-09 10:56:18 EDT ---

So I can reproduce it with the 2.6.32-131.6.1.el6 kernel. Right now I'm simply using tar 

(cd /mnt/tmp/fhgfs_meta && /root/tar -cf - . --xattrs --sparse) | (cd /mnt/tmp2/fhgfs_meta/ && /root/tar -xpf -)

to copy files from /mnt/tmp, which has 512 Byte ext4 inodes to /mnt/tmp, which only has 128 Byte ext4 inodes.

After about 300,000 inodes "perf top" shows 30% mb_cache_entry_insert() and 24% __mb_cache_entry_find(). While I'm writing it up here, the numbers are uncreasing and now already

------------------------------------------------------------------------------
   PerfTop:   23470 irqs/sec  kernel:92.3% [100000 cycles],  (all, 2 CPUs)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

           119181.00 - 37.2% : mb_cache_entry_insert    [mbcache]
            81295.00 - 25.4% : __mb_cache_entry_find    [mbcache]
             3456.00 -  1.1% : __d_lookup
             3364.00 -  1.1% : avc_has_perm_noaudit
             1967.00 -  0.6% : __link_path_walk
             1880.00 -  0.6% : inode_has_perm
             1619.00 -  0.5% : _spin_lock


(with about 470,000 copied files). I guess it will have 80-90% mbcache once tar is almost done (we have about 16,000,000 files on that test file system).

--- Additional comment from bernd.schubert.de on 2011-08-09 11:45:48 EDT ---

I now tested with a recent 3.1-git kernel and with that version "perf top" does not show anything related to the mbcache.

--- Additional comment from esandeen on 2011-08-17 17:42:33 EDT ---

Ok, patch backports without trouble.  I thought we'd have kabi issues but I guess mbcache isn't on the whitelist after all.