Bug 729261
| Summary: | ext3/ext4 mbcache causes high CPU load | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Bernd Schubert <bernd.schubert> | |
| Component: | kernel | Assignee: | Eric Sandeen <esandeen> | |
| Status: | CLOSED ERRATA | QA Contact: | Eryu Guan <eguan> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 5.7 | CC: | ccui, eguan, esandeen, perfbz, rwheeler | |
| Target Milestone: | rc | |||
| Target Release: | --- | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | kernel-2.6.18-283.el5 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 731585 (view as bug list) | Environment: | ||
| Last Closed: | 2012-02-21 03:51:22 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 731585 | |||
|
Description
Bernd Schubert
2011-08-09 09:38:59 UTC
Do you see the same issue in RHEL6? Thanks! I have not tested it yet, but I guess so, as commit 3a48ee8a4ad26c3a538b6fc11a86a8f80c3dce18 is not included in 2.6.32-131.6.1.el6 yet. I will test with that kernel version as soon as possible and then will report results here. Cheers, Bernd So I can reproduce it with the 2.6.32-131.6.1.el6 kernel. Right now I'm simply using tar
(cd /mnt/tmp/fhgfs_meta && /root/tar -cf - . --xattrs --sparse) | (cd /mnt/tmp2/fhgfs_meta/ && /root/tar -xpf -)
to copy files from /mnt/tmp, which has 512 Byte ext4 inodes to /mnt/tmp, which only has 128 Byte ext4 inodes.
After about 300,000 inodes "perf top" shows 30% mb_cache_entry_insert() and 24% __mb_cache_entry_find(). While I'm writing it up here, the numbers are uncreasing and now already
------------------------------------------------------------------------------
PerfTop: 23470 irqs/sec kernel:92.3% [100000 cycles], (all, 2 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
119181.00 - 37.2% : mb_cache_entry_insert [mbcache]
81295.00 - 25.4% : __mb_cache_entry_find [mbcache]
3456.00 - 1.1% : __d_lookup
3364.00 - 1.1% : avc_has_perm_noaudit
1967.00 - 0.6% : __link_path_walk
1880.00 - 0.6% : inode_has_perm
1619.00 - 0.5% : _spin_lock
(with about 470,000 copied files). I guess it will have 80-90% mbcache once tar is almost done (we have about 16,000,000 files on that test file system).
I now tested with a recent 3.1-git kernel and with that version "perf top" does not show anything related to the mbcache. Ok, patch backports without trouble. I thought we'd have kabi issues but I guess mbcache isn't on the whitelist after all. This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. Patch(es) available in kernel-2.6.18-283.el5 You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed. Thanks for updating the kernel. We will test as soon as possible (I just need to finish some other work first). I used a modified fs_mark to create xattr(key=user.testname/value=$filename) on each file created ./fs_mark -s 0 -w 0 -p 64 -r 64 -d /mnt/ext4/test -n NUM NUM will be 1000 2000 5000 10000 20000 50000 On 2.6.18-298.el5 kernel Count Files/sec App Overhead 1000 58.9 26808 2000 58.8 57948 5000 55.2 146221 10000 53.7 294792 20000 47.0 596290 50000 57.9 1581170 On 2.6.18-274.el5 kernel Count Files/sec App Overhead 1000 57.1 47332 2000 57.2 94190 5000 55.2 256476 10000 48.3 572964 20000 42.6 1448644 50000 44.7 5406174 So -298 kernel shows a bit improvement oprofile shows more clear result On -274 kernel, 50000 file fs_mark run samples % symbol name 25141 3.5907 mb_cache_entry_get 18872 2.6953 .text.__mb_cache_entry_find 10497 1.4992 mb_cache_entry_insert On -300 kernel mb_* related functions took much less resource 978 0.1575 mb_cache_entry_get 852 0.1372 .text.__mb_cache_entry_find 515 0.0829 mb_cache_entry_insert Set to VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-0150.html |