Red Hat Bugzilla – Bug 479502
SeLinux severely affects `install` performance
Last modified: 2009-08-12 06:18:11 EDT
Description of problem: install from coreutils works terribly slow with SeLinux turned on
Version-Release number of selected component (if applicable): n/a (all known versions)
How reproducible: always
Steps to Reproduce:
1. Fetch the latest wine sources ( http://sourceforge.net/project/showfiles.php?group_id=6241&package_id=77449 )
2. tar xf source.tar.bz2; cd sources; ./configure --prefix=/opt/wine && make depend && make
3. time make install
Actual results: over five minutes for `make install`
Expected results: fast `make install` - with SeLinux turned off this process takes 20 seconds maximum
Additional info: with selinux turned on every invocation of `install` takes a whopping 0.2 seconds!
The install patch that adds matchpathcon should be removed. This is a known problem and never made it upstream for this reason.
Not true ... matchpathcon in install was reenabled (and made it to upstream) after horrible memory leak (6M per file) was fixed - and is activated by
by default - which was before 6.12 stable and 7.0 testing upstream release. There is no matchpathcon patch in Fedora at the moment (there was just patch fixing that leak) - we are using upstream code. Will try to check the performance impact, but do consider not just install time for SELinux disabled, but combination of install time without selinux and restorecon time after SELinux activated.
As additional note: Could be due to known issue of autotools (afaik fixed in upstream HEAD and planned for next autotools release) - this causes installing file by file - which has the biggest performance impact on install SELinux routine. Installing more files at once does improve install performance with SELinux enabled - so probably effect of SELinux will be decreased with new autotools release.
I did performance test on my machine (T60 notebook with 2G RAM) with Rawhide coreutils (7.0) - with activated matchpathcon code in install:
2.53user 5.52system 0:59.67elapsed 47%CPU - so 59,7 s with 47% CPU usage
SELinux disabled - consider needed additional restorecon afterwards:
7.48user 4.97system 0:56.01elapsed 22%CPU - so 56 s with 22% CPU usage
So elapsed time affected marginally, CPU usage was higher for SELinux enforcing, but this would change a bit with new autotools and installation of more files at once. Could you please provide:
1) Version of Coreutils package you have
2) Version of policycoreutils and selinux-policy packages
3) Filesystem where the performance impact occured
It works for me as expected - with low performance impact...
email@example.com : ping?
I'm now downloading Fedora 11 alpha to report meaningful data.
Well, it got better ... slightly:
Fedora 11 alpha (`make install` for wine 1.1.14 - runlevel 1, with no background tasks):
SeLinux turned on:
SeLinux turned off:
As you can see there's more than six times performance drop.
As you had clean F11 alpha, I guess my questions #1 and #2 are answered, but I still need information about filesystem.
I'm not able to reproduce the issue - on my machine it was 56s with SELinux off and 60s with SELinux on (tried only with root, will try with common user as well).
So I would like to know:
#1) architecture of your machine (i386/x86_64/ppc ? Was it real or virtual machine?)
#2) filesystem where you did the test (do you use LVM ?)
Is there something special on the machine or just default F11 alpha installation on common machine? Have you tried that with root or common user? I would like to address the issue - as 5-20% performance impact is expected, 100+% performance impact is not acceptable. Thanks in advance for those additional informations.
Created attachment 331435 [details]
List of installed RPMs
0. Arc = Intel x86, kernel-2.6.29-0.66.rc3.fc11.i586
1. Clean and default F11 alpha (with few missing packages)
2. AMD Athlon64 2600MHz CPU/1MB cache, 768MB RAM
3. LVM with only / (root) ext4 on top of it with default mount options, fast 7200RPM HDD. No swap, extra ext2 /boot partition.
4. Wine's source built tree, ready to 'make install' : http://rapidshare.com/files/196413900/wine-1.1.14.tar.7z.html
Results are 100% reproducible from run to run.
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.
More information and reason for this action is here:
This problem with the unbearable slow "make install" of a fully build Wine persists in F11 too. I can reproduce it on two separate machines.
Command: time make -j2 -s install
MSI Wind 100 netbook:
- selinux=0 real 0m42
- selinux=1 real 5m54
Dell Optiplex GX270 (2.8 GHz first generation P4):
- selinux=0 real 0m11
- selinux=1 real 4m39
make install is an order of magnitude slower with SELinux enabled!
I have run dstat during one the "make install" runs and on the Wind I see 95% usr CPU being burned aka the make install is CPU bound. With selinux disabled the "make install" becomes IO bound as it should be.
I have talked to Eric Paris and he wanted to see the oprofile of a "make install" run. I have done that and now I need to know in what form you guys want the oprofile (tar of /var/lib/oprofile/samples/ or any opreport output).
Thanks for reproducing and initiative. How big is a tar archive of /var/lib/oprofile/samples?
Additionally, I thought about the difference between common installation and my machine - and I think difference could be in used autotools (as latest autoconf(2.63)/automake(1.11) combination should afaik install files not one by one, but in groups, which should reduce performance impact on make install). Will try to reproduce it on my F-10 without latest autotools and package modifications (as I said in comment #8, I had just few percents performance impact on my machine).
/var/lib/oprofile/samples/ is 5.2MB big; as tar.bz2 it is only 350KB so I'll attach it.
Careful with trying to reproduce the problem on F10; F10 didn't exhibit this error. The error is specific to F11, I used myself F10 with SELinux enabled without any problems.
Also new autotools won't help at much. My Wine install tree has 1197 objects (files and directories) of which 557 are in the include/ directory which are the only ones that could be installed with a hand full of install calls. Due to a recursive makefile setup the rest of the files will be installed one by one. So for my Optiplex box even halving the time of the make install will still let the selinux=1 case be an order of magnitude too slow.
Created attachment 349919 [details]
tar.bz2 of /var/lib/oprofile/samples/ of a "make -j2 -s install" oprofile run
Thanks - good to know that it was ok with F-10 (the code is same/almost same, so it's likely it's not caused by coreutils code, anyway we have to find out what's the culprit)
It's autoconf-related problem, triggered by the following commit:
The function matchpathcon_init_prefix() is not found by the configure script because of lack of SELinux libraries. If I set HAVE_MATCHPATHCON_INIT_PREFIX to 1 manually it runs match faster.
The update should be available this week.
Proposed fix sent upstream:
Cool, thanks! Please let me know when you have a rpm ready for testing.
coreutils-7.2-3.fc11 has been submitted as an update for Fedora 11.
Michael, the RPMs (F-11 and rawhide) are ready. You can vote for the update at Bodhi. Thanks in advance for testing!
I have rerun the tests on my MSI Wind 100 netbook (newer Wine version and the box is also up2date). Test command was "time make -j2 -s install" in a fully build Wine tree and I took the best of 5 consecutive runs:
"real" time Setup
49.0s selinux=1 + coreutils-7.2-3.fc11
Quite an improvement. It's still 3 times slower but waaay better than being almost 22 times slower. I'll test it on my home desktop too and will report those numbers.
Thanks for the comprehensible statistics. It should be as fast as F-10 coreutils on the same configuration. It was a regression introduced in coreutils-7.0 which should be fixed by the update. Anyway if the install utility is still not fast enough, it's most likely question for the SELinux guys how to improve the performance.
I have repeated also the tests on my quad core with 8 GB at home. Test command was "time make -j4 -s install" in a fully build Wine tree and I took the best of 5 consecutive runs:
"real" time Setup
6.0s selinux=1 + coreutils-7.2-3.fc11
"Only" 2.4 times slower on the faster machine with the regression fixed in coreutils. Again waaay better than the 19 times slower with the regression.
I had F10 only on the netbook and the 40ish - 50ish seconds install time sounds familiar. So the regression in coreutils is fixed. Many thanks for that.
In regards to "still not fast enough": Well ... I didn't add "selinux=0" back to /etc/grub.conf. But a 2.4x - 3x performance decrease in a real use case (as opposed to a synthetic micro-benchmark) is quite a hefty penalty. I would be interested in having that improved too. I figure opening a new bugzilla for that would be the right way to approach it, right?
(In reply to comment #24)
Let me note the particular performance depends also on the file system used. I tested it just now (not sure if I have "a fully build Wine tree", install ran 1103x) and it took me ~5s with -j4 and ~15s with -j1 (ext3, SELinux turned on in both cases).
Another way to speed it up is to install more files at once, then the matchpathcon_init_prefix() optimization is really taking place. You can raise the issue at the wine mailing list ;-)
If you're going to open a new bz, add me to CC. I'll provide some profiling of the install utility in my sparse time.
> Another way to speed it up is to install more files at once, then the
> matchpathcon_init_prefix() optimization is really taking place. You can raise
> the issue at the wine mailing list ;-)
I am not familiar with autotools, but I think the key problem is that wine does not use automake at all. Therefore it is not easy to switch it to more efficient way of installation. Namely include/Makefile.in where is the install utility executed in a loop for tons of .h files. Three times of install's invocation should be enough for three target directories.
Created attachment 356472 [details]
patch for wine git HEAD - please do not send to wine developers as is
You can try the attached patch for wine. It significantly improves the performance of header's installation. You need to have wine already built of course.
$ ./configure --prefix=...
$ time make -s -C include install
$ patch -p1 < fast-incdir-install.patch
$ ./configure --prefix=...
$ time make -s -C include install
I think this is the way to go.
Sorry but I beg to differ:
a.) Wine's method of installing the header files might not be optimal but the real problem is in libselinux: determining what context a file should have seems to be very expensive. Pushing the workaround (the cache) for that into the application (install) is a questionable approach at best. But blaming the user (Wine) of that application for those shortcomings is unwarranted.
b.) Wine developers are not concerned about the "make install" speed at all. Most of them use "in source tree builds" and Wine can be run from the source dir without installing. It just happens that for the work I'm doing (mostly janitorial work) using separate build trees (multiple build trees per source tree) is the way to go. Thus I have to use "make install" if I want to run one of the build trees in a reliable way.
Anyway as I install Wine as a user in a directory on a separate junk partition and SELinux doesn't know anyway anything about that path I found my workaround for the issue:
make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install
That gives the exact same result but has a negligible impact; that finishes in 2.6s versus the 2.5s of the selinux=0 case on my desktop box. But most the times 6s for the the "make -j4 -s install" are fine too. Of course it's is a different story on my netbook where the absolute penalty is a lot higher.
But please don't get me wrong, if you find a clean (*) and portable (**) way of installing the header files which also speeds up "make install" in the non-SELinux (***) case a patch would be much appreciated. But that would ameliorate the impact of the problem with libselinux only for a short time as the addition of Win64 will immediately add over 500 binary files (Win32 and Win64 libs need to be both installed in parallel). Of course new modules (DLLs and programs) keep being added as well.
(*) Using automake is not a clean solution as it would introduce more problems than it solves especially in the light that most Makefile.in are already generated by tools/make_makefiles from the files git knows about.
(**) Needs to work on at least Linux, FreeBSD, MacOSX, Solaris 9/10; bonus points for NetBSD and OpenBSD. Native make aka no GNU make dependency. Native install too.
(***) Most Wine developers use distributions that don't have SELinux or don't enable it by default. The rest that do use Fedora I would bet have SELinux disabled. Thus a patch that is only a workaround for a problem in SELinux would have a hard time to be accepted.
(In reply to comment #28)
> a.) Wine's method of installing the header files might not be optimal but the
> real problem is in libselinux: determining what context a file should have
> seems to be very expensive. Pushing the workaround (the cache) for that into
> the application (install) is a questionable approach at best. But blaming the
> user (Wine) of that application for those shortcomings is unwarranted.
That's more likely question for Dan Walsh as (probably) original author of the optimization:
> Anyway as I install Wine as a user in a directory on a separate junk partition
> and SELinux doesn't know anyway anything about that path I found my workaround
> for the issue:
> make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install
Looks good. It gives me c.c.a. 5x higher performance. What about mentioning it in the info documentation?
coreutils-7.2-3.fc11 has been pushed to the Fedora 11 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
su -c 'yum --enablerepo=updates-testing update coreutils'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-8337
(In reply to comment #29)
> (In reply to comment #28)
> > Anyway as I install Wine as a user in a directory on a separate junk partition
> > and SELinux doesn't know anyway anything about that path I found my workaround
> > for the issue:
> > make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install
> Looks good. It gives me c.c.a. 5x higher performance. What about mentioning it
> in the info documentation?
It does make sense. Anyway - I would suggest using "-Z <default context of destination dir>" instead of unconfined_u:object_r:default_t:s0 context and calling just one restorecon on destination dir after finish. That will help even more (correct context, just one additional SELinux tool call) and should be easily scriptable - as the most expensive thing is generating of the SELinux context submap for every file as is done in case of make install.
(In reply to comment #29)
> That's more likely question for Dan Walsh as (probably) original author of the
And this is the relevant thread at bug-coreutils:
I think the temporal cache at application level (though in a dynamic library) can't be sufficient in all cases. We need cache with longer time of life. IMHO the cache should take place at the file system level (as well as the cache for inodes and dentrys). It looks to me like a design flaw in the SELinux implementation, but maybe I am wrong.
In regards to comment #31:
I agree, "unconfined_u:object_r:default_t:s0" happened to be the context of my dest dir. But if one runs restorecon anyway afterward it really doesn't matters what the initial context is as long as one doesn't get a permission denied while setting it.
A far better approach would be to add a "--no-selinux" command line switch to install which doesn't sets any SELinux context and lets the normal context inheritance take place (from parent directory or existing file that gets overwritten). A short form of the option would be nice and probably a better name for the long option. Would you guys accept a feature request for that? I would create one in bugzilla if the chances are high enough.
coreutils-7.2-3.fc11 has been pushed to the Fedora 11 stable repository. If problems still persist, please make note of it in this bug report.
(In reply to comment #24)
> "Only" 2.4 times slower on the faster machine with the regression fixed in
> coreutils. Again waaay better than the 19 times slower with the regression.
Is there a way to decrease SeLinux penalty even further?
There are some ways how to decrease SELinux penalty - already mentioned there. So as summary, here are options:
1) Call install utility with more arguments - to install more files at once - the most expensive generating of SELinux submap is performed just once per `install` execution. That's the way how it is done with latest autotools...
2) Call install utility with -Z option - which turns off automated SELinux context setting. After installation just run restorecon on destination directory - should be much faster as the submap is created only once.
I doubt there is anything to improve performance in coreutils, maybe something could be done on side libselinux. But I wouldn't expect much improvement as the SELinux policies still grow... Maybe having some temporary cache could help - having last submap stored in cache.
Anyway, using option #2 should have good result even without libselinux improvement.