Bug 479502 - SeLinux severely affects `install` performance
SeLinux severely affects `install` performance
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: coreutils (Show other bugs)
11
All Linux
low Severity medium
: ---
: ---
Assigned To: Ondrej Vasik
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-01-09 23:49 EST by Artem S. Tashkinov
Modified: 2009-08-12 06:18 EDT (History)
9 users (show)

See Also:
Fixed In Version: 7.2-3.fc11
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-08-11 18:39:08 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
rpmpkgs (15.35 KB, text/plain)
2009-02-10 11:38 EST, Artem S. Tashkinov
no flags Details
tar.bz2 of /var/lib/oprofile/samples/ of a "make -j2 -s install" oprofile run (346.50 KB, application/x-bzip2)
2009-06-30 04:51 EDT, Michael Stefaniuc
no flags Details
patch for wine git HEAD - please do not send to wine developers as is (1.47 KB, patch)
2009-08-06 03:24 EDT, Kamil Dudka
no flags Details | Diff

  None (edit)
Description Artem S. Tashkinov 2009-01-09 23:49:12 EST
Description of problem: install from coreutils works terribly slow with SeLinux turned on

Version-Release number of selected component (if applicable): n/a (all known versions)

How reproducible: always

Steps to Reproduce:
1. Fetch the latest wine sources ( http://sourceforge.net/project/showfiles.php?group_id=6241&package_id=77449 )
2. tar xf source.tar.bz2; cd sources; ./configure --prefix=/opt/wine && make depend && make
3. time make install
  
Actual results: over five minutes for `make install`

Expected results: fast `make install` - with SeLinux turned off this process takes 20 seconds maximum

Additional info: with selinux turned on every invocation of `install` takes a whopping 0.2 seconds!
Comment 1 Daniel Walsh 2009-01-12 15:51:18 EST
The install patch that adds matchpathcon should be removed.  This is a known problem and never made it upstream for this reason.
Comment 2 Ondrej Vasik 2009-01-13 04:41:55 EST
Not true ... matchpathcon in install was reenabled (and made it to upstream) after horrible memory leak (6M per file) was fixed - and is activated by
http://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=eba365275bdbb35df80fedcc08598ef21ace4061
by default - which was before 6.12 stable and 7.0 testing upstream release. There is no matchpathcon patch in Fedora at the moment (there was just patch fixing that leak) - we are using upstream code. Will try to check the performance impact, but do consider not just install time for SELinux disabled, but combination of install time without selinux and restorecon time after SELinux activated.
Comment 3 Ondrej Vasik 2009-01-13 04:57:14 EST
As additional note: Could be due to known issue of autotools (afaik fixed in upstream HEAD and planned for next autotools release) - this causes installing file by file - which has the biggest performance impact on install SELinux routine. Installing more files at once does improve install performance with SELinux enabled - so probably effect of SELinux will be decreased with new autotools release.
Comment 4 Ondrej Vasik 2009-01-14 06:53:58 EST
I did performance test on my machine (T60 notebook with 2G RAM) with Rawhide coreutils (7.0) - with activated matchpathcon code in install:
SELinux enforcing:
2.53user 5.52system 0:59.67elapsed 47%CPU  - so 59,7 s with 47% CPU usage
SELinux disabled - consider needed additional restorecon afterwards:
7.48user 4.97system 0:56.01elapsed 22%CPU  - so 56 s with 22% CPU usage

So elapsed time affected marginally, CPU usage was higher for SELinux enforcing, but this would change a bit with new autotools and installation of more files at once. Could you please provide:
1) Version of Coreutils package you have
2) Version of policycoreutils and selinux-policy packages
3) Filesystem where the performance impact occured

It works for me as expected - with low performance impact...
Comment 5 Ondrej Vasik 2009-02-06 07:10:37 EST
t.artem@mailcity.com : ping?
Comment 6 Artem S. Tashkinov 2009-02-06 11:44:18 EST
I'm now downloading Fedora 11 alpha to report meaningful data.
Comment 7 Artem S. Tashkinov 2009-02-09 12:38:03 EST
Well, it got better ... slightly:

Fedora 11 alpha (`make install` for wine 1.1.14 - runlevel 1, with no background tasks):

SeLinux turned on:

real    2m33.580s
user    1m51.057s
sys     0m39.034s

SeLinux turned off:

real    0m24.625s
user    0m2.453s
sys     0m19.515s

As you can see there's more than six times performance drop.
Comment 8 Ondrej Vasik 2009-02-10 04:23:55 EST
As you had clean F11 alpha, I guess my questions #1 and #2 are answered, but I still need information about filesystem.
I'm not able to reproduce the issue - on my machine it was 56s with SELinux off and 60s with SELinux on (tried only with root, will try with common user as well).

So I would like to know:
#1) architecture of your machine (i386/x86_64/ppc ? Was it real or virtual machine?)
#2) filesystem where you did the test (do you use LVM ?)

Is there something special on the machine or just default F11 alpha installation on common machine? Have you tried that with root or common user? I would like to address the issue - as 5-20% performance impact is expected, 100+% performance impact is not acceptable. Thanks in advance for those additional informations.
Comment 9 Artem S. Tashkinov 2009-02-10 11:38:47 EST
Created attachment 331435 [details]
rpmpkgs

List of installed RPMs

0. Arc = Intel x86, kernel-2.6.29-0.66.rc3.fc11.i586
1. Clean and default F11 alpha (with few missing packages)
2. AMD Athlon64 2600MHz CPU/1MB cache, 768MB RAM
3. LVM with only / (root) ext4 on top of it with default mount options, fast 7200RPM HDD. No swap, extra ext2 /boot partition.
4. Wine's source built tree, ready to 'make install' : http://rapidshare.com/files/196413900/wine-1.1.14.tar.7z.html

Results are 100% reproducible from run to run.
Comment 10 Bug Zapper 2009-06-09 06:38:42 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 11 Michael Stefaniuc 2009-06-29 18:11:32 EDT
This problem with the unbearable slow "make install" of a fully build Wine persists in F11 too. I can reproduce it on two separate machines.
Command: time make -j2 -s install

MSI Wind 100 netbook:
- selinux=0  real 0m42
- selinux=1  real 5m54

Dell Optiplex GX270 (2.8 GHz first generation P4):
- selinux=0  real 0m11
- selinux=1  real 4m39

make install is an order of magnitude slower with SELinux enabled!

I have run dstat during one the "make install" runs and on the Wind I see 95% usr CPU being burned aka the make install is CPU bound. With selinux disabled the "make install" becomes IO bound as it should be.

I have talked to Eric Paris and he wanted to see the oprofile of a "make install" run. I have done that and now I need to know in what form you guys want the oprofile (tar of /var/lib/oprofile/samples/ or any opreport output).
Comment 12 Ondrej Vasik 2009-06-30 02:08:20 EDT
Thanks for reproducing and initiative. How big is a tar archive of /var/lib/oprofile/samples? 
Additionally, I thought about the difference between common installation and my machine - and I think difference could be in used autotools (as latest autoconf(2.63)/automake(1.11) combination should afaik install files not one by one, but in groups, which should reduce performance impact on make install). Will try to reproduce it on my F-10 without latest autotools and package modifications (as I said in comment #8, I had just few percents performance impact on my machine).
Comment 13 Michael Stefaniuc 2009-06-30 04:48:25 EDT
/var/lib/oprofile/samples/ is 5.2MB big; as tar.bz2 it is only 350KB so I'll attach it.

Careful with trying to reproduce the problem on F10; F10 didn't exhibit this error. The error is specific to F11, I used myself F10 with SELinux enabled without any problems.

Also new autotools won't help at much. My Wine install tree has 1197 objects (files and directories) of which 557 are in the include/ directory which are the only ones that could be installed with a hand full of install calls. Due to a recursive makefile setup the rest of the files will be installed one by one. So for my Optiplex box even halving the time of the make install will still let the selinux=1 case be an order of magnitude too slow.
Comment 14 Michael Stefaniuc 2009-06-30 04:51:42 EDT
Created attachment 349919 [details]
tar.bz2 of /var/lib/oprofile/samples/ of a "make -j2 -s install" oprofile run
Comment 15 Ondrej Vasik 2009-06-30 07:34:12 EDT
Thanks - good to know that it was ok with F-10 (the code is same/almost same, so it's likely it's not caused by coreutils code, anyway we have to find out what's the culprit)
Comment 16 Kamil Dudka 2009-08-04 07:15:36 EDT
It's autoconf-related problem, triggered by the following commit:
http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=0647f3eb5aeaa783ce21ddee268367d2ba7248df

The function matchpathcon_init_prefix() is not found by the configure script because of lack of SELinux libraries. If I set HAVE_MATCHPATHCON_INIT_PREFIX to 1 manually it runs match faster.

The update should be available this week.
Comment 17 Kamil Dudka 2009-08-04 10:30:45 EDT
Proposed fix sent upstream:
http://article.gmane.org/gmane.comp.gnu.coreutils.bugs/17580
Comment 19 Michael Stefaniuc 2009-08-04 11:14:56 EDT
Cool, thanks! Please let me know when you have a rpm ready for testing.
Comment 20 Fedora Update System 2009-08-05 05:13:41 EDT
coreutils-7.2-3.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/coreutils-7.2-3.fc11
Comment 21 Kamil Dudka 2009-08-05 05:18:24 EDT
Michael, the RPMs (F-11 and rawhide) are ready. You can vote for the update at Bodhi. Thanks in advance for testing!
Comment 22 Michael Stefaniuc 2009-08-05 10:55:39 EDT
Cool, thanks!

I have rerun the tests on my MSI Wind 100 netbook (newer Wine version and the box is also up2date). Test command was "time make -j2 -s install" in a fully build Wine tree and I took the best of 5 consecutive runs:

"real" time     Setup
-------------------------------
 16.5s          selinux=0
361.0s          selinux=1
 49.0s          selinux=1 + coreutils-7.2-3.fc11

Quite an improvement. It's still 3 times slower but waaay better than being almost 22 times slower. I'll test it on my home desktop too and will report those numbers.
Comment 23 Kamil Dudka 2009-08-05 14:11:19 EDT
Thanks for the comprehensible statistics. It should be as fast as F-10 coreutils on the same configuration. It was a regression introduced in coreutils-7.0 which should be fixed by the update. Anyway if the install utility is still not fast enough, it's most likely question for the SELinux guys how to improve the performance.
Comment 24 Michael Stefaniuc 2009-08-05 16:55:40 EDT
I have repeated also the tests on my quad core with 8 GB at home. Test command was "time make -j4 -s install" in a fully build Wine tree and I took the best of 5 consecutive runs:

"real" time     Setup
-------------------------------
 2.5s           selinux=0
47.3s           selinux=1
 6.0s           selinux=1 + coreutils-7.2-3.fc11

"Only" 2.4 times slower on the faster machine with the regression fixed in coreutils. Again waaay better than the 19 times slower with the regression.

I had F10 only on the netbook and the 40ish - 50ish seconds install time sounds familiar. So the regression in coreutils is fixed. Many thanks for that.

In regards to "still not fast enough": Well ... I didn't add "selinux=0" back to /etc/grub.conf. But a 2.4x - 3x performance decrease in a real use case (as opposed to a synthetic micro-benchmark) is quite a hefty penalty. I would be interested in having that improved too. I figure opening a new bugzilla for that would be the right way to approach it, right?
Comment 25 Kamil Dudka 2009-08-05 17:38:50 EDT
(In reply to comment #24)

Let me note the particular performance depends also on the file system used. I tested it just now (not sure if I have "a fully build Wine tree", install ran 1103x) and it took me ~5s with -j4 and ~15s with -j1 (ext3, SELinux turned on in both cases).

Another way to speed it up is to install more files at once, then the matchpathcon_init_prefix() optimization is really taking place. You can raise the issue at the wine mailing list ;-)

If you're going to open a new bz, add me to CC. I'll provide some profiling of the install utility in my sparse time.
Comment 26 Kamil Dudka 2009-08-05 18:33:06 EDT
> Another way to speed it up is to install more files at once, then the
> matchpathcon_init_prefix() optimization is really taking place. You can raise
> the issue at the wine mailing list ;-)

I am not familiar with autotools, but I think the key problem is that wine does not use automake at all. Therefore it is not easy to switch it to more efficient way of installation. Namely include/Makefile.in where is the install utility executed in a loop for tons of .h files. Three times of install's invocation should be enough for three target directories.
Comment 27 Kamil Dudka 2009-08-06 03:24:29 EDT
Created attachment 356472 [details]
patch for wine git HEAD - please do not send to wine developers as is

You can try the attached patch for wine. It significantly improves the performance of header's installation. You need to have wine already built of course.

$ ./configure --prefix=...
$ time make -s -C include install

real    0m5.194s
user    0m4.240s
sys     0m0.981s

$ patch -p1 < fast-incdir-install.patch
$ ./configure --prefix=...
$ time make -s -C include install

real    0m1.607s
user    0m0.670s
sys     0m0.973s

I think this is the way to go.
Comment 28 Michael Stefaniuc 2009-08-06 18:13:45 EDT
Sorry but I beg to differ:
a.) Wine's method of installing the header files might not be optimal but the real problem is in libselinux: determining what context a file should have seems to be very expensive. Pushing the workaround (the cache) for that into the application (install) is a questionable approach at best. But blaming the user (Wine) of that application for those shortcomings is unwarranted.

b.) Wine developers are not concerned about the "make install" speed at all. Most of them use "in source tree builds" and Wine can be run from the source dir without installing. It just happens that for the work I'm doing (mostly janitorial work) using separate build trees (multiple build trees per source tree) is the way to go. Thus I have to use "make install" if I want to run one of the build trees in a reliable way. 

Anyway as I install Wine as a user in a directory on a separate junk partition and SELinux doesn't know anyway anything about that path I found my workaround for the issue:
  make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install
That gives the exact same result but has a negligible impact; that finishes in 2.6s versus the 2.5s of the selinux=0 case on my desktop box. But most the times 6s for the the "make -j4 -s install" are fine too. Of course it's is a different story on my netbook where the absolute penalty is a lot higher.


But please don't get me wrong, if you find a clean (*) and portable (**) way of installing the header files which also speeds up "make install" in the non-SELinux (***) case a patch would be much appreciated. But that would ameliorate the impact of the problem with libselinux only for a short time as the addition of Win64 will immediately add over 500 binary files (Win32 and Win64 libs need to be both installed in parallel). Of course new modules (DLLs and programs) keep being added as well.

  (*) Using automake is not a clean solution as it would introduce more problems than it solves especially in the light that most Makefile.in are already generated by tools/make_makefiles from the files git knows about.

 (**) Needs to work on at least Linux, FreeBSD, MacOSX, Solaris 9/10; bonus points for NetBSD and OpenBSD. Native make aka no GNU make dependency. Native install too.

(***) Most Wine developers use distributions that don't have SELinux or don't enable it by default. The rest that do use Fedora I would bet have SELinux disabled. Thus a patch that is only a workaround for a problem in SELinux would have a hard time to be accepted.
Comment 29 Kamil Dudka 2009-08-06 18:50:05 EDT
(In reply to comment #28)

> a.) Wine's method of installing the header files might not be optimal but the
> real problem is in libselinux: determining what context a file should have
> seems to be very expensive. Pushing the workaround (the cache) for that into
> the application (install) is a questionable approach at best. But blaming the
> user (Wine) of that application for those shortcomings is unwarranted.

That's more likely question for Dan Walsh as (probably) original author of the optimization:
http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=56e3106e934796f993decd08b3c4224d3830209a

> Anyway as I install Wine as a user in a directory on a separate junk partition
> and SELinux doesn't know anyway anything about that path I found my workaround
> for the issue:
>   make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install

Looks good. It gives me c.c.a. 5x higher performance. What about mentioning it in the info documentation?
Comment 30 Fedora Update System 2009-08-07 01:03:05 EDT
coreutils-7.2-3.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update coreutils'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-8337
Comment 31 Ondrej Vasik 2009-08-07 02:30:34 EDT
(In reply to comment #29)
> (In reply to comment #28)
> > Anyway as I install Wine as a user in a directory on a separate junk partition
> > and SELinux doesn't know anyway anything about that path I found my workaround
> > for the issue:
> >   make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install
> 
> Looks good. It gives me c.c.a. 5x higher performance. What about mentioning it
> in the info documentation?  

It does make sense. Anyway - I would suggest using "-Z <default context of destination dir>" instead of unconfined_u:object_r:default_t:s0 context and calling just one restorecon on destination dir after finish. That will help even more (correct context, just one additional SELinux tool call) and should be easily scriptable - as the most expensive thing is generating of the SELinux context submap for every file as is done in case of make install.
Comment 32 Kamil Dudka 2009-08-07 03:14:01 EDT
(In reply to comment #29)
> That's more likely question for Dan Walsh as (probably) original author of the
> optimization:
> http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=56e3106e934796f993decd08b3c4224d3830209a

And this is the relevant thread at bug-coreutils:
http://lists.gnu.org/archive/html/bug-coreutils/2007-11/msg00069.html

I think the temporal cache at application level (though in a dynamic library) can't be sufficient in all cases. We need cache with longer time of life. IMHO the cache should take place at the file system level (as well as the cache for inodes and dentrys). It looks to me like a design flaw in the SELinux implementation, but maybe I am wrong.
Comment 33 Michael Stefaniuc 2009-08-07 03:50:18 EDT
In regards to comment #31:
I agree, "unconfined_u:object_r:default_t:s0" happened to be the context of my dest dir. But if one runs restorecon anyway afterward it really doesn't matters what the initial context is as long as one doesn't get a permission denied while setting it.

A far better approach would be to add a "--no-selinux" command line switch to install which doesn't sets any SELinux context and lets the normal context inheritance take place (from parent directory or existing file that gets overwritten). A short form of the option would be nice and probably a better name for the long option. Would you guys accept a feature request for that? I would create one in bugzilla if the chances are high enough.
Comment 34 Fedora Update System 2009-08-11 18:39:02 EDT
coreutils-7.2-3.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 35 Artem S. Tashkinov 2009-08-12 03:30:47 EDT
(In reply to comment #24)

> "Only" 2.4 times slower on the faster machine with the regression fixed in
> coreutils. Again waaay better than the 19 times slower with the regression.
> 

Is there a way to decrease SeLinux penalty even further?
Comment 36 Ondrej Vasik 2009-08-12 06:18:11 EDT
There are some ways how to decrease SELinux penalty - already mentioned there. So as summary, here are options:
1) Call install utility with more arguments - to install more files at once - the most expensive generating of SELinux submap is performed just once per `install` execution. That's the way how it is done with latest autotools...
2) Call install utility with -Z option - which turns off automated SELinux context setting. After installation just run restorecon on destination directory - should be much faster as the submap is created only once.

I doubt there is anything to improve performance in coreutils, maybe something could be done on side libselinux. But I wouldn't expect much improvement as the SELinux policies still grow... Maybe having some temporary cache could help - having last submap stored in cache. 

Anyway, using option #2 should have good result even without libselinux improvement.

Note You need to log in before you can comment on or make changes to this bug.