Bug 479502

The install patch that adds matchpathcon should be removed.  This is a known problem and never made it upstream for this reason.

Not true ... matchpathcon in install was reenabled (and made it to upstream) after horrible memory leak (6M per file) was fixed - and is activated by
http://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=eba365275bdbb35df80fedcc08598ef21ace4061
by default - which was before 6.12 stable and 7.0 testing upstream release. There is no matchpathcon patch in Fedora at the moment (there was just patch fixing that leak) - we are using upstream code. Will try to check the performance impact, but do consider not just install time for SELinux disabled, but combination of install time without selinux and restorecon time after SELinux activated.

As additional note: Could be due to known issue of autotools (afaik fixed in upstream HEAD and planned for next autotools release) - this causes installing file by file - which has the biggest performance impact on install SELinux routine. Installing more files at once does improve install performance with SELinux enabled - so probably effect of SELinux will be decreased with new autotools release.

I did performance test on my machine (T60 notebook with 2G RAM) with Rawhide coreutils (7.0) - with activated matchpathcon code in install:
SELinux enforcing:
2.53user 5.52system 0:59.67elapsed 47%CPU  - so 59,7 s with 47% CPU usage
SELinux disabled - consider needed additional restorecon afterwards:
7.48user 4.97system 0:56.01elapsed 22%CPU  - so 56 s with 22% CPU usage

So elapsed time affected marginally, CPU usage was higher for SELinux enforcing, but this would change a bit with new autotools and installation of more files at once. Could you please provide:
1) Version of Coreutils package you have
2) Version of policycoreutils and selinux-policy packages
3) Filesystem where the performance impact occured

It works for me as expected - with low performance impact...

t.artem : ping?

I'm now downloading Fedora 11 alpha to report meaningful data.

Well, it got better ... slightly:

Fedora 11 alpha (`make install` for wine 1.1.14 - runlevel 1, with no background tasks):

SeLinux turned on:

real    2m33.580s
user    1m51.057s
sys     0m39.034s

SeLinux turned off:

real    0m24.625s
user    0m2.453s
sys     0m19.515s

As you can see there's more than six times performance drop.

As you had clean F11 alpha, I guess my questions #1 and #2 are answered, but I still need information about filesystem.
I'm not able to reproduce the issue - on my machine it was 56s with SELinux off and 60s with SELinux on (tried only with root, will try with common user as well).

So I would like to know:
#1) architecture of your machine (i386/x86_64/ppc ? Was it real or virtual machine?)
#2) filesystem where you did the test (do you use LVM ?)

Is there something special on the machine or just default F11 alpha installation on common machine? Have you tried that with root or common user? I would like to address the issue - as 5-20% performance impact is expected, 100+% performance impact is not acceptable. Thanks in advance for those additional informations.

Created attachment 331435 [details]
rpmpkgs

List of installed RPMs

0. Arc = Intel x86, kernel-2.6.29-0.66.rc3.fc11.i586
1. Clean and default F11 alpha (with few missing packages)
2. AMD Athlon64 2600MHz CPU/1MB cache, 768MB RAM
3. LVM with only / (root) ext4 on top of it with default mount options, fast 7200RPM HDD. No swap, extra ext2 /boot partition.
4. Wine's source built tree, ready to 'make install' : http://rapidshare.com/files/196413900/wine-1.1.14.tar.7z.html

Results are 100% reproducible from run to run.

This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

This problem with the unbearable slow "make install" of a fully build Wine persists in F11 too. I can reproduce it on two separate machines.
Command: time make -j2 -s install

MSI Wind 100 netbook:
- selinux=0  real 0m42
- selinux=1  real 5m54

Dell Optiplex GX270 (2.8 GHz first generation P4):
- selinux=0  real 0m11
- selinux=1  real 4m39

make install is an order of magnitude slower with SELinux enabled!

I have run dstat during one the "make install" runs and on the Wind I see 95% usr CPU being burned aka the make install is CPU bound. With selinux disabled the "make install" becomes IO bound as it should be.

I have talked to Eric Paris and he wanted to see the oprofile of a "make install" run. I have done that and now I need to know in what form you guys want the oprofile (tar of /var/lib/oprofile/samples/ or any opreport output).

Thanks for reproducing and initiative. How big is a tar archive of /var/lib/oprofile/samples? 
Additionally, I thought about the difference between common installation and my machine - and I think difference could be in used autotools (as latest autoconf(2.63)/automake(1.11) combination should afaik install files not one by one, but in groups, which should reduce performance impact on make install). Will try to reproduce it on my F-10 without latest autotools and package modifications (as I said in comment #8, I had just few percents performance impact on my machine).

/var/lib/oprofile/samples/ is 5.2MB big; as tar.bz2 it is only 350KB so I'll attach it.

Careful with trying to reproduce the problem on F10; F10 didn't exhibit this error. The error is specific to F11, I used myself F10 with SELinux enabled without any problems.

Also new autotools won't help at much. My Wine install tree has 1197 objects (files and directories) of which 557 are in the include/ directory which are the only ones that could be installed with a hand full of install calls. Due to a recursive makefile setup the rest of the files will be installed one by one. So for my Optiplex box even halving the time of the make install will still let the selinux=1 case be an order of magnitude too slow.

Created attachment 349919 [details]
tar.bz2 of /var/lib/oprofile/samples/ of a "make -j2 -s install" oprofile run

Thanks - good to know that it was ok with F-10 (the code is same/almost same, so it's likely it's not caused by coreutils code, anyway we have to find out what's the culprit)

It's autoconf-related problem, triggered by the following commit:
http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=0647f3eb5aeaa783ce21ddee268367d2ba7248df

The function matchpathcon_init_prefix() is not found by the configure script because of lack of SELinux libraries. If I set HAVE_MATCHPATHCON_INIT_PREFIX to 1 manually it runs match faster.

The update should be available this week.

Proposed fix sent upstream:
http://article.gmane.org/gmane.comp.gnu.coreutils.bugs/17580

Committed upstream:
http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=1762092901adf040445d59931d14837ef7f2ebe1

Cool, thanks! Please let me know when you have a rpm ready for testing.

coreutils-7.2-3.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/coreutils-7.2-3.fc11

Michael, the RPMs (F-11 and rawhide) are ready. You can vote for the update at Bodhi. Thanks in advance for testing!

Cool, thanks!

I have rerun the tests on my MSI Wind 100 netbook (newer Wine version and the box is also up2date). Test command was "time make -j2 -s install" in a fully build Wine tree and I took the best of 5 consecutive runs:

"real" time     Setup
-------------------------------
 16.5s          selinux=0
361.0s          selinux=1
 49.0s          selinux=1 + coreutils-7.2-3.fc11

Quite an improvement. It's still 3 times slower but waaay better than being almost 22 times slower. I'll test it on my home desktop too and will report those numbers.

Thanks for the comprehensible statistics. It should be as fast as F-10 coreutils on the same configuration. It was a regression introduced in coreutils-7.0 which should be fixed by the update. Anyway if the install utility is still not fast enough, it's most likely question for the SELinux guys how to improve the performance.

I have repeated also the tests on my quad core with 8 GB at home. Test command was "time make -j4 -s install" in a fully build Wine tree and I took the best of 5 consecutive runs:

"real" time     Setup
-------------------------------
 2.5s           selinux=0
47.3s           selinux=1
 6.0s           selinux=1 + coreutils-7.2-3.fc11

"Only" 2.4 times slower on the faster machine with the regression fixed in coreutils. Again waaay better than the 19 times slower with the regression.

I had F10 only on the netbook and the 40ish - 50ish seconds install time sounds familiar. So the regression in coreutils is fixed. Many thanks for that.

In regards to "still not fast enough": Well ... I didn't add "selinux=0" back to /etc/grub.conf. But a 2.4x - 3x performance decrease in a real use case (as opposed to a synthetic micro-benchmark) is quite a hefty penalty. I would be interested in having that improved too. I figure opening a new bugzilla for that would be the right way to approach it, right?

(In reply to comment #24)

Let me note the particular performance depends also on the file system used. I tested it just now (not sure if I have "a fully build Wine tree", install ran 1103x) and it took me ~5s with -j4 and ~15s with -j1 (ext3, SELinux turned on in both cases).

Another way to speed it up is to install more files at once, then the matchpathcon_init_prefix() optimization is really taking place. You can raise the issue at the wine mailing list ;-)

If you're going to open a new bz, add me to CC. I'll provide some profiling of the install utility in my sparse time.

> Another way to speed it up is to install more files at once, then the
> matchpathcon_init_prefix() optimization is really taking place. You can raise
> the issue at the wine mailing list ;-)

I am not familiar with autotools, but I think the key problem is that wine does not use automake at all. Therefore it is not easy to switch it to more efficient way of installation. Namely include/Makefile.in where is the install utility executed in a loop for tons of .h files. Three times of install's invocation should be enough for three target directories.

Created attachment 356472 [details]
patch for wine git HEAD - please do not send to wine developers as is

You can try the attached patch for wine. It significantly improves the performance of header's installation. You need to have wine already built of course.

$ ./configure --prefix=...
$ time make -s -C include install

real    0m5.194s
user    0m4.240s
sys     0m0.981s

$ patch -p1 < fast-incdir-install.patch
$ ./configure --prefix=...
$ time make -s -C include install

real    0m1.607s
user    0m0.670s
sys     0m0.973s

I think this is the way to go.

Sorry but I beg to differ:
a.) Wine's method of installing the header files might not be optimal but the real problem is in libselinux: determining what context a file should have seems to be very expensive. Pushing the workaround (the cache) for that into the application (install) is a questionable approach at best. But blaming the user (Wine) of that application for those shortcomings is unwarranted.

b.) Wine developers are not concerned about the "make install" speed at all. Most of them use "in source tree builds" and Wine can be run from the source dir without installing. It just happens that for the work I'm doing (mostly janitorial work) using separate build trees (multiple build trees per source tree) is the way to go. Thus I have to use "make install" if I want to run one of the build trees in a reliable way. 

Anyway as I install Wine as a user in a directory on a separate junk partition and SELinux doesn't know anyway anything about that path I found my workaround for the issue:
  make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install
That gives the exact same result but has a negligible impact; that finishes in 2.6s versus the 2.5s of the selinux=0 case on my desktop box. But most the times 6s for the the "make -j4 -s install" are fine too. Of course it's is a different story on my netbook where the absolute penalty is a lot higher.


But please don't get me wrong, if you find a clean (*) and portable (**) way of installing the header files which also speeds up "make install" in the non-SELinux (***) case a patch would be much appreciated. But that would ameliorate the impact of the problem with libselinux only for a short time as the addition of Win64 will immediately add over 500 binary files (Win32 and Win64 libs need to be both installed in parallel). Of course new modules (DLLs and programs) keep being added as well.

  (*) Using automake is not a clean solution as it would introduce more problems than it solves especially in the light that most Makefile.in are already generated by tools/make_makefiles from the files git knows about.

 (**) Needs to work on at least Linux, FreeBSD, MacOSX, Solaris 9/10; bonus points for NetBSD and OpenBSD. Native make aka no GNU make dependency. Native install too.

(***) Most Wine developers use distributions that don't have SELinux or don't enable it by default. The rest that do use Fedora I would bet have SELinux disabled. Thus a patch that is only a workaround for a problem in SELinux would have a hard time to be accepted.

(In reply to comment #28)

> a.) Wine's method of installing the header files might not be optimal but the
> real problem is in libselinux: determining what context a file should have
> seems to be very expensive. Pushing the workaround (the cache) for that into
> the application (install) is a questionable approach at best. But blaming the
> user (Wine) of that application for those shortcomings is unwarranted.

That's more likely question for Dan Walsh as (probably) original author of the optimization:
http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=56e3106e934796f993decd08b3c4224d3830209a

> Anyway as I install Wine as a user in a directory on a separate junk partition
> and SELinux doesn't know anyway anything about that path I found my workaround
> for the issue:
>   make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install

Looks good. It gives me c.c.a. 5x higher performance. What about mentioning it in the info documentation?

coreutils-7.2-3.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update coreutils'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-8337

(In reply to comment #29)
> (In reply to comment #28)
> > Anyway as I install Wine as a user in a directory on a separate junk partition
> > and SELinux doesn't know anyway anything about that path I found my workaround
> > for the issue:
> >   make INSTALL_FLAGS="-Z unconfined_u:object_r:default_t:s0" -j4 -s install
> 
> Looks good. It gives me c.c.a. 5x higher performance. What about mentioning it
> in the info documentation?  

It does make sense. Anyway - I would suggest using "-Z <default context of destination dir>" instead of unconfined_u:object_r:default_t:s0 context and calling just one restorecon on destination dir after finish. That will help even more (correct context, just one additional SELinux tool call) and should be easily scriptable - as the most expensive thing is generating of the SELinux context submap for every file as is done in case of make install.

(In reply to comment #29)
> That's more likely question for Dan Walsh as (probably) original author of the
> optimization:
> http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=56e3106e934796f993decd08b3c4224d3830209a

And this is the relevant thread at bug-coreutils:
http://lists.gnu.org/archive/html/bug-coreutils/2007-11/msg00069.html

I think the temporal cache at application level (though in a dynamic library) can't be sufficient in all cases. We need cache with longer time of life. IMHO the cache should take place at the file system level (as well as the cache for inodes and dentrys). It looks to me like a design flaw in the SELinux implementation, but maybe I am wrong.

In regards to comment #31:
I agree, "unconfined_u:object_r:default_t:s0" happened to be the context of my dest dir. But if one runs restorecon anyway afterward it really doesn't matters what the initial context is as long as one doesn't get a permission denied while setting it.

A far better approach would be to add a "--no-selinux" command line switch to install which doesn't sets any SELinux context and lets the normal context inheritance take place (from parent directory or existing file that gets overwritten). A short form of the option would be nice and probably a better name for the long option. Would you guys accept a feature request for that? I would create one in bugzilla if the chances are high enough.

coreutils-7.2-3.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

(In reply to comment #24)

> "Only" 2.4 times slower on the faster machine with the regression fixed in
> coreutils. Again waaay better than the 19 times slower with the regression.
> 

Is there a way to decrease SeLinux penalty even further?

There are some ways how to decrease SELinux penalty - already mentioned there. So as summary, here are options:
1) Call install utility with more arguments - to install more files at once - the most expensive generating of SELinux submap is performed just once per `install` execution. That's the way how it is done with latest autotools...
2) Call install utility with -Z option - which turns off automated SELinux context setting. After installation just run restorecon on destination directory - should be much faster as the submap is created only once.

I doubt there is anything to improve performance in coreutils, maybe something could be done on side libselinux. But I wouldn't expect much improvement as the SELinux policies still grow... Maybe having some temporary cache could help - having last submap stored in cache. 

Anyway, using option #2 should have good result even without libselinux improvement.