Bug 981918 - Inexplicable differences between "identical" installations (security issue?)
Inexplicable differences between "identical" installations (security issue?)
Product: Fedora
Classification: Fedora
Component: distribution (Show other bugs)
i686 Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Bill Nottingham
Bill Nottingham
Depends On:
  Show dependency treegraph
Reported: 2013-07-06 21:46 EDT by Steve
Modified: 2014-03-16 23:33 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-02-05 17:04:47 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Steve 2013-07-06 21:46:30 EDT
After rkhunter kept reporting weird changes to my system, I decided to do my best to investigate whether my system had been compromised.

I did this by installing a clean copy of the same version of Fedora on an encrypted partition, and putting the boot partition on a thumb drive that (since being repartitioned/reformatted) has only been connected to my machine while this "clean" installation was running, or when running rescue mode straight from the install DVD.

The "clean" system has only accessed the Internet through wget (to grab the rpmfusion release RPMs) and through package management (i.e. yum and fedup).  I've never run a web browser on it, or X Windows for that matter; immediately after installation, I changed the default runlevel to multi-user (i.e. 3).  I don't know how to be any more careful; if anyone has some advice, I'd like to hear it.

After making sure all the packages on my in-use system had been installed, I booted into the clean system, mounted the in-use system, and did a binary diff (i.e. "diff -raq") between the two, and found several inexplicable differences.  The really weird thing is, both systems pass "rpm -V" on the affected packages.

There are more differences than these, but these include the scariest ones, e.g. java and sendmail.  (The in-use system is mounted as "/dirtyroot".)

-rw-r--r--. 1 root root 681 Jun 27 20:51 /dirtyroot/usr/lib/gtk-3.0/3.0.0/immodules.cache
-rw-r--r--. 1 root root 672 Jul  5 22:37 /usr/lib/gtk-3.0/3.0.0/immodules.cache

645012e603ca81e5a8145aa8c3c0986fe3a9e29f0ba7f86c0fed50b2ec89c9d0  /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
7bcab8ebb50a3c79be714b021384ff3cb233b093c8afdb3c265ed0e31f12be0b  /dirtyroot/usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache

5e9566df82165328a90a9ef4700763eee41819555ace60e13c7ee072f9419c2c  /usr/lib/gio/modules/giomodule.cache
64193c96f3c2a02a638633c37d81877905d4692744a9549f02d6b6f12a387ddb  /dirtyroot/usr/lib/gio/modules/giomodule.cache

5d52988c67f67e0c16eb1ba3d3bf6df5730fc9b6b57b789aaf5d8eacc29af685  /usr/lib/gtk-3.0/3.0.0/immodules.cache
12ad72140dd723a11cbb38a6dad5818f9fa86258a9c8ce36d71d8bc5bc7d7ce9  /dirtyroot/usr/lib/gtk-3.0/3.0.0/immodules.cache

da0eba8489e1ea7e5ad3c80de4d91c0ebcaa49e4f13737b3afc03fae82b6d9c5  /usr/lib/jvm/java-1.7.0-openjdk-
d2f9bebee910f36ffa6f8975cc7d37206c8b3fab0d27c6addacf6fcfd2088b73  /dirtyroot/usr/lib/jvm/java-1.7.0-openjdk-

da0eba8489e1ea7e5ad3c80de4d91c0ebcaa49e4f13737b3afc03fae82b6d9c5  /usr/lib/jvm/jre-1.7.0-openjdk/lib/i386/server/classes.jsa
d2f9bebee910f36ffa6f8975cc7d37206c8b3fab0d27c6addacf6fcfd2088b73  /dirtyroot/usr/lib/jvm/jre-1.7.0-openjdk/lib/i386/server/classes.jsa

d4d7dca0429dc2e35123582d72038b14099c6e80deb55b1fc080387123add20e  /usr/share/t1lib/FontDatabase
8c9d7bedc2ab05489af5f3a54895439ca81fc82ef6616cd9e7e57706229da277  /dirtyroot/usr/share/t1lib/FontDatabase

f128b934b4fbbb45b385da5f64a82ed4af244eeed546529428ad89e011cfcf5e  /usr/share/t1lib/t1lib.config
d9876860bbcd945ff311eca5b14a380f6ce2aa3f3dcc2c0dc4227bb103fbf008  /dirtyroot/usr/share/t1lib/t1lib.config

adc0ed2c6d05d38b30763bed179a8aa8675397dab9b36c143af2e6dbcf2b274b  /etc/mail/access.db
7c2979d6adb0c8e00314eaefe243d4840d473f5e71aed4f63199594c7e4cf152  /dirtyroot/etc/mail/access.db

6f26fcc80a059ca5b30791529dc411fdf7466082003fa6fefb49da52dc642519  /etc/mail/domaintable.db
f7acafdb07c4d21b2480e258f2215b1aa647e7ef550d01eb574002c01bc3ab28  /dirtyroot/etc/mail/domaintable.db

bc2b8d132802597bca5c6e88909d9956ad966e91791d610db13eaf3fcd594558  /etc/mail/mailertable.db
df58fe2298da69787ad091d92b71ed77ccd60270e54c5040f77cc76c5f331a98  /dirtyroot/etc/mail/mailertable.db

eded081d4112bb6a878aae6452fb3fc0f95fc6736be898024231f53fb6f65811  /etc/mail/virtusertable.db
40b883559585624958aed94b9d119b5fd9fd57836c24c4723370fec98c4f2642  /dirtyroot/etc/mail/virtusertable.db

I was also having a problem with /usr/lib/libnss_compat_ossl.so.0.0.0 (from the nss_compat_ossl package), but that went away with a "yum reinstall".  Keep in mind that both installations passed "rpm -V".

It seems like there might be more than one version of these packages out in the yum-repo mirrors, but since they're all digitally signed with Fedora's private key, I don't see how this can be.  Still...something very strange is going on, and I fear it's security-related.

I was having similar issues with F17, but since that just went end-of-life, I redid the investigation with F18, and found pretty much the same thing.  java and sendmail come up consistently, and pkcs11-helper has shown up too, but isn't at the moment.

Finally, practically every .mod file in /boot/grub2/i386-pc came up different, though those apparently aren't associated with any package.

Any idea what could be behind these differences?
Comment 1 Kevin Fenzi 2013-07-06 23:27:39 EDT
All of those seem to be files that are generated at rpm install time via scriptlets, and contain things like timestamps/dates.
Comment 2 Steve 2013-07-07 11:52:29 EDT
Still, that doesn't explain the oddness with nss_compat_ossl and pkcs11-helper, does it?  Or how practically every .mod file in /boot/grub2/i386-pc is different?

Also...is the process that I outlined for creating a clean system a sensible one?  None of these comparisons have any meaning if it's just as compromised as my in-use system.

I guess I should have kept the first version of my possibly-compromised system, but at the time, I wasn't thinking about reporting the problem, just fixing it.
Comment 3 Kevin Fenzi 2013-07-07 12:31:49 EDT
(In reply to Steve from comment #2)
> Still, that doesn't explain the oddness with nss_compat_ossl and
> pkcs11-helper, does it?  

Yeah, those I think are a real prelink bug with it not unprelinking them correctly. 

You could test by installing prelink, reinstall and prelink one of them, unprelink and see if it fails to verify. 

>Or how practically every .mod file in
> /boot/grub2/i386-pc is different?

Those are generated at install time. 

> Also...is the process that I outlined for creating a clean system a sensible
> one?  None of these comparisons have any meaning if it's just as compromised
> as my in-use system.

I have no idea what threats you are trying to protect against, so I can't say if this is a good model to protect against them. This should be a good way to have a reference system sure.
Comment 4 Steve 2013-07-07 15:49:08 EDT
I ran "prelink -ua; rpm -e prelink" on both my in-use and clean systems, so hopefully the problem will go away.  I'll report back here if anything else strange happens.

Thanks for pointing out %ghost, i.e. files that get generated at install time and so would be different between otherwise identical installations.

I'm trying to detect if/when my system gets compromised in any way.  I'm hoping to maintain my "clean" system by not using it for anything but installing/updating packages, so that I can compare it to my in-use system and detect any changes, e.g. if tripwire or "rpm -V" somehow get compromised.

Now that I'm running Firefox in an SELinux sandbox, without Adobe Flash, and with the noscript addon, hopefully I've closed off all possible attack vectors.  I wish I could find more net.resources for this sort of thing.  linux-sec.net looks decent, but has a LOT of information.
Comment 5 Fedora End Of Life 2013-12-21 09:16:07 EST
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 6 Fedora End Of Life 2014-02-05 17:04:47 EST
Fedora 18 changed to end-of-life (EOL) status on 2014-01-14. Fedora 18 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.