Bug 728863 - Wrongly SELinux-labelled files in /lib(64) prevent boot of Fedora 16 Alpha RC systems with SELinux enabled
Wrongly SELinux-labelled files in /lib(64) prevent boot of Fedora 16 Alpha RC...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dracut (Show other bugs)
16
All Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Harald Hoyer
Fedora Extras Quality Assurance
AcceptedBlocker
:
: 729550 (view as bug list)
Depends On:
Blocks: F16Alpha/F16AlphaBlocker
  Show dependency treegraph
 
Reported: 2011-08-08 04:04 EDT by He Rui
Modified: 2011-08-23 16:37 EDT (History)
24 users (show)

See Also:
Fixed In Version: dracut-013-4.fc16
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-08-23 16:37:05 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
audit.log from F16 Alpha RC1 KDE Live, booted with enforcing=0 (127.24 KB, text/x-log)
2011-08-08 04:44 EDT, Sandro Mathys
no flags Details
audit.log from F16 Alpha RC1 KDE Live, booted with enforcing=0 - including installation with liveinst (164.40 KB, text/plain)
2011-08-08 05:04 EDT, Sandro Mathys
no flags Details
broken boot log (23.60 KB, text/plain)
2011-08-08 06:00 EDT, Kamil Páral
no flags Details
audit.log from F16 Alpha RC2 KDE Live, booted with enforcing=0 - including installation with liveinst (136.71 KB, text/plain)
2011-08-09 04:31 EDT, Sandro Mathys
no flags Details
audit.log from booting Alpha RC3 desktop x86_64 with enforcing=0 (88.54 KB, text/plain)
2011-08-09 15:53 EDT, Adam Williamson
no flags Details
listing of all the var_run_t files in /lib64 on desktop rc3 boot (2.66 KB, text/plain)
2011-08-09 15:57 EDT, Adam Williamson
no flags Details
audit.log with enforcing=0 from system installed with Alpha RC3 KDE live media (58.10 KB, text/plain)
2011-08-10 04:51 EDT, Sandro Mathys
no flags Details
/var/log/messages (291.17 KB, text/plain)
2011-08-10 06:41 EDT, He Rui
no flags Details
audit.log with enforcing=0 from system installed with Alpha RC3 DVD (63.84 KB, text/x-log)
2011-08-10 07:00 EDT, Sandro Mathys
no flags Details

  None (edit)
Description He Rui 2011-08-08 04:04:45 EDT
Description of problem:
Boot F16-alpha-rc1, it failed to access to the gdm login where the window was black with the cursor kept loading. Tried to login from other ttys, it didn't allow root or liveuser login. 

When appending enforcing=0(or selinux=0) as kernel option, it can accessed to desktop then.

Version-Release number of selected component (if applicable):
F16-Alpha-RC1

libselinux-2.0.102-6.fc16.x86_64
libselinux-python-2.0.102-6.fc16.x86_64
libselinux-utils-2.0.102-6.fc16.x86_64
selinux-policy-targeted-3.10.0-10.fc16.noarch
selinux-policy-3.10.0-10.fc16.noarch

How reproducible:
100%
Comment 1 Sandro Mathys 2011-08-08 04:43:08 EDT
I can confirm this with both, KDE and Gnome RC1 live media (Xfce not tested).

Problem doesn't seem to be gdm-specific, at least iptables and kdm don't start either. Also, audit.log reveals lots of issues even with the system's core functions.
Comment 2 Sandro Mathys 2011-08-08 04:44:38 EDT
Created attachment 517132 [details]
audit.log from F16 Alpha RC1 KDE Live, booted with enforcing=0
Comment 3 Sandro Mathys 2011-08-08 05:04:13 EDT
Created attachment 517142 [details]
audit.log from F16 Alpha RC1 KDE Live, booted with enforcing=0 - including installation with liveinst

Using liveinst to complete an installation from the live media added some more selinux denials, therefore I updated the audit.log
Comment 4 Kamil Páral 2011-08-08 06:00:28 EDT
Created attachment 517155 [details]
broken boot log

Not only it does not start gdm for me, it doesn't boot at all if I don't specify selinux=0. Attached log.
Comment 5 Sandro Mathys 2011-08-08 06:58:37 EDT
Kamil: I think you're misinterpreting this - it looks like it won't boot at all, but you can eventually switch to a tty and get a getty (login won't work, though). At least, that's what I see with the KDE live media. With Gnome live media I see the plymouth splash all fine but I'm then dropped to a tty after loading X/gdm fails.

Interestingly, if you install the livecd's (kde based) system, X/kdm will work. Login still won't, though (not with getty, not with kdm - not as user, not as root).
Comment 6 Sandro Mathys 2011-08-08 07:53:46 EDT
On the installed system, the update to selinux-policy-3.10.0-15.fc16 did resolve all problems. Including this version on the live media might also fix the issues there.
Comment 7 Miroslav Grepl 2011-08-08 08:12:33 EDT
Yes, these issues should be solved in the latest F16 build.
Comment 8 Tim Flink 2011-08-08 12:39:12 EDT
Discussed in the 2011-08-08 Fedora QA meeting. Accepted as a Fedora 16 alpha blocker as it violates the following alpha release criterion [1]:

... after firstboot is completed and on subsequent boots, a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to a working graphical environment without unintended user intervention

[1] https://fedoraproject.org/wiki/Fedora_16_Alpha_Release_Criteria
Comment 9 Adam Williamson 2011-08-08 16:26:43 EDT
Miroslav, can you mark the update as fixing this bug?

Update is:

https://admin.fedoraproject.org/updates/selinux-policy-3.10.0-15.fc16

Marking this as ON_QA.
Comment 10 Sandro Mathys 2011-08-09 04:10:28 EDT
The Alpha RC2 KDE live media has selinux-policy-3.10.0-15.fc16.noarch but fails to boot, i.e. shows lots of denials if booted in permissive mode.

Will check whether the installed system does any better.
Comment 11 Sandro Mathys 2011-08-09 04:31:25 EDT
Created attachment 517364 [details]
audit.log from F16 Alpha RC2 KDE Live, booted with enforcing=0 - including installation with liveinst
Comment 12 Miroslav Grepl 2011-08-09 04:51:57 EDT
Well, the problem is, the /lib(64)? directory is mislabeled and libraries are labeled as var_run_t instead of lib_t.

# restorecon -R -v /lib64

will fix the issue.
Comment 13 Jens Petersen 2011-08-09 05:00:41 EDT
Ok, but what is the fix for Alpha Live media?
Do we need another bug for that or do we move this back to ASSIGNED?

I just tried Fedora-16-Alpha-i686-Live-Desktop.iso
(which has selinux-policy-3.10.0-15.fc16) and still
can't reach to Gnome without using enforcing=0.
Comment 14 Jens Petersen 2011-08-09 05:03:48 EDT
> I just tried Fedora-16-Alpha-i686-Live-Desktop.iso

Sorry, as in Alpha RC2.
Comment 15 Jens Petersen 2011-08-09 05:10:52 EDT
Same story with RC2 Fedora-16-Alpha-i686-Live-XFCE.iso.

I see lots of system services failing to start.
Comment 16 Sandro Mathys 2011-08-09 05:18:36 EDT
Who is responsible for the labels on the livecd? I figure we should reassign this to that person.
Comment 17 Miroslav Grepl 2011-08-09 05:28:30 EDT
I am looking what is wrong.
Comment 18 Miroslav Grepl 2011-08-09 07:21:38 EDT
Well I have more issues:

1. I have installed Fedora 16 Alpha RC2 from Fedora-16-Alpha-x86_64-netinst.iso and I see SELinux is disabled by default.

2. I am trying to start Live CD media but I see the same issue. It looks like to me Live media is built incorrectly.
Comment 19 Miroslav Grepl 2011-08-09 07:22:54 EDT
Where could I get kickstarts which are used to build the Live media?
Comment 20 Charles R. Anderson 2011-08-09 08:53:44 EDT
Just like RC1, RC2 Desktop Live also won't boot up fully without
enforcing=0.  Lots of services fail startup, like "Remount Root FS",
"Initialize storage subsystems (RAID, LVM, etc.)", tmpfiles, and
"Network Manager".  It just hangs at 99% completion on the Fedora
bubble logo.  Last message on text screen is "Started D-Bus System
Message Bus."

Ctrl-Alt-Delete won't reboot either.  Have to force power off.

enforcing=0 allows boot to proceed normally to Live Desktop.

This is on a Lenovo ThinkPad T520.
Comment 21 Daniel Walsh 2011-08-09 09:27:31 EDT
What AVC's are you seeing when you login in permissive mode?
Comment 22 Kevin Fenzi 2011-08-09 13:00:43 EDT
kickstarts are at: 
http://git.fedorahosted.org/git/?p=spin-kickstarts.git

They do a restorecon at the end, so not sure whats going on.
Comment 23 Adam Williamson 2011-08-09 15:34:53 EDT
I recall we've had cases in the past where this kind of problem can involve the live compose environment: specifically, composing the lives with selinux disabled can lead to selinux trouble. Dennis, what was the environment for this live compose?
Comment 24 Adam Williamson 2011-08-09 15:52:36 EDT
attaching an audit.log from rc3 GNOME boot, but it's much the same as sandro's - definitely those var_run_t files in /lib64 are the issue. i'm not right in comment #23, apparently, we've been building lives with selinux disabled for a while now (live compose was fixed so you can do that), so I have no idea why this is :( dan, can you please look into this? it's crucial to alpha and we have the go/no-go tomorrow...thanks! you can get the image to test at http://dl.fedoraproject.org/pub/alt/stage/16-Alpha.RC3/Live/ .
Comment 25 Adam Williamson 2011-08-09 15:53:30 EDT
Created attachment 517475 [details]
audit.log from booting Alpha RC3 desktop x86_64 with enforcing=0
Comment 26 Adam Williamson 2011-08-09 15:56:54 EDT
Here's a listing of all the files with var_run_t in /lib64, no immediate link between them jumps out at me.
Comment 27 Adam Williamson 2011-08-09 15:57:24 EDT
Created attachment 517476 [details]
listing of all the var_run_t files in /lib64 on desktop rc3 boot
Comment 28 Adam Williamson 2011-08-09 19:09:25 EDT
Dan said he had a potential fix for this, but then dropped off line and I haven't been able to raise him since. Dan?
Comment 29 Jens Petersen 2011-08-09 20:50:53 EDT
As far as I know mclasen is on vacation, so not a good person
to own this bug and anyway LiveCD just means Live Desktop (Gnome)
I suppose and this is a generic problem.

(In reply to comment #18)
> 1. I have installed Fedora 16 Alpha RC2 from Fedora-16-Alpha-x86_64-netinst.iso
> and I see SELinux is disabled by default.

LOL, so it is! (ie same here for my RC1 net install)
Is there a bug filed for that?

So I am guessing now this issue is not specific to Live at all?
Comment 30 Adam Williamson 2011-08-09 21:39:47 EDT
jens: well, selinux being disabled by default in a non-live install would be a different issue. of course, it may be the case that if you turn it on, you hit this bug, but we don't know that for sure yet.
Comment 31 Adam Williamson 2011-08-09 22:12:41 EDT
Dan's latest thoughts on this, for anyone who wants to pick this up:

<dwalsh> I think there is something wrong with the final step.
 Basically I look on disk and I see XAttrs
 When it makes up the final ISO, noxattrs.
 I unloaded the iso and looked and it has no xattrs on the ext3 file system.
 I am trying to stop after the internal relabel now and see if labels are truly there, if they are then something in making the filesystem is broken.
<dwalsh> The funny thing is this all works with F15, and I don't think it has anything to do with the actual OS.  Something changed in one of the file systems...
 Or livecd.
<dwalsh> I just looked in /var/tmp/img.. and the labels are on disk
<adamw> so it's somewhere in the image creation step that the xattrs get lost?
 er, filesystem rather
<dwalsh> That is my guess
<adamw> okay
<dwalsh> While you are building you can run ls -lZ /var/tmp/imgcreate-XXXXX/install_root/lib64
 And you will see labels.
Comment 32 Jens Petersen 2011-08-10 02:45:05 EDT
*** Bug 729550 has been marked as a duplicate of this bug. ***
Comment 33 Adam Williamson 2011-08-10 03:47:08 EDT
I checked out a DVD install: indeed selinux is disabled by default, and that's wrong. I enabled selinux and rebooted, and it vomited all over its shoes, but the fail looks a bit different from live. i'll reboot with permissive and see what the exact avcs are, and hence whether dvd installs are actually suffering from this too.
Comment 34 Jens Petersen 2011-08-10 03:51:36 EDT
(In reply to comment #18)
> 1. I have installed Fedora 16 Alpha RC2 from Fedora-16-Alpha-x86_64-netinst.iso
> and I see SELinux is disabled by default.

I filed bug 729563 for this.
Comment 35 Adam Williamson 2011-08-10 03:57:19 EDT
ug. unfortunately, if you boot with permissive, it does a relabel, which seems
to resolve the issue, then reboots. so i lost the avcs. should be fairly easy
to reproduce, though: install from dvd, then switch from 'disabled' to
'enforcing' in /etc/selinux/config and reboot.
Comment 36 Sandro Mathys 2011-08-10 04:07:58 EDT
To prevent the relabelling, you need to remove /.autorelabel before enabling selinux (in permissive or enforcing) and rebooting.
Comment 37 Sandro Mathys 2011-08-10 04:51:24 EDT
Created attachment 517545 [details]
audit.log with enforcing=0 from system installed with Alpha RC3 KDE live media
Comment 38 He Rui 2011-08-10 06:41:47 EDT
Created attachment 517570 [details]
/var/log/messages

I installed F16-alpha-rc3 dvd. Then set selinux as enforcing, removed autorelabel file, and reboot. It failed to login so I reboot again with enforcing=o and got the log.
Comment 39 Sandro Mathys 2011-08-10 07:00:54 EDT
Created attachment 517580 [details]
audit.log with enforcing=0 from system installed with Alpha RC3 DVD
Comment 40 Daniel Walsh 2011-08-10 11:03:04 EDT
It looks like this is a systemd/dracut problem.

I have submitted some fixes to livecd-tools to handle the SELinux errors on build, but it looks like the major problem is with systemd.

I have a F16 build where the files are all labeled corectly during the build but when I boot I see a number of mislabeled files in /lib64 and /usr/lib64.  These are all libraries labeled var_run_t.  

Funny think is, the exact same paths exist under /run/initramfs/.

For example I see the following on my F17 box.

ls -lZ /run/initramfs/lib64/libply.so.2.0.0 /lib64/libply.so.2.0.0 
-rwxr-xr-x. root root system_u:object_r:lib_t:s0       /lib64/libply.so.2.0.0
-rwxr-xr-x. root root system_u:object_r:var_run_t:s0   /run/initramfs/lib64/libply.so.2.0.0


But on the livecd for F16 I see something like

ls -lZ /run/initramfs/lib64/libply.so.2.0.0 /lib64/libply.so.2.0.0 
-rwxr-xr-x. root root system_u:object_r:var_run_t:s0       /lib64/libply.so.2.0.0
-rwxr-xr-x. root root system_u:object_r:var_run_t:s0   /run/initramfs/lib64/libply.so.2.0.0


I know that systemd does some relabeling during the boot especially of the /run directory.  I believe there is a bug where it is asking SELinux how to label /run/initramfs/lib64/BLAH and then applying the
Comment 41 Adam Williamson 2011-08-10 12:04:56 EDT
making summary more accurate to our current knowledge of the bug.
Comment 42 Fedora Update System 2011-08-10 13:05:05 EDT
dracut-011-40.git20110810 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-011-40.git20110810
Comment 43 Fedora Update System 2011-08-10 14:24:44 EDT
dracut-011-41.git20110810 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-011-41.git20110810
Comment 44 Fedora Update System 2011-08-10 16:45:05 EDT
Package dracut-011-41.git20110810:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-011-41.git20110810'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/dracut-011-41.git20110810
then log in and leave karma (feedback).
Comment 45 Adam Williamson 2011-08-10 18:43:56 EDT
the update does seem to fix this specific issue...i built a compose with the updated dracut, and it doesn't get the same errors and avcs, and it boots further (to a spinning cursor).

but it's still somewhat broken: getting to the spinning cursor takes a long time (I think something earlier in boot takes out), it never reaches GNOME, you can get a login prompt on tty3, 4, 5, 6 (but not 1 or 2) and you can't log in to the console: it displays some kind of error very briefly, too briefly for me to read, and loops back to the login prompt.

I was composing with a slightly patched python-imgcreate (some fixes Dan made as part of a first attempt to fix this bug), so I'll confirm with a non-patched one. But I think it's just still a bit broken even with this fixed.
Comment 46 Adam Williamson 2011-08-10 19:25:06 EDT
okay, after downgrading python-imgcreate (i.e. reverting dan's 'fix') and using livecd-desktop.ks rather than live-desktop.ks, it looks good: boots to a working gnome session, with no delay, with SELinux enabled. Fix looks good to me, for live at least.
Comment 47 Jens Petersen 2011-08-11 02:19:16 EDT
Great, I can also confirm that Fedora-16-Alpha-i686-Live-Desktop.iso
from http://koji.fedoraproject.org/koji/taskinfo?taskID=3264441
with dracut-011-40.git20110810 boots to desktop normally. :-)
Comment 48 Harald Hoyer 2011-08-11 03:47:24 EDT
(In reply to comment #47)
> Great, I can also confirm that Fedora-16-Alpha-i686-Live-Desktop.iso
> from http://koji.fedoraproject.org/koji/taskinfo?taskID=3264441
> with dracut-011-40.git20110810 boots to desktop normally. :-)

please use dracut-011-41.git20110810
Comment 49 Fedora Update System 2011-08-12 10:34:53 EDT
dracut-013-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-013-1.fc16
Comment 50 Tim Flink 2011-08-15 10:53:00 EDT
There have been two reports of this being fixed, moving to VERIFIED.

Note that bug 730579 was filed against dracut-013-1.fc16 and unless that bug is resolved, it shouldn't be pulled in for alpha and dracut-011-41.git20110810 should be used for now.
Comment 51 Jens Petersen 2011-08-16 06:33:28 EDT
(In reply to comment #48)
> please use dracut-011-41.git20110810

Now finally RC4 now has dracut-011-41 and looks fine too.
Comment 52 Fedora Update System 2011-08-17 02:44:36 EDT
dracut-013-3.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-013-3.fc16
Comment 53 Fedora Update System 2011-08-17 08:08:37 EDT
dracut-013-4.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-013-4.fc16
Comment 54 Fedora Update System 2011-08-17 08:08:50 EDT
dracut-013-4.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/dracut-013-4.fc16
Comment 55 Fedora Update System 2011-08-17 16:23:48 EDT
Package dracut-013-4.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-013-4.fc16'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/dracut-013-4.fc16
then log in and leave karma (feedback).
Comment 56 Adam Williamson 2011-08-17 17:11:15 EDT
biff-baff!
Comment 57 Harald Hoyer 2011-08-18 08:46:26 EDT
(In reply to comment #56)
> biff-baff!

which means dracut-013-4.fc16 works?
Comment 58 Tim Flink 2011-08-18 10:30:40 EDT
(In reply to comment #57)
> (In reply to comment #56)
> > biff-baff!
> 
> which means dracut-013-4.fc16 works?

This bug was considered fixed with dracut-011-41.git20110810 and that's what has been tested for release with Fedora 16 alpha.
Comment 59 Adam Williamson 2011-08-18 15:10:04 EDT
harald: ah, no, sorry - i shouldn't have moved it back to VERIFIED. forgot you kept bumping this. for alpha we used 011-41 as tim said, we haven't really tested post-011.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 60 Fedora Update System 2011-08-23 16:36:26 EDT
dracut-013-4.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.