Bug 1116450 - Can't login to fresh rawhide installation (2014-07-04) if SELinux is enforcing
Summary: Can't login to fresh rawhide installation (2014-07-04) if SELinux is enforcing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: lorax
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Brian Lane
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 1052317 1118192 (view as bug list)
Depends On:
Blocks: F21AlphaBlocker
TreeView+ depends on / blocked
 
Reported: 2014-07-04 16:24 UTC by Petr Spacek
Modified: 2014-07-17 21:14 UTC (History)
21 users (show)

Fixed In Version: lorax-21.16-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-07-17 21:14:32 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
audit.log.bz2 (4.08 KB, application/x-bzip)
2014-07-04 16:24 UTC, Petr Spacek
no flags Details
journal of an affected boot. booted at 17:06, attempted to log in graphically at 17:07, attempted to log in via console at 17:08, rebooted at 17:09 (173.24 KB, text/plain)
2014-07-05 00:25 UTC, Adam Williamson
no flags Details
journal.log from 2014-05-12 (unaffected) installation (537.04 KB, text/plain)
2014-07-11 00:28 UTC, Adam Williamson
no flags Details
journal.log from 2014-07-10 (affected) installation (719.09 KB, text/plain)
2014-07-11 00:29 UTC, Adam Williamson
no flags Details

Description Petr Spacek 2014-07-04 16:24:32 UTC
Created attachment 914746 [details]
audit.log.bz2

Description of problem:
I was presented with standard shell login screen. I entered my root credentials, hit enter, screen flashed and I was back on login screen. Tried this a couple of times and result was always the same.

This was happening with selinux in enforcing mode. (Eventually I could log in with init=/bin/bash on kernel command line, then I disabled SELinux and I was able to log in properly.)


Version-Release number of selected component (if applicable):
selinux-policy-3.13.1-62.fc21.noarch

How reproducible:
100 %

Steps to Reproduce:
1. Install fresh rawhide (I did that on 2014-07-04)
2. Try to log-in to text console

Actual results:
Access to /usr/bin/bash is denied.

Additional info:
I'm going to attach audit.log.

This part is probably the most interesting:
type=CRED_REFR msg=audit(1404489277.629:84): pid=745 uid=0 auid=0 ses=2 subj=system_u:system_r:kernel_t:s0 msg='op=PAM:setcred acct="root" exe="/usr/bin/login" hostname=? a
ddr=? terminal=tty1 res=success'
type=USER_LOGIN msg=audit(1404489277.630:85): pid=745 uid=0 auid=0 ses=2 subj=system_u:system_r:kernel_t:s0 msg='op=login id=0 exe="/usr/bin/login" hostname=? addr=? termin
al=tty1 res=success'
type=AVC msg=audit(1404489277.637:86): avc:  denied  { transition } for  pid=838 comm="login" path="/usr/bin/bash" dev="dm-1" ino=4208 scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process permissive=0

Comment 1 Adam Williamson 2014-07-05 00:23:17 UTC
Myself, satellit and xmrbrz from #fedora-qa all also saw this issue. It affects regular user accounts as well as root, and both console and graphical logins. Login works successfully when booted with 'enforcing=0'. Updating to selinux-policy-targeted 3.13.1-63.fc21 does not help. I did a default network install from 2014-07-04 x86_64 boot.iso to a KVM.

Logs don't seem terribly helpful, but I'll attach one shortly for reference.

Nominating as an Alpha blocker, violates "A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility." - https://fedoraproject.org/wiki/Fedora_21_Alpha_Release_Criteria#Expected_installed_system_boot_behavior .

Comment 2 Adam Williamson 2014-07-05 00:25:31 UTC
Created attachment 914796 [details]
journal of an affected boot. booted at 17:06, attempted to log in graphically at 17:07, attempted to log in via console at 17:08, rebooted at 17:09

Comment 3 Miroslav Grepl 2014-07-07 06:05:47 UTC
Did you try to fix labeling on your system? The problem is we see kernel_t.

Comment 4 Petr Spacek 2014-07-07 06:33:24 UTC
Could you be more specific, please? What should I try?

I did clean installation so I didn't feel necessity of 'fixing' anything, it should just work :-) I can give you access to VM console if you want to look into it yourself.

Comment 5 Miroslav Grepl 2014-07-07 06:56:55 UTC
(In reply to Petr Spacek from comment #4)
> Could you be more specific, please? What should I try?
> 
> I did clean installation so I didn't feel necessity of 'fixing' anything, it
> should just work :-) 

Yes it should. But maybe there is a bug.

>I can give you access to VM console if you want to look
> into it yourself.

It would be great.

Comment 6 Miroslav Grepl 2014-07-07 07:41:00 UTC
The problem is we have

lrwxrwxrwx. root root system_u:object_r:root_t:s0      sbin -> /usr/sbin

..
..

dr-xr-xr-x. root root system_u:object_r:root_t:s0      bin
drwxr-xr-x. root root system_u:object_r:root_t:s0      games
drwxr-xr-x. root root system_u:object_r:root_t:s0      include
dr-xr-xr-x. root root system_u:object_r:root_t:s0      lib
dr-xr-xr-x. root root system_u:object_r:lib_t:s0       lib64
drwxr-xr-x. root root system_u:object_r:root_t:s0      libexec
drwxr-xr-x. root root system_u:object_r:root_t:s0      local
dr-xr-xr-x. root root system_u:object_r:root_t:s0      sbin
drwxr-xr-x. root root system_u:object_r:root_t:s0      share
drwxr-xr-x. root root system_u:object_r:root_t:s0      src
lrwxrwxrwx. root root system_u:object_r:root_t:s0      tmp -> ../var/tmp

Comment 7 Miroslav Grepl 2014-07-07 07:44:59 UTC
Adam,
do you have the same labeling issue?

Comment 8 Bruno Roberto Zanuzzo 2014-07-07 08:15:48 UTC
I logged with enforcing=0 (boot parameter) and used the "setenforce 1" command before you list the permissions of context, if it is correct, these are my results:

[root@localhost /]# ls -Z /
lrwxrwxrwx. root root system_u:object_r:root_t:s0      sbin -> usr/sbin
[...]

[root@localhost /]# ls -Z /usr/
dr-xr-xr-x. root root system_u:object_r:root_t:s0      bin
drwxr-xr-x. root root system_u:object_r:root_t:s0      games
drwxr-xr-x. root root system_u:object_r:root_t:s0      include
dr-xr-xr-x. root root system_u:object_r:root_t:s0      lib
dr-xr-xr-x. root root system_u:object_r:lib_t:s0       lib64
drwxr-xr-x. root root system_u:object_r:root_t:s0      libexec
drwxr-xr-x. root root system_u:object_r:root_t:s0      local
dr-xr-xr-x. root root system_u:object_r:root_t:s0      sbin
drwxr-xr-x. root root system_u:object_r:root_t:s0      share
drwxr-xr-x. root root system_u:object_r:root_t:s0      src
lrwxrwxrwx. root root system_u:object_r:root_t:s0      tmp -> ../var/tmp

Comment 9 Tim Flink 2014-07-09 17:47:40 UTC
Discussed at the 2014-07-09 Fedora 21 alpha blocker review meeting. Accepted as a blocker for Fedora 21 alpha due to violation of the following alpha release criteria [1]:

A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility.

A system installed without a graphical package set must boot to a state where it is possible to log in through at least one of the default virtual consoles. 

[1] https://fedoraproject.org/wiki/Fedora_21_Alpha_Release_Criteria#Expected_installed_system_boot_behavior

Comment 10 Adam Williamson 2014-07-10 00:33:41 UTC
the filesystem package has not changed since March 2014, and I don't quite understand what it is you're saying is wrong. You said "the problem is we have" and then posted a directory listing, but didn't unpack beyond that: what element of that directory listing is the problem, and are we sure it's caused by filesystem? If so, why did this break now and not in March?

Thanks!

Comment 11 Miroslav Grepl 2014-07-10 13:44:20 UTC
Ah, I wanted to move it to anaconda. 

The problem is we end up with root_t labeling for "sbin" and "bin" top level symlinks. So we need to find out who places these symlinks.

Comment 12 satellitgo 2014-07-10 18:01:06 UTC
'enforcing=0' required for f21 20140710 live soas x86_64 spin in VirtualBox to start.
liveinst from terminal boots VirtualBox HD install without requiring 'enforcing=0'

Comment 13 Adam Williamson 2014-07-10 20:47:14 UTC
and presumably the correct labelling is what I have on my previously-installed system:

[adamw@adam ~]$ ls -lZ /*bin
lrwxrwxrwx. root root system_u:object_r:bin_t:s0       /bin -> usr/bin
lrwxrwxrwx. root root system_u:object_r:bin_t:s0       /sbin -> usr/sbin
[adamw@adam ~]$ ls -ldZ /usr/*bin
dr-xr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin
dr-xr-xr-x. root root system_u:object_r:bin_t:s0       /usr/sbin

right? actually, it seems like the labelling of several of the other directories at the top level and in /usr is wrong, compared to my existing installed system.

I'll try and do a bit of triage on this using older boot.iso images and hopefully live images, if I can find some that install - at least then we can get an idea whether the breakage is in anaconda, or some package that's part of the installed system, or what.

Comment 14 David Shea 2014-07-10 20:51:40 UTC
*** Bug 1118192 has been marked as a duplicate of this bug. ***

Comment 15 Adam Williamson 2014-07-10 22:50:34 UTC
I just did one test at least which points to anaconda (or the installer environment, at least). I did an install from an older known-good boot.iso I keep around, from 2014-05-12. That's a network install, so it installed a completely contemporary package set - selinux-policy-targeted-3.13.1-63.fc21 , for instance, and kernel 3.16.0-0.rc4.git2.1.fc21 . But I can log into the installed system successfully, and the labels on /bin and /sbin are correct (bin_t).

so, this at least indicates it's not the packages getting installed, but something the installer's doing or something in the installer environment that broke.

in any case, it's definitely not filesystem, so i'm gonna punt it to anaconda for now. I'll compare install logs from the 2014-05-12 install and a buggy install with recent boot.iso and see if anything looks different.

Comment 16 Adam Williamson 2014-07-10 23:33:27 UTC
OK, comparing logs from a 2014-05-12 boot.iso install with a 2014-07-10 boot.iso install (obviously, I confirmed the 2014-05-12 install is not affected by the bug, the 2014-07-10 install is affected by it), I see one possibly significant difference in the journal output.

20140710 has this, which 20140512 does not have, during selinux initialization:

Jul 10 22:58:25 localhost kernel: SELinux:  Permission audit_read in class capability2 not defined in policy.
Jul 10 22:58:25 localhost kernel: SELinux: the above unknown classes and permissions will be allowed

That's the only thing that leaps out at me, but the logs are long and I'm not sure what I'm looking for exactly. I'll attach both.

Comment 17 Adam Williamson 2014-07-11 00:28:48 UTC
Created attachment 917206 [details]
journal.log from 2014-05-12 (unaffected) installation

Comment 18 Adam Williamson 2014-07-11 00:29:19 UTC
Created attachment 917207 [details]
journal.log from 2014-07-10 (affected) installation

Comment 19 Brian Lane 2014-07-11 00:57:47 UTC
I ran into this using a boot.iso from last week, relabeling the system fixed it so I think we just need to wait for an updated boot.iso

Comment 20 Miroslav Grepl 2014-07-11 08:06:39 UTC
*** Bug 1052317 has been marked as a duplicate of this bug. ***

Comment 21 Miroslav Grepl 2014-07-11 08:18:11 UTC
The problem is the labels are placed wrongly by a tool during install/upgrade phase.

Comment 22 Miroslav Grepl 2014-07-11 12:29:03 UTC
So I don't see it as selinux-policy issue.

Comment 23 Adam Williamson 2014-07-11 18:41:28 UTC
anaconda devs are saying there is no 'tool' that sets labels during install/upgrade phase. presumably it gets done by selinux *somehow*. all anaconda does is install packages.

Comment 24 Colin Walters 2014-07-11 18:43:04 UTC
That comment from anaconda is incorrect, see 80-setfilecons.ks.

However, I doubt it's the source of this bug.

Comment 25 Brian Lane 2014-07-11 18:45:14 UTC
Here's what /mnt/sysimage looks like during (and after) an install from a boot.iso created on 7/11:

lrwxrwxrwx. root root system_u:object_r:root_t:s0      bin -> usr/bin
dr-xr-xr-x. root root system_u:object_r:boot_t:s0      boot
drwxr-xr-x. root root system_u:object_r:device_t:s0    dev
drwxr-xr-x. root root system_u:object_r:root_t:s0      etc
drwxr-xr-x. root root system_u:object_r:root_t:s0      home
lrwxrwxrwx. root root system_u:object_r:root_t:s0      lib -> usr/lib
lrwxrwxrwx. root root system_u:object_r:lib_t:s0       lib64 -> usr/lib64
drwx------. root root system_u:object_r:lost_found_t:s0 lost+found
drwxr-xr-x. root root system_u:object_r:root_t:s0      media
drwxr-xr-x. root root system_u:object_r:root_t:s0      mnt
drwxr-xr-x. root root system_u:object_r:usr_t:s0       opt
dr-xr-xr-x. root root system_u:object_r:proc_t:s0      proc
dr-xr-x---. root root system_u:object_r:admin_home_t:s0 root
drwxr-xr-x. root root system_u:object_r:var_run_t:s0   run
lrwxrwxrwx. root root system_u:object_r:root_t:s0      sbin -> usr/sbin
-rw-r--r--. root root system_u:object_r:etc_runtime_t:s0 selinux-root-sysimage.txt
drwxr-xr-x. root root system_u:object_r:var_t:s0       srv
dr-xr-xr-x. root root system_u:object_r:sysfs_t:s0     sys
drwxrwxrwt. root root system_u:object_r:tmp_t:s0       tmp
drwxr-xr-x. root root system_u:object_r:root_t:s0      usr
drwxr-xr-x. root root system_u:object_r:root_t:s0      var

A number of these are clearly labeled wrong. filesystem owns these but some are created before filesystem is installed (the package ordering for this installation was libgcc, fedora-release, fedora-repos-rawhide, fedora-repos, setup, filesystem, ...)

A boot.iso from 06/27 works correctly. The labels are initially root_t but after about 10 packages are installed they switch to their correct values (about the time basesystem is installed). filesystem and basesystem have not changed in that time period so I'm fairly sure this is a selinux change causing this.

Comment 26 Colin Walters 2014-07-11 18:52:53 UTC
I wonder if this is actually fallout from http://rpm.org/gitweb?p=rpm.git;a=commitdiff;h=713914bde185393003b6a5fd1a229ebffd1fa784

The thing is of course when installing a pile of RPMs, since selinux-policy is itself an RPM[1], there's no policy loaded into the transaction until we get to that package.

I'm pretty sure the way it worked before was that sepolicy plugin was calling restorecon -R / in the install root.

But now we no longer have a plugin.

(BTW, this bug doesn't affect OSTree installs because we do SELinux labeling in a much more hardcoded, direct, and reliable fashion)

Comment 27 satellitgo 2014-07-11 19:47:01 UTC
(In reply to satellit from comment #12)
> 'enforcing=0' required for f21 20140710 live soas x86_64 spin in VirtualBox
> to start.
> liveinst from terminal boots VirtualBox HD install without requiring
> 'enforcing=0'

I have not been able to repeat this on bare metal or in later Virtual Box installs

Comment 28 Daniel Walsh 2014-07-11 20:03:46 UTC
Anaconda is supposed to have an selinux-policy in its build root that allows rpm to do proper labeling when content gets installed.  This looks like early part of install is screwed up,  Or these file/directories get created by anaconda and not rpm, which means they would get created with the wrong label.  We could setup a group of filename transitions for the directories created in / which would create them with the correct label.

Comment 29 Adam Williamson 2014-07-11 20:36:53 UTC
Colin: "That comment from anaconda is incorrect"

Please blame it on me, not them - I was paraphrasing from IRC notes, probably incorrectly. Apologies.

Comment 30 Brian Lane 2014-07-11 22:21:30 UTC
(In reply to Daniel Walsh from comment #28)
> Anaconda is supposed to have an selinux-policy in its build root that allows
> rpm to do proper labeling when content gets installed.  This looks like
> early part of install is screwed up,  Or these file/directories get created
> by anaconda and not rpm, which means they would get created with the wrong
> label.  We could setup a group of filename transitions for the directories
> created in / which would create them with the correct label.

It looks like they are created by yum or rpm at the start of the transaction -- at least I see them show up after anaconda-yum is started, but before any actual packages have been reported as being installed.

I built a 2nd boot.iso today using selinux-3.13.1-62 to see if that would fix the problem, and it doesn't. So I'm stumped as to what's going on here.

Comment 31 Brian Lane 2014-07-12 00:09:18 UTC
Ends up this was a combination of rpm splitting out selinux into a plugin and lorax removing too much from rpm when trimming down the boot.iso

Comment 32 Adam Williamson 2014-07-17 21:14:32 UTC
confirming this is fixed with a 2014-07-17 nightly F21 boot.iso. closing.


Note You need to log in before you can comment on or make changes to this bug.