+++ This bug was initially created as a clone of Bug #1228489 +++

Description of problem:

Fedora 22 minimal network installs complete with /etc/passwd and /etc/group with an SELinux context of system_u:object_r:shadow_t, making them effectively unreadable.  This causes numerous failures during boot, long stalls, and eventually a system that rejects all login attempts.

The same (Fedora server) netinst media was used for a half dozen installs over the 5/30-5/31 weekend with no issues; issue being reported was first encountered on 6/4, and was reproducible in a VM on 6/5.  I haven't been able to track down which package(s) might have introduced this, so I'm opening against anaconda for now, although I suspect the real issue is elsewhere.


How reproducible:

100% on Fedora Server netinst media and current minimal package set from 2015-06-04.


Steps to Reproduce:
1. Boot Fedora Server netinst media
2. Select minimal package set, proceed with installation
3. Set a root password for recovery
4. Observe long delays, service failures during first boot
5. Single-user system (rescue boot doesn't work, either)
6. Run restorecon -Rv /etc
7. Reboot, observe normal boot process and functional login prompt


Additional info:

Here's what restorecon will do on a system in this state:

[root@test ~]# restorecon -Rv /etc
restorecon reset /etc/group context system_u:object_r:shadow_t:s0->system_u:object_r:passwd_file_t:s0
restorecon reset /etc/passwd context system_u:object_r:shadow_t:s0->system_u:object_r:passwd_file_t:s0

--- Additional comment from David Shea on 2015-06-05 15:27:55 CEST ---

Please attach the log files from the install, available in /var/log/anaconda on the installed system.

--- Additional comment from Rob Foehl on 2015-06-05 16:40:53 CEST ---



--- Additional comment from Michal Schmidt on 2015-06-05 18:03:32 CEST ---

I'm seeing this bug too.

I made a script to observe the changes to /etc/passwd's SELinux context during installation. I run it in the shell on TTY1:

while :; do
  a=$(ls -Z /mnt/sysimage/etc/passwd 2>/dev/null)
  if [ "$a" != "$b" ]; then
    tail -n1 /tmp/dnf.rpm.log
    echo "Change: $b -> $a"
  fi
done

Then from its output and the contents of my /tmp/dnf.rpm.log I deduced these facts:

1. /etc/passwd gets created during installation of the "setup" package with the correct label: passwd_file_t.
2. When "avahi-autoipd" is being installed, its %pre scriptlet calls groupadd and useradd. At this moment the context of /etc/passwd changes to shadow_t.
3. Possibly relevant: "selinux-policy-targeted" is installed later, as one of the last packages.

--- Additional comment from Michal Schmidt on 2015-06-05 18:06:32 CEST ---

(In reply to Michal Schmidt from comment #3)
I lost a line in the script. Fixed script:

while :; do
  a=$(ls -Z /mnt/sysimage/etc/passwd 2>/dev/null)
  if [ "$a" != "$b" ]; then
    tail -n1 /tmp/dnf.rpm.log
    echo "Change: $b -> $a"
    b="$a"
  fi
done

--- Additional comment from Michal Schmidt on 2015-06-05 18:16:22 CEST ---

If instead of Minimal Install I select the default Fedora Server, the sequence is the following:

1. /etc/passwd gets created during installation of the "setup" package with the correct label: passwd_file_t.  (Same as with Minimal)
2. When "tog-pegasus-libs" is being installed, its %pre scriptlet calls groupadd and useradd. At this moment the context of /etc/passwd changes to shadow_t.
3. At some point "selinux-policy-targeted" is installed.
4. The first package afterwards to use groupadd/useradd in its %pre is "tcpdump". When it is being installed, the context of /etc/passwd changes back to passwd_file_t.

I hope this helps understanding why the problem is seen only with Minimal install and not with Server.

--- Additional comment from Michal Schmidt on 2015-06-05 18:36:29 CEST ---

I wanted to test what happens when installing without the updates repo enabled, but I am unable to, because of bug 1211907.

--- Additional comment from Michal Schmidt on 2015-06-05 18:58:37 CEST ---

OK, I was able to test minimal installation without updates by entering the URL instead of using "Closest mirror".

When not using the updates repo for installation I can see that /etc/passwd gets created as passwd_file_t and it never changes. This is as it should be.

So which update broke it? I'm guessing... this libselinux update which went stable recently:
https://admin.fedoraproject.org/updates/FEDORA-2015-8872/libselinux-2.3-10.fc22

--- Additional comment from David Shea on 2015-06-05 19:16:19 CEST ---

Changing whatever update caused the problem to not break F22 installs is probably for the best, but in the world of tomorrow anaconda could just go ahead and relabel the password and group files in %post. It already does that for shadow. https://github.com/rhinstaller/anaconda/pull/129

--- Additional comment from Rob Foehl on 2015-06-05 19:23:26 CEST ---

Interesting -- I'd attempted to test yesterday without installing updates and had the same issue, but I wasn't aware of bug 1211097.  That and the useradd hint mentioned above were starting to make me think I'd somehow dodged this in earlier installs and the issue may have been around longer than initially thought.

Just tried to reproduce again while allowing chrony to be installed, but that user/group addition didn't change anything.  The libselinux update looks like a good candidate...

--- Additional comment from Rob Foehl on 2015-06-05 19:25:12 CEST ---

Oops, that should've been bug 1211907...

--- Additional comment from Michal Schmidt on 2015-06-05 19:29:35 CEST ---

To confirm that the change in libselinux is the cause of this bug I made a mockbuild of libselinux-2.3-9 (the original F22 release) just with the release field bumped to 10.fc22.m1 (so that it looks newer than the release currently in updates). I pointed Anaconda to it as an additional repo:
  https://michich.fedorapeople.org/f22-libselinux/

With it the resulting installation has /etc/passwd labeled correctly and there's no problem booting it.

--- Additional comment from David Shea on 2015-06-08 16:45:32 CEST ---

Pushed the anaconda change to rawhide.

--- Additional comment from Petr Lautrbach on 2015-06-08 16:46:02 CEST ---

It's really related to the latest libselinux change. We consider SELinux as disabled when there's no /etc/selinux/config file, see bug #1219045.

It means that for groupadd, a system seams to be with SELinux disabled until the selinux-policy package is installed. In this case, groupadd doesn't care of a context of a new /ec/group file and this file gets context from the load policy and its type transition rule - when a file is created in a 'etc_t' directory by a 'groupadd_t' process then the will have 'shadow_t' type.

I can see several solutions here:

1. revert the change for #1219045 and consider SELinux disabled only if a system is run with a kernel without SELinux support or with selinux=0 - it will break setup without selinux-policy and without explicitly disabled SELinux 

2. change anaconda to copy /etc/selinux/config file from a ramdisk to chroot

3. install selinux-policy-targeted package as soon as possible in the installation transaction

--- Additional comment from Brian Lane on 2015-06-09 02:15:25 CEST ---

Maybe anaconda, or selinux (with some switch to control the behavior) should be relabeling everything? Since we can't really know for sure what's going to be installed with the wrong label before selinux is installed I think we need a more generic solution.

Having anaconda do it at the end of the installation would be safest, but add the most time. Having some kind of switch to communicate to the selinux rpm install would be faster since it would only run on what has been installed up to that point.

Note that there is no way for us to control the order that things are installed in.

--- Additional comment from  on 2015-06-09 03:12:01 CEST ---

Duplicate of bug 1228200

--- Additional comment from Michal Schmidt on 2015-06-09 12:50:10 CEST ---



--- Additional comment from Petr Lautrbach on 2015-06-09 16:11:39 CEST ---

Currently, selinux-policy package restores contexts only on "/root /var/log /run" during install while it also restores contexts on "/etc/passwd* /etc/group* /etc/*shadow*" during update. I think that selinux-policy could safely restore contexts of "/etc/passwd* /etc/group* /etc/*shadow*" in the install phase as well.

--- a/selinux-policy.spec
+++ b/selinux-policy.spec
@@ -306,7 +306,7 @@ if [ -e /etc/selinux/%2/.rebuild ]; then \
 fi; \
 [ "${SELINUXTYPE}" == "%2" ] && selinuxenabled && load_policy; \
 if [ %1 -eq 1 ]; then \
-   /sbin/restorecon -R /root /var/log /run 2> /dev/null; \
+   /sbin/restorecon -R /root /var/log /run /etc/passwd* /etc/group* /etc/*shadow* 2> /dev/null; \
 else \
 %relabel %2 \
 fi;

--- Additional comment from Miroslav Grepl on 2015-06-09 16:13:34 CEST ---

Yes, I agree.

Fixed in selinux-policy-3.13.1-128.1.fc22.

--- Additional comment from Fedora Update System on 2015-06-09 16:39:42 CEST ---

selinux-policy-3.13.1-128.1.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/selinux-policy-3.13.1-128.1.fc22

--- Additional comment from Petr Schindler on 2015-06-09 17:15:44 CEST ---

I tried minimal installation with new selinux-policy 3.13.1-128.1 and it booted correctly. This bug seems to be fixed.

--- Additional comment from Michal Schmidt on 2015-06-09 17:20:08 CEST ---

That's great! But we should let Bodhi move this through the Bugzilla states (ON_QA and eventually CLOSED).

--- Additional comment from Rob Foehl on 2015-06-09 17:41:55 CEST ---

Would it not also make sense to implement the suggestion from comment 13 that /etc/selinux/config be copied to the chroot early in the install process?

Relabeling the errant files after the fact is fine, but having broken SELinux state throughout the install process (enabled, but userspace doesn't think so) seems like it's asking for more trouble.