Bug 519653

Summary: (initrd-generic) selinux was disabled via kernel update
Product: [Fedora] Fedora Reporter: Jim Meyering <meyering>
Component: policycoreutilsAssignee: Daniel Walsh <dwalsh>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: rawhideCC: dwalsh, eparis, harald, hdegoede, mgrepl, pjones, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-09 17:53:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
list of installed packages
none
hi Harald, here's that the output of lsinitrd /boot/initrd-generic-2.6.31-0.180.rc7.git4.fc12.x86_64.img (note, this is the rebuilt, working one)
none
requested dmesg output
none
dmesg, after booting into result of dracut-001-7 none

Description Jim Meyering 2009-08-27 10:14:19 UTC
[Dan, I've chosen selinux as the component, but it's probably something else.
 You're better placed to say which, and reassign, if needed.  Thanks.  ]

Description of problem: I have a bare-metal system that runs rawhide with SELinux in enforcing mode, and I update it pretty frequently.  Yesterday I noticed that it was failing a test that it normally passed.  The cause: SELinux was turned off in spite of my config setting of SELINUX=enforcing, and nothing in grub.conf that would imply enforcing=0.  I.e., getenforce reported Disabled, selinuxenabled would exit with status "1".  dmesg output suggested that the kernel was not initializing SELinux, while earlier kernels definitely did do that. Here's one failing kernel version for which I have dmesg output:
2.6.31-0.167.rc6.git6.fc12.x86_64.  Yesterday I was running this one: 
2.6.31-0.174.rc7.git2.fc12.x86_64.

This shows that the kernel did not initialize SELinux:
    $ grep -i selinux /tmp/old-dmesg
    SELinux:  Initializing.
    SELinux:  Starting in permissive mode
    SELinux:  Registering netfilter hooks

At Dan Walsh's suggestion, I rebuilt initrd like this:

 mkinitrd -f \
      /boot/initrd-generic-2.6.31-0.174.rc7.git2.fc12.x86_64.img \
                           2.6.31-0.174.rc7.git2.fc12.x86_64

Then I rebooted, waited a few minutes for a relabel, and when it came up, SELinux was once again in enforcing mode.

Version-Release number of selected component (if applicable): kernel 2.6.31-0.174.rc7.git2.fc12.x86_64


How reproducible: hard to reproduce, probably

Steps to Reproduce:
1. enable SELinux enforcing months ago, run "yum upgrade" starting a few kernels ago, and see that SELinux becomes disabled upon reboot into new kernel.
2.
3.
  
Actual results: SELinux was disabled


Expected results: SELinux remains in Enforcing mode


Additional info:

Comment 1 Jim Meyering 2009-08-28 13:08:17 UTC
minutes ago I updated to kernel 2.6.31-0.180.rc7.git4.fc12.x86_64, and rebooted.

When the system came back, I found that SELinux is once again disabled.
getenforce reports "Disabled".

Comment 2 Jim Meyering 2009-08-28 13:14:45 UTC
Remove question mark from subject.  it's no longer a question.

Comment 3 Warren Togami 2009-08-28 13:41:16 UTC
[warren@newcaprica ~]$ uname -a
Linux newcaprica 2.6.31-0.180.rc7.git4.fc12.x86_64 #1 SMP Wed Aug 26 16:15:07 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[warren@newcaprica ~]$ getenforce 
Enforcing

This isn't happening for me.

Comment 4 Jim Meyering 2009-08-28 14:19:07 UTC
FYI, I rebuilt with mkinitrd, as above, rebooted, and now all's well here.

Comment 5 Jim Meyering 2009-08-28 14:20:50 UTC
Warren, want to compare /etc settings or installed package sets?

Comment 6 Jim Meyering 2009-08-28 14:24:40 UTC
Created attachment 359071 [details]
list of installed packages

Comment 7 Harald Hoyer 2009-08-28 14:32:30 UTC
please attach the output of

# lsinitrd <working initrd>

Comment 8 Jim Meyering 2009-08-28 14:47:06 UTC
Created attachment 359078 [details]
hi Harald, here's that the output of lsinitrd /boot/initrd-generic-2.6.31-0.180.rc7.git4.fc12.x86_64.img (note, this is the rebuilt, working one)

Comment 9 Harald Hoyer 2009-08-28 15:21:09 UTC
do you have /usr/sbin/load_policy on your system?

Comment 10 Harald Hoyer 2009-08-28 15:24:49 UTC
dracut does:

chroot /sysroot /usr/sbin/load_policy -i

before it switches to the new root.

Comment 11 Jim Meyering 2009-08-28 15:34:19 UTC
Harald,

Yes, it has that file:

$ lsattr /usr/sbin/load_policy
-------------e- /usr/sbin/load_policy

Comment 12 Jim Meyering 2009-08-28 19:53:49 UTC
Could it be that some file it requires is missing?

$ ldd /usr/sbin/load_policy
        linux-vdso.so.1 =>  (0x00007fffb13df000)
        libsepol.so.1 => /lib64/libsepol.so.1 (0x00007f8027aa8000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f802788a000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f802750a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f8027306000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f80270e9000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8027ce4000)

Comment 13 Jim Meyering 2009-09-04 05:10:06 UTC
FYI, the problem is still here, with latest rawhide kernel, mkinitrd, etc:
Rebooting into new kernel turns off SELinux:

  $ uname -r; getenforce
  2.6.31-0.199.rc8.git2.fc12.x86_64
  Disabled

any suggestions?

Comment 14 Hans de Goede 2009-09-04 06:13:27 UTC
Jim, what does "cat /etc/sysconfig/selinux" say ?

Comment 15 Jim Meyering 2009-09-04 06:41:02 UTC
Hi Hans,

As mentioned in the description, SELINUX=enforcing.
Also SELINUXTYPE=targeted, fwiw.

And nothing in grub options explains the difference either.

To work around the problem, I did this:

v=$(uname -r); mkinitrd -f /boot/initrd-generic-$v.img $v

then rebooted.  That put it back in enforcing mode.

Comment 16 Hans de Goede 2009-09-04 07:18:33 UTC
Jim,

Can you install a newer rawhide kernel, and then add: rdbreak
To the kernel cmdline, that should give you a shell before
the switchroot takes place, then from that shell do:

chroot /sysroot /usr/sbin/load_policy -i

And see if you get any error messages, after this you can exit the shell and the boot will continue. It would also be good to know if doing this manually fixed
the not being in enforcing mode issue.

Comment 17 Jim Meyering 2009-09-04 07:28:42 UTC
Hans,

Yes, I can install a newer rawhide kernel.  Just tell me which.
Then I'll perform those tests.

Comment 18 Hans de Goede 2009-09-04 07:32:15 UTC
Jim,

Any newer kernel will do, I just want you to get an original dracut initrd again,
as you've overwritten it with your mkinitrd -f workaround.

Next time please use a different name with mkinitrd and edit grub.conf to point to the new one, this way its easy to switch between the 2.

Comment 19 Jim Meyering 2009-09-04 08:50:14 UTC
Oh, there's no problem, since I made a backup first.  But even if I hadn't, it's easy to remove and reinstall.  Will reboot and see, shortly.

Comment 20 Jim Meyering 2009-09-04 13:56:40 UTC
Hans,

Rebooted into the rawhide .199 kernel with " rdbreak" suffix on the boot line.
The result was not usable.  Here's what I saw:

error: unexpectedly disconnected from boot status daemon
            Dropping to debug shell
                                   sh: can't access tty: job control turned off
#usb 3-3: new full speed...
usb 3-3: device not accepting address 5, error -62
...15 lines of usb device stuff...
generic-usb 0003:...

C-Alt-N didn't do anything.
So I power cycled it.

Comment 21 Hans de Goede 2009-09-04 14:20:45 UTC
Hmm,

Did you try pressing enter to get a prompt, maybe the dmesg output overwrote the prompt ?

Also can you try again with:
rdbreak nomodeset 1

Maybe plymouth is getting in the way.

Comment 22 Harald Hoyer 2009-09-04 14:31:55 UTC
(In reply to comment #21)
> Maybe plymouth is getting in the way.  

remove "rhgb"

Comment 23 Jim Meyering 2009-09-04 15:46:28 UTC
bummer.  it stopped with same symptoms as above.
I tried first simply replacing rhgb with rdbreak,
and then doing s/rhgb.*/rdbreak/

Comment 24 Jim Meyering 2009-09-06 20:53:30 UTC
FYI, SELinux is disabled via very latest kernel in rawhide.

Comment 25 Harald Hoyer 2009-09-07 09:06:01 UTC
"rdshell rdbreak nomodeset 1"

Comment 26 Jim Meyering 2009-09-08 07:27:40 UTC
Hi Harald, thanks for the tip.
With that, the 20+ USB lines made the useful info scroll off the top
(yeah, still no serial).  But regardless, the console still stopped, and did not respond to any input.  Removing nomodeset gave the same output as in #20, but with one additional line:

     Break before switch_root

which came just before the "error: unexpectedly disconnected from boot status daemon" line.

Finally, I tried just "rdshell 1", and that did it.  I got a shell.

Taking Hans' suggestion from #16, I was going to run this
  chroot /sysroot /usr/sbin/load_policy -i
but there is no /sysroot directory.

I can imagine that would cause trouble ;-)

Comment 27 Jim Meyering 2009-09-08 07:29:24 UTC
in case it matters, root's shell is /bin/zsh

Comment 28 Harald Hoyer 2009-09-08 09:09:53 UTC
err... it seems you are past switch_root already.. anyway, I can't reproduce your selinux problem. :-/

Comment 29 Jim Meyering 2009-09-08 10:48:43 UTC
so something about rdbreak isn't working.  could it be that I have a GPT partition table?

any other hints on how to pursue it?

If I'm the only one affected, I can just forget about it, nuke this system and install an F12 snapshot directly.  Up to you.

Comment 30 Harald Hoyer 2009-09-08 14:38:25 UTC
ok, please install dracut-001-6.gitf5c4374d.fc12 from

http://koji.fedoraproject.org/koji/buildinfo?buildID=131035

on your system and recreate initrd-generic with dracut (like with mkinitrd)

# dracut -f \
      /boot/initrd-generic-2.6.31-0.174.rc7.git2.fc12.x86_64.img \
                           2.6.31-0.174.rc7.git2.fc12.x86_64

Comment 31 Jim Meyering 2009-09-08 15:49:06 UTC
done.  rebooted into it, and it turned off SELinux, but I suppose that's still to be expected, for me.

BTW, I installed only dracut-001...  and to do that, I had to first install dash.

I'm about to reboot into it again, but with rdbreak this time.

Comment 32 Jim Meyering 2009-09-08 16:01:49 UTC
I still get a shell via "rdshell 1", and now /sysroot does exist.
But it's empty.

Comment 33 Harald Hoyer 2009-09-09 11:47:34 UTC
"1" is only for the runlevel in real root.

no "rhgb" and "rdinfo rdbreak rdshell" should really really give you a shell and something in /sysroot ... otherwise you would not be able to boot.

Here is a new version to try..

https://koji.fedoraproject.org/koji/buildinfo?buildID=131151

please also attach the output of 

# dmesg

if you boot from this one.

Comment 34 Harald Hoyer 2009-09-09 11:52:35 UTC
Also note that the shell you are dropped into does not take any <ctrl> commands and there is no <tab> completion.

You might generate the image with the debug module, so that you have a bash instead of dash.

# dracut -a debug /boot/dracut-$(uname -r).img $(uname -r)

Comment 35 Jim Meyering 2009-09-09 12:06:15 UTC
Thanks, Harald.

Local changes:

  rpm -ivh /t/dracut-001-7.git71094bee.fc12.src.rpm
  v=$(uname -r); f=/boot/initrd-generic-$v.img; backup $f
  dracut -a debug -f $f $v

I'm preparing to reboot into this:

title Fedora (2.6.31-0.204.rc9.fc12.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.31-0.204.rc9.fc12.x86_64 ro root=UUID=c5e669eb-0b26-4f27-a590-94558a12e9fc rdinfo rdbreak rdshell
        initrd /initrd-generic-2.6.31-0.204.rc9.fc12.x86_64.img

Comment 36 Jim Meyering 2009-09-09 12:50:57 UTC
rats.  same story.  I see the prompt (above 20 lines of USB diags),
but keyboard is unresponsive.

I'm using USB kbd and mouse.
Note that the USB lines mentioned in comment #21 are regarding precisely those devices, so maybe they're just a little too late?  I also tried with a conventional keyboard.  no difference.

dmesg output coming up

Comment 37 Jim Meyering 2009-09-09 12:55:17 UTC
Created attachment 360205 [details]
requested dmesg output

Comment 38 Jim Meyering 2009-09-09 14:03:35 UTC
Created attachment 360218 [details]
dmesg, after booting into result of dracut-001-7

Comment 39 Harald Hoyer 2009-09-09 15:20:22 UTC
the selinux part is missing completely.. very strange.

next step.. boot with

"rdshell rdbreak rdinitdebug rdinfo"

and please take a photo of the last pages of the output

Comment 40 Harald Hoyer 2009-09-09 17:53:59 UTC
ok, this is because load_policy is in /usr/sbin and Jim has a separate /usr partition, which is of course not mounted at that point of time.

*** This bug has been marked as a duplicate of bug 521932 ***

Comment 41 Jim Meyering 2009-09-09 17:57:02 UTC
FYI, after many iterations, we (mostly Harald) finally figured out the cause:
I have /usr on a separate partition, and 
/usr/share/dracut/modules.d/99base/selinux-loadpolicy.sh
depended on /usr/sbin/load_policy being availble before /usr was mounted.

Temporary work-around:

cp -a /usr/sbin/load_policy /sbin

That got me past the initial problem.
Further along, I saw some AVCs and boot failed.
rebooting with enforcing=0 got me in.
details tomorrow