Bug 591813 - [abrt] crash in kernel: SELinux: WARNING: inside open_file_to_av with unknown mode:600
[abrt] crash in kernel: SELinux: WARNING: inside open_file_to_av with unknown...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.0
x86_64 Linux
high Severity high
: beta
: ---
Assigned To: Eric Paris
Red Hat Kernel QE team
abrt_hash:2069189119
:
Depends On:
Blocks: 519823 584207 595673
  Show dependency treegraph
 
Reported: 2010-05-13 04:42 EDT by Eduard Benes
Modified: 2010-07-02 15:20 EDT (History)
7 users (show)

See Also:
Fixed In Version: kernel-2.6.32-28.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 595673 (view as bug list)
Environment:
Last Closed: 2010-07-02 15:20:52 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Test program to poke small parts of the system like filecap -a (1.17 KB, text/plain)
2010-05-14 11:36 EDT, Eric Paris
no flags Details

  None (edit)
Description Eduard Benes 2010-05-13 04:42:17 EDT
abrt 1.0.7 detected a crash.

architecture: x86_64
cmdline: not_applicable
comment: System/kernel crash caused by running 'filecap -a' as a root
component: kernel
executable: kernel
kernel: undefined
package: kernel
reason: SELinux: WARNING: inside open_file_to_av with unknown mode:600
release: Red Hat Enterprise Linux release 6.0 Beta (Santiago)

kerneloops
-----
SELinux: WARNING: inside open_file_to_av with unknown mode:600
SELinux: WARNING: inside open_file_to_av with unknown mode:600
 ...
SELinux: WARNING: inside open_file_to_av with unknown mode:600

<snip>
May 13 10:32:48 godot kernel: SELinux: WARNING: inside open_file_to_av with unknown mode:600
May 13 10:32:48 godot kernel: SELinux: WARNING: inside open_file_to_av with unknown mode:600
May 13 10:32:48 godot kernel: SELinux: WARNING: inside open_file_to_av with unknown mode:600
May 13 10:32:48 godot kernel: SELinux: WARNING: inside open_file_to_av with unknown mode:600
May 13 10:32:48 godot kernel: iTCO_wdt: Unexpected close, not stopping watchdog!
May 13 10:32:48 godot kernel: PPP generic driver version 2.4.2
May 13 10:32:48 godot kernel: tun: Universal TUN/TAP device driver, 1.6
May 13 10:32:48 godot kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
May 13 10:32:48 godot kernel: iTCO_wdt: Unexpected close, not stopping watchdog!
</snip>

How to reproduce
-----
Note: Save all your work. This will most likely crash your system :)
1. install libcap-ng-utils-0.6.2-5.el6.x86_64
2. as a root run command: # filecap -a
3. wait :)
4. watch your system to crash, and you can power-cycle it

selinux-policy-3.7.19-3.el6.noarch
libcap-ng-utils-0.6.2-5.el6.x86_64
kernel-2.6.32-22.el6.x86_64
Comment 1 Eric Paris 2010-05-13 07:41:39 EDT
There is very little chance this is an 'SELinux' bug.  The inode in question somehow does not have correct inode->i_mode flags.  The message:

unknown mode:600

means that the inode->i_mode == 0600  (octal)

Which means it was clearly not created properly (hint, it should have bits saying if this is a file, dir, chr, blk, sock, lnk, etc)

As long as it is as easy to reproduce as you say I'll see if we can't make the kernel dump a stack trace to figure out who is created these broken inodes....

-Eric
Comment 3 Eric Paris 2010-05-13 18:04:24 EDT
It was anon_inode:[signalfd] which sets inode->i_mode = S_IRUSR | S_IWUSR;  I added S_IFREG in there and it was able to run filecap -a without the SELinux spam.  I did see some console output:

PPP generic driver version 2.4.2
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
PM: Marking nosave pages: 000000000009f000 - 0000000000100000
PM: Marking nosave pages: 00000000bfff0000 - 0000000100000000
PM: Basic memory bitmaps created
PM: Basic memory bitmaps freed
PM: Marking nosave pages: 000000000009f000 - 0000000000100000
PM: Marking nosave pages: 00000000bfff0000 - 0000000100000000
PM: Basic memory bitmaps created
PM: Basic memory bitmaps freed

Which I assume is the result of opening some file in /proc or /sys and closing it.

If you're machine has any problems other than just the console spam please open a new BZ, and I'm going to use this one just to silence the SELinux console spam.
Comment 4 Eduard Benes 2010-05-14 04:38:43 EDT
> If you're machine has any problems other than just the console spam please open
> a new BZ, and I'm going to use this one just to silence the SELinux console
> spam.    

Eric, if I let it run long enough the system crashed *every time* and I had to powercycle it as mentioned in the bug report. Tested on x84_64/bare and i386/kvm.
Logged messages are just a cosmetic issue.
Comment 5 Eric Paris 2010-05-14 10:28:39 EDT
I've been able to run filecap -a to completion 4 or 5 times after silencing the SELinux spam.  I'll attach the patch in question and send it towards upstream and rhkernel-list.  If you can reproduce with the patch hopefully we can open a new bug with information to point towards your problem.  Namely we need either the panic/oops backtrace or we need to make sure the watchdogs (nmi_watchdog) are on to catch hard lockups, or you need to collect sysrq-t before the reboot.  So we can figure out who should be looking for a problem.
Comment 6 Eric Paris 2010-05-14 11:36:32 EDT
Created attachment 414090 [details]
Test program to poke small parts of the system like filecap -a

This test program will emulate the behaviour of filecap -a only in a more directed manor.  You can run

./test /proc

to just troll through /proc (which should generate the SELinux SPAM)

you can run i through /sys and maybe hit on other problems.....
Comment 7 Eric Paris 2010-05-14 16:13:52 EDT
moving this bug to POST as I posted a fix for the anon_inode + SELinux issue.  If we can reproduce a crash and collect some info about it, lets open a new bug/clone this bug for that new issue.
Comment 8 Eduard Benes 2010-05-19 14:04:22 EDT
(In reply to comment #7)
> moving this bug to POST as I posted a fix for the anon_inode + SELinux issue. 
> If we can reproduce a crash and collect some info about it, lets open a new
> bug/clone this bug for that new issue.    

Tried to collect some info with help of one our kernel qe guys, but without any success (bare metal). If it helps, we were able to reproduce the crash also with disabled SELinux. I'll try to reproduce it on a virt guest and try collect some info. Any ideas/suggestions are welcomed :)
Comment 9 Eric Paris 2010-05-19 15:05:10 EDT
Did you do it using filecap -a or using my program?

I would suggest you can make my program print the file it is about to deal with.  Then run it against /dev.  If that works run it against /sys.  I'm guessing it is one of those filesystems that you will run into your crash.  It might narrow it down to a particular file.

If that doesn't work you can run filecap under strace and you'll possibly learn the file in question (or at least a file close to it)......
Comment 10 Aristeu Rozanski 2010-05-20 17:59:13 EDT
Patch(es) available on kernel-2.6.32-28.el6
Comment 13 Eduard Benes 2010-05-24 13:14:46 EDT
Tested on 2.6.32-28.el6.x86_64 and the system still crashes though "SELinux: Warning" messages are gone. Eric do you want me to open a separate bug for the "crash" part?

Before the crash following messages appeared in /var/log/messages:

May 24 11:13:53 godot kernel: iTCO_wdt: Unexpected close, not stopping watchdog!
May 24 11:13:53 godot kernel: PPP generic driver version 2.4.2
May 24 11:13:53 godot kernel: tun: Universal TUN/TAP device driver, 1.6
May 24 11:13:53 godot kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
May 24 11:13:53 godot kernel: iTCO_wdt: Unexpected close, not stopping watchdog!
Comment 14 Eric Paris 2010-05-24 13:28:15 EDT
I say we let this BZ just be for the SELinux printk spam as a patch was committed to fix that problem.  I'd suggest we open a new BZ for the crash, hopefully it can also contain more details on what is crashing the kernel by modifying the program in comment #6 with the changes and test suggested in comment #9
Comment 15 Eduard Benes 2010-05-28 05:56:01 EDT
Verified on 2.6.32-28.el6.x86_64. SELinux warning messages / spam are now fixed. 
The kernel crash part opened as a separate bug 595673. You may close this bug, if no more testing is required.
Comment 16 releng-rhel@redhat.com 2010-07-02 15:20:52 EDT
Red Hat Enterprise Linux Beta 2 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.