Bug 493144 - panic in SELinux code with shrinkable NFS mounts
panic in SELinux code with shrinkable NFS mounts
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.4
All Linux
low Severity medium
: rc
: ---
Assigned To: Eric Paris
Red Hat Kernel QE team
:
: 493145 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-31 15:12 EDT by Jeff Layton
Modified: 2011-12-29 23:39 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-02 04:57:04 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
potential fix (1.49 KB, patch)
2009-03-31 15:22 EDT, Eric Paris
no flags Details | Diff

  None (edit)
Description Jeff Layton 2009-03-31 15:12:21 EDT
Reported upstream:

SELinux: initialized (dev 0:1d, type nfs4), uses genfs_contexts
Unable to handle kernel paging request at ffff8801e09b6000 RIP:
 [<ffffffff80261b9e>] copy_page+0x32/0xe4
PGD 1f4a067 PUD 2d52067 PMD 2e57067 PTE 0
Oops: 0000 [1] SMP
last sysfs file: 
/devices/pci0000:00/0000:00:1c.0/0000:04:00.0/0000:05:00.0/irq
CPU 0
Modules linked in: hfsplus loop tun xt_physdev netloop netbk blktap 
blkbk ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink 
ipt_REJECT xt_tcpudp iptable_filter i
p_tables x_tables bridge ipmi_devintf ipmi_si ipmi_msghandler dell_rbu 
nfsd exportfs autofs4 hidp nfs lockd fscache nfs_acl rfcomm l2cap 
bluetooth rpcsec_gss_krb5 auth_rpcgss de
s sunrpc ipv6 xfrm_nalgo crypto_api mptctl dm_multipath video sbs 
backlight i2c_ec i2c_core button battery asus_acpi ac parport_pc lp 
parport st joydev sr_mod ide_cd i5000_edac
edac_mc pcspkr aic7xxx cdrom bnx2 serial_core serio_raw shpchp 
dm_snapshot dm_zero dm_mirror dm_mod mppVhba(U) usb_storage ata_piix 
libata mptsas mptscsih scsi_transport_sas mpt
base aic79xx scsi_transport_spi megaraid_sas mppUpper(U) sg sd_mod 
scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 6640, comm: smbd Tainted: G      2.6.18-92.1.17.el5xen #1
RIP: e030:[<ffffffff80261b9e>]  [<ffffffff80261b9e>] copy_page+0x32/0xe4
RSP: e02b:ffff8801e09b59f8  EFLAGS: 00010202
RAX: 0000000000000246 RBX: 00007fffdaa61c38 RCX: 0000000000000025
RDX: 000000000000e02b RSI: ffff8801e09b5fe8 RDI: ffff880193182540
RBP: ffffffff885ba9a0 R08: 00007fffdaa63530 R09: 00007fffdaa62d30
R10: 0000000000000004 R11: 00002b0bd27f2135 R12: 0000000000000033
R13: ffff880193182000 R14: ffff880193182000 R15: ffff880156642fd5
FS:  00002b0bd3a1fb70(0000) GS:ffffffff805af000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process smbd (pid: 6640, threadinfo ffff8801e09b4000, task ffff880161dc2860)
Stack:  ffff88014f08bec0  ffff8801e09b5aa8  ffff880193182000  
ffffffff80315001
 ffff8801e09b5aa8  ffff88014f08bec0  ffffffff885ba9a0  ffff88014f08bec0
 ffffffff885ba9a0  ffff8801e09b5aa8
Call Trace:
 [<ffffffff80315001>] selinux_sb_copy_data+0x23/0x1c5
 [<ffffffff802d0101>] vfs_kern_mount+0x79/0x11a
 [<ffffffff8858dc70>] :nfs:nfs_do_submount+0xc0/0xdb
 [<ffffffff8858dd92>] :nfs:nfs_follow_mountpoint+0xe3/0x1d9
 [<ffffffff8031295e>] avc_has_perm+0x43/0x55
 [<ffffffff8858108e>] :nfs:nfs_access_get_cached+0xab/0xfa
 [<ffffffff80317c28>] selinux_inode_follow_link+0x5f/0x6a
 [<ffffffff8020a79a>] __link_path_walk+0xb71/0xf42
 [<ffffffff8020eb09>] link_path_walk+0x5c/0xe5
 [<ffffffff8858c99a>] :nfs:nfs_sync_inode_wait+0x83/0x1db
 [<ffffffff8020cf46>] do_path_lookup+0x270/0x2e8
 [<ffffffff80212bcc>] getname+0x15b/0x1c1
 [<ffffffff8022415b>] __user_walk_fd+0x37/0x4c
 [<ffffffff8022905f>] vfs_stat_fd+0x1b/0x4a
 [<ffffffff80223f01>] sys_newstat+0x19/0x31
 [<ffffffff80260295>] tracesys+0x47/0xb2
 [<ffffffff802602f5>] tracesys+0xa7/0xb2

We think the problem is that SELinux is trying to copy the page that holds the shrinkable mount args struct, but that struct is a stack allocation and copying the page that holds it can lead off into la-la land.
Comment 1 Jeff Layton 2009-03-31 15:19:59 EDT
*** Bug 493145 has been marked as a duplicate of this bug. ***
Comment 2 Eric Paris 2009-03-31 15:22:12 EDT
Created attachment 337390 [details]
potential fix
Comment 3 Jeff Layton 2009-03-31 19:46:09 EDT
I've added that patch to the test kernels on my people.redhat.com page:

http://people.redhat.com/jlayton/

If you have a non-critical place to test it (and an easy way to reproduce this), it would be helpful.
Comment 4 Jeff Layton 2009-04-01 09:45:53 EDT
Had a peek at the core to make sure that everything matches up. Took me a little while realize that the core was actually from a -128.el5xen kernel:

Here's the stack trace from crash.

crash> bt
PID: 31374  TASK: ffff8801ab45a040  CPU: 0   COMMAND: "smbd"
 #0 [ffff8800bda4f750] crash_kexec at ffffffff802a3b97
 #1 [ffff8800bda4f810] __die at ffffffff80264349
 #2 [ffff8800bda4f850] do_page_fault at ffffffff80266971
 #3 [ffff8800bda4f940] error_exit at ffffffff8025f82b
    [exception RIP: copy_page+50]
    RIP: ffffffff80260b9e  RSP: ffff8800bda4f9f8  RFLAGS: 00010202
    RAX: 0000000000000246  RBX: 00007fff16e6f4a8  RCX: 0000000000000025
    RDX: 000000000000e02b  RSI: ffff8800bda4ffe8  RDI: ffff880121807540
    RBP: ffffffff886439c0   R8: 00007fff16e70da0   R9: 00007fff16e705a0
    R10: 0000000000000004  R11: 00002ab8963e6135  R12: 0000000000000033
    R13: ffff880121807000  R14: ffff880121807000  R15: ffff880188844fc7
    ORIG_RAX: ffffffffffffffff  CS: e030  SS: e02b
 #4 [ffff8800bda4f9f0] __alloc_pages at ffffffff8020f477
 #5 [ffff8800bda4fa60] vfs_kern_mount at ffffffff802d2197
 #6 [ffff8800bda4faa0] nfs_do_submount at ffffffff88616cdc
 #7 [ffff8800bda4fb10] nfs_follow_mountpoint at ffffffff88616df7
 #8 [ffff8800bda4fca0] __link_path_walk at ffffffff8020a7df
 #9 [ffff8800bda4fd10] link_path_walk at ffffffff8020eb6e
#10 [ffff8800bda4fdd0] do_path_lookup at ffffffff8020cfa5
#11 [ffff8800bda4fe10] __user_walk_fd at ffffffff8022394e
#12 [ffff8800bda4fe40] vfs_stat_fd at ffffffff8022889b
#13 [ffff8800bda4fef0] sys_newstat at ffffffff802236f4
#14 [ffff8800bda4ff80] tracesys at ffffffff8025f2f9 (via system_call)
    RIP: 00002ab8963e6135  RSP: 00007fff16e6f4a8  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: ffffffff8025f2f9  RCX: ffffffffffffffff
    RDX: 00007fff16e70da0  RSI: 00007fff16e70da0  RDI: 00007fff16e705a0
    RBP: 00002ab8b1a65c3b   R8: fefefefefefefeff   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000246  R12: 00007fff16e70da0
    R13: 00007fff16e70da0  R14: 0000000000000104  R15: 00002ab8b1a6eee4
    ORIG_RAX: 0000000000000004  CS: 0033  SS: e02b


...so it looks like this is the problem that we think it is.
Comment 5 Ondrej Valousek 2009-04-01 09:50:13 EDT
Tried the kernel on my experimental VM - works fine so far - but that's no
surprise as the conditions in VM are very different to the production box where
I experienced this problem.

Couple of questions:
1. The bug is only evident on a system acting as a NFS server with SELinux
enabled or permissive. Disabling SELinux completely should fix the problem,
too. Am I right here?
2. When will RedHat release an official kernel update containing this fix?
Seems to me quite critical one (the .128.el5xen kernel crashes very often due to this
bug so it is hard to believe I will have to wait for U4 to get this fixed).

Thanks,
Ondrej
Comment 6 Eric Paris 2009-04-01 09:56:49 EDT
1) that is correct.  The bug only presents itself when SELinux is NOT disabled.

2) I don't have an answer on when it would be released in a official kernel, you'll have to work with the support organization to find that out.

Can you explain what your NFS server is doing which is out of the ordinary?  Is this because of NFSv4 referals?  are you using the nohide option?  I guess it's possible either of those could tickle the problem, but I'd love to get an idea what you are doing so I can try to write an internal test case.

I see a bug in the code and that's where your kernel came from, but I just can't figure out how to hit that bug the way you are.
Comment 7 Ondrej Valousek 2009-04-01 10:11:20 EDT
My /etc/exports:
/exports                        *(ro,no_root_squash,sync,fsid=0)
/exports/int1                   *(ro,no_root_squash,sync,nohide,fsid=10) hercules(rw,no_root_squash,sync)
/exports/int1/applications      *(rw,no_root_squash,sync,nohide)
/exports/int2                   *(ro,no_root_squash,sync,fsid=11) hercules(rw,no_root_squash,sync)
/exports/int2/rhel4/snapshot    *(rw,no_root_squash,sync,fsid=13)
/exports/int2/centos5/snapshot  *(rw,no_root_squash,sync,fsid=14)
/exports/int2/ltsp-gutsy        ts32(rw,no_root_squash,sync)
/exports/int2/ltsp-hardy-test   ts32(rw,no_root_squash,sync)

/exports/ext1                   *(rw,sync,fsid=15) deneb(no_root_squash)
/exports/ext1/tmp               *(rw,sync,fsid=16)

... so nothing extraordinary, as you can see. NFSv4 is used rarely on this system.
Is the error in NFS client or server code?
Jeff also mentioned that XEN could be also involved here. Is that possible?
I have another machine (NFS client only, but heavily stressed) running .92.1.10.el5 with SELinux enabled (no XEN) which is rock stable.
Ondrej
Comment 8 Jeff Layton 2009-04-01 10:20:10 EDT
The patch here fixes a problem in the way that submounts are being handled in RHEL5. It's being done incorrectly in all kernels.

I don't think xen is really "involved" per-se. It necessarily handles memory a little differently than "normal" kernels and it may just be that that's enough to make this bug cause a crash.
Comment 9 Eric Paris 2009-04-01 10:40:36 EDT
I just noticed you are saying 'server.'  But the code paths in question in the 2 backtraces are nfs client code paths.  Is the machine in question both a client and a server?  My fix only affects NFS client mounting.  Shouldn't have any affect on a server, so maybe we've got another problem.....

I see you have 'nohide,' which is one of the areas I thought could be a problem.  If the problem is what I think it is, when on the client you follow from say /exports/int1 to /exports/int1/applications you are going to perform another mount (this all happens transparently on the client) and it is here that we might hit a problem.

I guess if the xen kernel changes how the stack frame is positioned (and especially what is on the page directly above the top of the stack frame) we could have a problem.

If you can get this test kernel to panic, please provide the backtrace, but my guess is that you shouldn't have any problems with it.
Comment 10 Ondrej Valousek 2009-04-01 16:12:20 EDT
/exports/int1 and below is a single filesystem so hence same FSID so no extra mount should be necessary (but I do not know, I am not NFS expert). But anyway there should be no requests to mount /exports/int1 - the automounter entry only contains key for /exports/int1/applications...

Yes, the machine is client and server.
The "nohide" option I introduced here for NFSv4 compatibility.

Unfortunately I am running with SELinux disabled at the moment and enabling it again would mean relabeling the whole 3TB disk array attached. And besides, this is a production box, I am not going to risk my career here :-(
So I am not going to put the test kernel into production - sorry.
Comment 11 Eric Paris 2009-04-01 16:23:07 EDT
I certainly wouldn't want you to put yourself in a bad position.  You are safe from the bug I fixed in the test kernel by just using selinux=0.  I'll take your exports and try to play with them to reproduce it here, but from what information I've seen and looking at the code I hope that I've already fixed the only bug in this area.

Thank you for the report!
Comment 12 Eric Paris 2009-05-04 17:46:42 EDT
posted 5/4
Comment 13 RHEL Product and Program Management 2009-05-04 18:09:24 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 14 Don Zickus 2009-05-12 13:41:18 EDT
in kernel-2.6.18-146.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please do NOT transition this bugzilla state to VERIFIED until our QE team
has sent specific instructions indicating when to do so.  However feel free
to provide a comment indicating that this fix has been verified.
Comment 16 Chris Ward 2009-07-03 14:28:19 EDT
~~ Attention - RHEL 5.4 Beta Released! ~~

RHEL 5.4 Beta has been released! There should be a fix present in the Beta release that addresses this particular request. Please test and report back results here, at your earliest convenience. RHEL 5.4 General Availability release is just around the corner!

If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity.

Please do not flip the bug status to VERIFIED. Only post your verification results, and if available, update Verified field with the appropriate value.

Questions can be posted to this bug or your customer or partner representative.
Comment 18 errata-xmlrpc 2009-09-02 04:57:04 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1243.html

Note You need to log in before you can comment on or make changes to this bug.