Reported upstream: SELinux: initialized (dev 0:1d, type nfs4), uses genfs_contexts Unable to handle kernel paging request at ffff8801e09b6000 RIP: [<ffffffff80261b9e>] copy_page+0x32/0xe4 PGD 1f4a067 PUD 2d52067 PMD 2e57067 PTE 0 Oops: 0000 [1] SMP last sysfs file: /devices/pci0000:00/0000:00:1c.0/0000:04:00.0/0000:05:00.0/irq CPU 0 Modules linked in: hfsplus loop tun xt_physdev netloop netbk blktap blkbk ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter i p_tables x_tables bridge ipmi_devintf ipmi_si ipmi_msghandler dell_rbu nfsd exportfs autofs4 hidp nfs lockd fscache nfs_acl rfcomm l2cap bluetooth rpcsec_gss_krb5 auth_rpcgss de s sunrpc ipv6 xfrm_nalgo crypto_api mptctl dm_multipath video sbs backlight i2c_ec i2c_core button battery asus_acpi ac parport_pc lp parport st joydev sr_mod ide_cd i5000_edac edac_mc pcspkr aic7xxx cdrom bnx2 serial_core serio_raw shpchp dm_snapshot dm_zero dm_mirror dm_mod mppVhba(U) usb_storage ata_piix libata mptsas mptscsih scsi_transport_sas mpt base aic79xx scsi_transport_spi megaraid_sas mppUpper(U) sg sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 6640, comm: smbd Tainted: G 2.6.18-92.1.17.el5xen #1 RIP: e030:[<ffffffff80261b9e>] [<ffffffff80261b9e>] copy_page+0x32/0xe4 RSP: e02b:ffff8801e09b59f8 EFLAGS: 00010202 RAX: 0000000000000246 RBX: 00007fffdaa61c38 RCX: 0000000000000025 RDX: 000000000000e02b RSI: ffff8801e09b5fe8 RDI: ffff880193182540 RBP: ffffffff885ba9a0 R08: 00007fffdaa63530 R09: 00007fffdaa62d30 R10: 0000000000000004 R11: 00002b0bd27f2135 R12: 0000000000000033 R13: ffff880193182000 R14: ffff880193182000 R15: ffff880156642fd5 FS: 00002b0bd3a1fb70(0000) GS:ffffffff805af000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 Process smbd (pid: 6640, threadinfo ffff8801e09b4000, task ffff880161dc2860) Stack: ffff88014f08bec0 ffff8801e09b5aa8 ffff880193182000 ffffffff80315001 ffff8801e09b5aa8 ffff88014f08bec0 ffffffff885ba9a0 ffff88014f08bec0 ffffffff885ba9a0 ffff8801e09b5aa8 Call Trace: [<ffffffff80315001>] selinux_sb_copy_data+0x23/0x1c5 [<ffffffff802d0101>] vfs_kern_mount+0x79/0x11a [<ffffffff8858dc70>] :nfs:nfs_do_submount+0xc0/0xdb [<ffffffff8858dd92>] :nfs:nfs_follow_mountpoint+0xe3/0x1d9 [<ffffffff8031295e>] avc_has_perm+0x43/0x55 [<ffffffff8858108e>] :nfs:nfs_access_get_cached+0xab/0xfa [<ffffffff80317c28>] selinux_inode_follow_link+0x5f/0x6a [<ffffffff8020a79a>] __link_path_walk+0xb71/0xf42 [<ffffffff8020eb09>] link_path_walk+0x5c/0xe5 [<ffffffff8858c99a>] :nfs:nfs_sync_inode_wait+0x83/0x1db [<ffffffff8020cf46>] do_path_lookup+0x270/0x2e8 [<ffffffff80212bcc>] getname+0x15b/0x1c1 [<ffffffff8022415b>] __user_walk_fd+0x37/0x4c [<ffffffff8022905f>] vfs_stat_fd+0x1b/0x4a [<ffffffff80223f01>] sys_newstat+0x19/0x31 [<ffffffff80260295>] tracesys+0x47/0xb2 [<ffffffff802602f5>] tracesys+0xa7/0xb2 We think the problem is that SELinux is trying to copy the page that holds the shrinkable mount args struct, but that struct is a stack allocation and copying the page that holds it can lead off into la-la land.
*** Bug 493145 has been marked as a duplicate of this bug. ***
Created attachment 337390 [details] potential fix
I've added that patch to the test kernels on my people.redhat.com page: http://people.redhat.com/jlayton/ If you have a non-critical place to test it (and an easy way to reproduce this), it would be helpful.
Had a peek at the core to make sure that everything matches up. Took me a little while realize that the core was actually from a -128.el5xen kernel: Here's the stack trace from crash. crash> bt PID: 31374 TASK: ffff8801ab45a040 CPU: 0 COMMAND: "smbd" #0 [ffff8800bda4f750] crash_kexec at ffffffff802a3b97 #1 [ffff8800bda4f810] __die at ffffffff80264349 #2 [ffff8800bda4f850] do_page_fault at ffffffff80266971 #3 [ffff8800bda4f940] error_exit at ffffffff8025f82b [exception RIP: copy_page+50] RIP: ffffffff80260b9e RSP: ffff8800bda4f9f8 RFLAGS: 00010202 RAX: 0000000000000246 RBX: 00007fff16e6f4a8 RCX: 0000000000000025 RDX: 000000000000e02b RSI: ffff8800bda4ffe8 RDI: ffff880121807540 RBP: ffffffff886439c0 R8: 00007fff16e70da0 R9: 00007fff16e705a0 R10: 0000000000000004 R11: 00002ab8963e6135 R12: 0000000000000033 R13: ffff880121807000 R14: ffff880121807000 R15: ffff880188844fc7 ORIG_RAX: ffffffffffffffff CS: e030 SS: e02b #4 [ffff8800bda4f9f0] __alloc_pages at ffffffff8020f477 #5 [ffff8800bda4fa60] vfs_kern_mount at ffffffff802d2197 #6 [ffff8800bda4faa0] nfs_do_submount at ffffffff88616cdc #7 [ffff8800bda4fb10] nfs_follow_mountpoint at ffffffff88616df7 #8 [ffff8800bda4fca0] __link_path_walk at ffffffff8020a7df #9 [ffff8800bda4fd10] link_path_walk at ffffffff8020eb6e #10 [ffff8800bda4fdd0] do_path_lookup at ffffffff8020cfa5 #11 [ffff8800bda4fe10] __user_walk_fd at ffffffff8022394e #12 [ffff8800bda4fe40] vfs_stat_fd at ffffffff8022889b #13 [ffff8800bda4fef0] sys_newstat at ffffffff802236f4 #14 [ffff8800bda4ff80] tracesys at ffffffff8025f2f9 (via system_call) RIP: 00002ab8963e6135 RSP: 00007fff16e6f4a8 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: ffffffff8025f2f9 RCX: ffffffffffffffff RDX: 00007fff16e70da0 RSI: 00007fff16e70da0 RDI: 00007fff16e705a0 RBP: 00002ab8b1a65c3b R8: fefefefefefefeff R9: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff16e70da0 R13: 00007fff16e70da0 R14: 0000000000000104 R15: 00002ab8b1a6eee4 ORIG_RAX: 0000000000000004 CS: 0033 SS: e02b ...so it looks like this is the problem that we think it is.
Tried the kernel on my experimental VM - works fine so far - but that's no surprise as the conditions in VM are very different to the production box where I experienced this problem. Couple of questions: 1. The bug is only evident on a system acting as a NFS server with SELinux enabled or permissive. Disabling SELinux completely should fix the problem, too. Am I right here? 2. When will RedHat release an official kernel update containing this fix? Seems to me quite critical one (the .128.el5xen kernel crashes very often due to this bug so it is hard to believe I will have to wait for U4 to get this fixed). Thanks, Ondrej
1) that is correct. The bug only presents itself when SELinux is NOT disabled. 2) I don't have an answer on when it would be released in a official kernel, you'll have to work with the support organization to find that out. Can you explain what your NFS server is doing which is out of the ordinary? Is this because of NFSv4 referals? are you using the nohide option? I guess it's possible either of those could tickle the problem, but I'd love to get an idea what you are doing so I can try to write an internal test case. I see a bug in the code and that's where your kernel came from, but I just can't figure out how to hit that bug the way you are.
My /etc/exports: /exports *(ro,no_root_squash,sync,fsid=0) /exports/int1 *(ro,no_root_squash,sync,nohide,fsid=10) hercules(rw,no_root_squash,sync) /exports/int1/applications *(rw,no_root_squash,sync,nohide) /exports/int2 *(ro,no_root_squash,sync,fsid=11) hercules(rw,no_root_squash,sync) /exports/int2/rhel4/snapshot *(rw,no_root_squash,sync,fsid=13) /exports/int2/centos5/snapshot *(rw,no_root_squash,sync,fsid=14) /exports/int2/ltsp-gutsy ts32(rw,no_root_squash,sync) /exports/int2/ltsp-hardy-test ts32(rw,no_root_squash,sync) /exports/ext1 *(rw,sync,fsid=15) deneb(no_root_squash) /exports/ext1/tmp *(rw,sync,fsid=16) ... so nothing extraordinary, as you can see. NFSv4 is used rarely on this system. Is the error in NFS client or server code? Jeff also mentioned that XEN could be also involved here. Is that possible? I have another machine (NFS client only, but heavily stressed) running .92.1.10.el5 with SELinux enabled (no XEN) which is rock stable. Ondrej
The patch here fixes a problem in the way that submounts are being handled in RHEL5. It's being done incorrectly in all kernels. I don't think xen is really "involved" per-se. It necessarily handles memory a little differently than "normal" kernels and it may just be that that's enough to make this bug cause a crash.
I just noticed you are saying 'server.' But the code paths in question in the 2 backtraces are nfs client code paths. Is the machine in question both a client and a server? My fix only affects NFS client mounting. Shouldn't have any affect on a server, so maybe we've got another problem..... I see you have 'nohide,' which is one of the areas I thought could be a problem. If the problem is what I think it is, when on the client you follow from say /exports/int1 to /exports/int1/applications you are going to perform another mount (this all happens transparently on the client) and it is here that we might hit a problem. I guess if the xen kernel changes how the stack frame is positioned (and especially what is on the page directly above the top of the stack frame) we could have a problem. If you can get this test kernel to panic, please provide the backtrace, but my guess is that you shouldn't have any problems with it.
/exports/int1 and below is a single filesystem so hence same FSID so no extra mount should be necessary (but I do not know, I am not NFS expert). But anyway there should be no requests to mount /exports/int1 - the automounter entry only contains key for /exports/int1/applications... Yes, the machine is client and server. The "nohide" option I introduced here for NFSv4 compatibility. Unfortunately I am running with SELinux disabled at the moment and enabling it again would mean relabeling the whole 3TB disk array attached. And besides, this is a production box, I am not going to risk my career here :-( So I am not going to put the test kernel into production - sorry.
I certainly wouldn't want you to put yourself in a bad position. You are safe from the bug I fixed in the test kernel by just using selinux=0. I'll take your exports and try to play with them to reproduce it here, but from what information I've seen and looking at the code I hope that I've already fixed the only bug in this area. Thank you for the report!
posted 5/4
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
in kernel-2.6.18-146.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please do NOT transition this bugzilla state to VERIFIED until our QE team has sent specific instructions indicating when to do so. However feel free to provide a comment indicating that this fix has been verified.
~~ Attention - RHEL 5.4 Beta Released! ~~ RHEL 5.4 Beta has been released! There should be a fix present in the Beta release that addresses this particular request. Please test and report back results here, at your earliest convenience. RHEL 5.4 General Availability release is just around the corner! If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity. Please do not flip the bug status to VERIFIED. Only post your verification results, and if available, update Verified field with the appropriate value. Questions can be posted to this bug or your customer or partner representative.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-1243.html