Bug 493144
| Summary: | panic in SELinux code with shrinkable NFS mounts | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Jeff Layton <jlayton> | ||||
| Component: | kernel | Assignee: | Eric Paris <eparis> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Red Hat Kernel QE team <kernel-qe> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | 5.4 | CC: | cward, dzickus, jiali, ondrejv, pasteur, staubach, steved, tao | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2009-09-02 08:57:04 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Jeff Layton
2009-03-31 19:12:21 UTC
*** Bug 493145 has been marked as a duplicate of this bug. *** Created attachment 337390 [details]
potential fix
I've added that patch to the test kernels on my people.redhat.com page: http://people.redhat.com/jlayton/ If you have a non-critical place to test it (and an easy way to reproduce this), it would be helpful. Had a peek at the core to make sure that everything matches up. Took me a little while realize that the core was actually from a -128.el5xen kernel:
Here's the stack trace from crash.
crash> bt
PID: 31374 TASK: ffff8801ab45a040 CPU: 0 COMMAND: "smbd"
#0 [ffff8800bda4f750] crash_kexec at ffffffff802a3b97
#1 [ffff8800bda4f810] __die at ffffffff80264349
#2 [ffff8800bda4f850] do_page_fault at ffffffff80266971
#3 [ffff8800bda4f940] error_exit at ffffffff8025f82b
[exception RIP: copy_page+50]
RIP: ffffffff80260b9e RSP: ffff8800bda4f9f8 RFLAGS: 00010202
RAX: 0000000000000246 RBX: 00007fff16e6f4a8 RCX: 0000000000000025
RDX: 000000000000e02b RSI: ffff8800bda4ffe8 RDI: ffff880121807540
RBP: ffffffff886439c0 R8: 00007fff16e70da0 R9: 00007fff16e705a0
R10: 0000000000000004 R11: 00002ab8963e6135 R12: 0000000000000033
R13: ffff880121807000 R14: ffff880121807000 R15: ffff880188844fc7
ORIG_RAX: ffffffffffffffff CS: e030 SS: e02b
#4 [ffff8800bda4f9f0] __alloc_pages at ffffffff8020f477
#5 [ffff8800bda4fa60] vfs_kern_mount at ffffffff802d2197
#6 [ffff8800bda4faa0] nfs_do_submount at ffffffff88616cdc
#7 [ffff8800bda4fb10] nfs_follow_mountpoint at ffffffff88616df7
#8 [ffff8800bda4fca0] __link_path_walk at ffffffff8020a7df
#9 [ffff8800bda4fd10] link_path_walk at ffffffff8020eb6e
#10 [ffff8800bda4fdd0] do_path_lookup at ffffffff8020cfa5
#11 [ffff8800bda4fe10] __user_walk_fd at ffffffff8022394e
#12 [ffff8800bda4fe40] vfs_stat_fd at ffffffff8022889b
#13 [ffff8800bda4fef0] sys_newstat at ffffffff802236f4
#14 [ffff8800bda4ff80] tracesys at ffffffff8025f2f9 (via system_call)
RIP: 00002ab8963e6135 RSP: 00007fff16e6f4a8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: ffffffff8025f2f9 RCX: ffffffffffffffff
RDX: 00007fff16e70da0 RSI: 00007fff16e70da0 RDI: 00007fff16e705a0
RBP: 00002ab8b1a65c3b R8: fefefefefefefeff R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff16e70da0
R13: 00007fff16e70da0 R14: 0000000000000104 R15: 00002ab8b1a6eee4
ORIG_RAX: 0000000000000004 CS: 0033 SS: e02b
...so it looks like this is the problem that we think it is.
Tried the kernel on my experimental VM - works fine so far - but that's no surprise as the conditions in VM are very different to the production box where I experienced this problem. Couple of questions: 1. The bug is only evident on a system acting as a NFS server with SELinux enabled or permissive. Disabling SELinux completely should fix the problem, too. Am I right here? 2. When will RedHat release an official kernel update containing this fix? Seems to me quite critical one (the .128.el5xen kernel crashes very often due to this bug so it is hard to believe I will have to wait for U4 to get this fixed). Thanks, Ondrej 1) that is correct. The bug only presents itself when SELinux is NOT disabled. 2) I don't have an answer on when it would be released in a official kernel, you'll have to work with the support organization to find that out. Can you explain what your NFS server is doing which is out of the ordinary? Is this because of NFSv4 referals? are you using the nohide option? I guess it's possible either of those could tickle the problem, but I'd love to get an idea what you are doing so I can try to write an internal test case. I see a bug in the code and that's where your kernel came from, but I just can't figure out how to hit that bug the way you are. My /etc/exports: /exports *(ro,no_root_squash,sync,fsid=0) /exports/int1 *(ro,no_root_squash,sync,nohide,fsid=10) hercules(rw,no_root_squash,sync) /exports/int1/applications *(rw,no_root_squash,sync,nohide) /exports/int2 *(ro,no_root_squash,sync,fsid=11) hercules(rw,no_root_squash,sync) /exports/int2/rhel4/snapshot *(rw,no_root_squash,sync,fsid=13) /exports/int2/centos5/snapshot *(rw,no_root_squash,sync,fsid=14) /exports/int2/ltsp-gutsy ts32(rw,no_root_squash,sync) /exports/int2/ltsp-hardy-test ts32(rw,no_root_squash,sync) /exports/ext1 *(rw,sync,fsid=15) deneb(no_root_squash) /exports/ext1/tmp *(rw,sync,fsid=16) ... so nothing extraordinary, as you can see. NFSv4 is used rarely on this system. Is the error in NFS client or server code? Jeff also mentioned that XEN could be also involved here. Is that possible? I have another machine (NFS client only, but heavily stressed) running .92.1.10.el5 with SELinux enabled (no XEN) which is rock stable. Ondrej The patch here fixes a problem in the way that submounts are being handled in RHEL5. It's being done incorrectly in all kernels. I don't think xen is really "involved" per-se. It necessarily handles memory a little differently than "normal" kernels and it may just be that that's enough to make this bug cause a crash. I just noticed you are saying 'server.' But the code paths in question in the 2 backtraces are nfs client code paths. Is the machine in question both a client and a server? My fix only affects NFS client mounting. Shouldn't have any affect on a server, so maybe we've got another problem..... I see you have 'nohide,' which is one of the areas I thought could be a problem. If the problem is what I think it is, when on the client you follow from say /exports/int1 to /exports/int1/applications you are going to perform another mount (this all happens transparently on the client) and it is here that we might hit a problem. I guess if the xen kernel changes how the stack frame is positioned (and especially what is on the page directly above the top of the stack frame) we could have a problem. If you can get this test kernel to panic, please provide the backtrace, but my guess is that you shouldn't have any problems with it. /exports/int1 and below is a single filesystem so hence same FSID so no extra mount should be necessary (but I do not know, I am not NFS expert). But anyway there should be no requests to mount /exports/int1 - the automounter entry only contains key for /exports/int1/applications... Yes, the machine is client and server. The "nohide" option I introduced here for NFSv4 compatibility. Unfortunately I am running with SELinux disabled at the moment and enabling it again would mean relabeling the whole 3TB disk array attached. And besides, this is a production box, I am not going to risk my career here :-( So I am not going to put the test kernel into production - sorry. I certainly wouldn't want you to put yourself in a bad position. You are safe from the bug I fixed in the test kernel by just using selinux=0. I'll take your exports and try to play with them to reproduce it here, but from what information I've seen and looking at the code I hope that I've already fixed the only bug in this area. Thank you for the report! posted 5/4 This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. in kernel-2.6.18-146.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please do NOT transition this bugzilla state to VERIFIED until our QE team has sent specific instructions indicating when to do so. However feel free to provide a comment indicating that this fix has been verified. ~~ Attention - RHEL 5.4 Beta Released! ~~ RHEL 5.4 Beta has been released! There should be a fix present in the Beta release that addresses this particular request. Please test and report back results here, at your earliest convenience. RHEL 5.4 General Availability release is just around the corner! If you encounter any issues while testing Beta, please describe the issues you have encountered and set the bug into NEED_INFO. If you encounter new issues, please clone this bug to open a new issue and request it be reviewed for inclusion in RHEL 5.4 or a later update, if it is not of urgent severity. Please do not flip the bug status to VERIFIED. Only post your verification results, and if available, update Verified field with the appropriate value. Questions can be posted to this bug or your customer or partner representative. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-1243.html |