Bug 432867
|
Description
Jeff Layton
2008-02-14 20:37:53 UTC
Hmmm...except that those patches look to be fairly invasive and probably not kabi-friendly. We may have to fix this by doing something different than how upstream did it... A quick check indicates that RHEL4 has the same problem. Fixing this isn't going to be trivial. The best method I can see is that when we spawn a new portmap call that we alloc a small memory region with a refcounter (maybe a kref even) and a field for the port number. The main rpc task and the portmap task will each hold a reference. The portmap task will release the reference when it gets the port (or an error), and the main task will release the ref when the portmap call completes (or the task is torn down). Created attachment 301324 [details]
patch -- add new reference counted response container for portmap calls
First pass at a patch to fix this problem. It seems to work, but I'm not sure
I'm cleaning up everything correctly if the child RPC task exits abnormally
(times out, etc).
The patch adds a new struct that's managed with a kref. The both the parent and
child RPC client will have a reference to it. The intent is to drop this
reference after the child task copies the port number into the container from
the RPC response. The parent will drop the reference when it copies the result
from the container into the cl_port field. We'll also need to drop the
reference if the tasks are killed.
It's this last part that I'm not sure I have correct, but I think I'm pretty
close.
I've also done my best to preserve kabi, but it may need more massaging to
prevent breakage.
Created attachment 306361 [details]
SUNRPC: prevent memory corruption from successful portmap call with dead parent task
Updated patch. I believe this one gets the refcounting right when the task is
killed.
Created attachment 306393 [details]
patch -- SUNRPC: prevent memory corruption from successful portmap call with dead parent task
Oof -- the last patch was wrongwrongwrong...
This patch does the right thing. It's also smaller.
Actually, the original patch was right. We have to do it this way (hook the put in rpc_destroy_client), so that the child task does a put regardless of how the RPC task exits. Created attachment 306475 [details]
SUNRPC: prevent memory corruption from successful portmap call with dead parent task
Small revision. Since there can be several portmap calls in flight at the same
time, it's possible that the cl_port will be non-zero when pmap_getport_done
runs. If a portmap call succeeds, and then a following one fails we don't want
to clobber the port we got from the earlier call, so check to make sure cl_port
zero before copying the port from the container.
Created attachment 306479 [details] small test program to set a blocking lock on a file To reproduce the memory corruption: 1) set up an NFS server on a machine booted to a kernel with memory poisoning enabled. The kernel-debug packages have this enabled already. It'll also be easier to reproduce this if the server's kernel has the patches for bug 254195, since that helps guarantee that the parent RPC task is killed when NFS is stopped. After starting the nfs service, wait 90s to make sure you're past the server's grace period. 2) mount an export from this server on another host (any RHEL client will do). 3) use the test program to set a lock on a file on the server: server# lockit /export/testfile hit return to exit 4) do a blocking lock of the same file on the client (make sure you're past the server's lock grace period): client# lockit /mnt/nfs/testfile (you'll get no output since the program is blocked waiting on the lock) 5) on the client, set up iptables rules to drop all traffic to and from the server to simulate a network partition: client# iptables -A INPUT -s <server addr> -j DROP client# iptables -A OUTPUT -d <server addr> -j DROP 6) release the lock on the server (hit return), and then run: server# service nfs stop ...on the server to bring down nfs and kill off lockd. 7) unpartition the network: client# service iptables stop ...the last 2 steps need to be timed right so that the portmap calls are still being reattempted, but the parent RPC callback is dead. When you remove the network parition, the portmap call succeeds and the port is copied into the parent RPC task. Since the parent RPC task should be dead at this point however, the result is copied into a rpc_clnt struct that has already been freed. The results of this are not always evident right away. The corruption will not be noticed until the memory is reused for other purposes. This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. Created attachment 312372 [details]
patch -- SUNRPC: prevent memory corruption from successful portmap call with dead parent task
Updated patch -- make sure we zero out new->cl_pmap_result in rpc_clone_client.
in kernel-2.6.18-99.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 This bug has been marked for inclusion in the Red Hat Enterprise Linux 5.3 Release Notes. To aid in the development of relevant and accurate release notes, please fill out the "Release Notes" field above with the following 4 pieces of information: Cause: What actions or circumstances cause this bug to present. Consequence: What happens when the bug presents. Fix: What was done to fix the bug. Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore') An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-0225.html |