Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 539560

Summary: tcp_disconnect should clear all of tp->rx_opt ....
Product: Red Hat Enterprise Linux 5 Reporter: David Bein <d.bein>
Component: kernelAssignee: Thomas Graf <tgraf>
Status: CLOSED ERRATA QA Contact: Network QE <network-qe>
Severity: medium Docs Contact:
Priority: low    
Version: 5.4CC: dwysocha, hjia, lzheng, qcai, rkhan
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-01-13 20:55:51 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
tcpdump on client which shows the sequence of the corruption
none
proposed patch none

Description David Bein 2009-11-20 15:21:49 UTC
Description of problem:

rpc xprt re-connections over tcp can have trash in the initial fsinfo
if the new connection has not negotiated tcp options, e.g. timestamp,
but the previous connection had negotiated tcp option. What it takes
to have this happen is a clustered server failover where the original
server negotiates timestamps and the new one does not. If the original
connection is nuked from an RST [e.g. from the new server], the logic
in tcp_disconnect() [via xprt connect to AF_UNSPEC before re-connecting
the same socket], the bits in tp->rx_opt are not reset (as they would
be if the original socket were tossed and a new one is created).

This manifests as checksum problems in the first FSINFO packet
because while th_off is 5 [from syn-ack negotiation] the bits
that control the tcp build options can still be set overwriting
the bytes in the beginning of the fsinfo packet and botching
the already computed checksums. What happens is that the packets
are dropped on the other side due to checksum failures. tcp resends
the same packet, but each time it is slightly different due to
timestamp option values. Eventually the connection times out
since the peer never acks the bogus packets.

There is a fix in 2.6.22 upstream that fixes this case.

See the commit for [TCP]: zero out rx_opt in tcp_disconnect().
The commit is b40b4f79ce789e9e28d382c85006f62be2725282.


Version-Release number of selected component (if applicable):


How reproducible:

Not trivial, but the corruption in the outbound fsinfo packet is
very predictable when it happens. This would also affect a user
program if it were emulating the re-use same privileged source
port options and when the connection breaks due to the RST, it
then connects to AF_UNSPEC and then re-connects to the same
destination address. It is not nfs/rpc specific, but impacts
the kernel nfs client badly when it hits.

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 David Bein 2009-11-20 16:16:34 UTC
Created attachment 372532 [details]
tcpdump on client which shows the sequence of the corruption

Comment 2 David Bein 2009-11-20 16:27:41 UTC
This is a problem in RH4 releases too.
Once tcp_build_and_update_options() adds options in tcp_transmit_skb()
if the payload begins at offset (th + 1), the result is corrupted.

Comment 4 Thomas Graf 2010-08-26 09:05:14 UTC
Created attachment 441147 [details]
proposed patch

Comment 5 RHEL Program Management 2010-09-07 23:19:15 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 7 Jarod Wilson 2010-09-15 13:59:56 UTC
in kernel-2.6.18-221.el5
You can download this test kernel from http://people.redhat.com/jwilson/el5

Detailed testing feedback is always welcomed.

Comment 9 David Bein 2010-11-15 04:02:31 UTC
(In reply to comment #7)
> in kernel-2.6.18-221.el5
> You can download this test kernel from http://people.redhat.com/jwilson/el5
> 
> Detailed testing feedback is always welcomed.

Jarod - sorry I've not had time to test it out as it was hard to repro
trivially before. I did however see the patch and we have been using the
same patch for almost 10 months w/o incident.

I hope this is in the next spin of rh5.5 for the benefit of
anyone using the kernel nfs client in the presence of clustered
back end nfs servers where the negotiated tcp options can be different
depending on which server it is.

Comment 10 Liang Zheng 2010-12-06 05:16:06 UTC
Since there is no reproducer and the reporter said it was hard to reproduce
see https://bugzilla.redhat.com/show_bug.cgi?id=539560#c9

so do the sanityonly 
Confirmed patch is in 2.6.18-235.el5
[root@ibm-x3650-04 SPECS]# cat kernel-2.6.spec | grep patch25677 -i
Patch25677: linux-2.6-net-tcp-zero-out-rx_opt-in-tcp_disconnect.patch
%patch25677 -p1

Comment 12 errata-xmlrpc 2011-01-13 20:55:51 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0017.html