Description of problem:
SCTP_COMM_LOST is incorrectly delivered from an SCTP stream socket _after_ 
recvmsg()=-1 ECONNRESET, if the other side performs an abortive shutdown of the 
association and recvmsg() is called after the association has been aborted.

Version-Release number of selected component (if applicable):
kernel-2.6.9-16.EL

How reproducible:
Every time.

Steps to Reproduce:
A script does all the work, so these are the setup steps, the script is run in 
the final step.
1. Download lksctp-tools-1.0.3, untar or install
2. Patch with lksctp-tools-1.0.3-comm_lost_test.patch (attached)
3. configure, make
4. cd to lksctp-tools-1.0.3/src/apps
5. Run comm_lost_test.sh
  
Actual results:
--- clip ---
$ ./comm_lost_test.sh

Case 1:
NOTIFICATION: ASSOC_CHANGE - COMM_LOST
recvmsg: -1 (Connection reset by peer)
recvmsg: -1 (Transport endpoint is not connected)

Case 2:
recvmsg: -1 (Connection reset by peer)
NOTIFICATION: ASSOC_CHANGE - COMM_LOST
recvmsg: -1 (Transport endpoint is not connected)
--- clip ---

In Case 1, recvmsg() has already been called (blocking) when the association is 
aborted. The COMM_LOST is delivered, then the next recvmsg() returns -1 with 
ECONNRESET. This is the correct behaviour.

In Case 2, the association is aborted, then recvmsg() is called; it returns -1 
with ECONNRESET. The next recvmsg() delivers the COMM_LOST notification. This 
is the incorrect behaviour.

Expected results:
Output of Case 2 should be identical to that of Case 1.

Additional info:
lksctp-tools-1.0.3-comm_lost_test.patch
    - allows client to receive and server to abort in sctp_darn
      (patch is a little messy but it works for this test)
    - adds the comm_lost_test.sh script (very simple)
    - adds input files that the script feeds to sctp_darn's interactive mode

I can reproduce this bug with 100% consistency, but as this is some kind of 
event ordering / timing problem, I can only hope that it can be reproduced by 
others also.

It would appear that this is being caused by the fact that sk_err is read
synchronously, causing the effects of setting it (sk_err) to take effect before
the COMM_LOST message is dequeued from the packet queue.  I'm not sure if thats
wrong or not, as POSIX appears to not address this clearly.  It certainly feels
wrong, however, I think we can probably get around it by delaying the setting so
sk_err to ECONNRESET until after the COMM_LOST message is read, although this
solution may be rejected by the upstream community.

I've tested this with TCP and it appears that tcp does indeed allow the
reception of user data that has already been received before returning an socket
level error to the user.

Created attachment 121417 [details]
patch to ensure that user data is all received from a socket before reporting socket errors

This is the patch that I'm going to propose upstream after the holidays.  It
passes the attached test case when applied against the upstream kernel.

The patch looks fine, and passes my tests.

committed in stream U4 build 34.4. A test kernel with this patch is available
from http://people.redhat.com/~jbaron/rhel4/

Kernel from http://people.redhat.com/~jbaron/rhel4/ (34.5 actually) passes my 
test.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2006-0575.html