Bug 679124

Summary:	SCTP CRC message error on the receiver side
Product:	Red Hat Enterprise Linux 5	Reporter:	Guy Streeter <streeter>
Component:	kernel	Assignee:	Thomas Graf <tgraf>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Red Hat Kernel QE team <kernel-qe>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.4	CC:	cevich, davem, nhorman, nobody, rkhan, tgraf
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-10-24 01:46:45 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Guy Streeter 2011-02-21 16:50:21 UTC

As reported by the customer:

At Feb-8 We have received a CRC error (dso_msg_fault calculated at the vtl ) for the trafic sent within network sctp sockets
the CRC error had resulted in a VTL core .

The Environment
We are using SCTP for internal cluster communication.
We open the SCTP sockets in one-to-many mode, i.e. SOCK_SEQPACKET.
All messages are sent out-of-order.
The sender threads assume that sending of a message is atomic, so we do not use any user-space locking mechanism to synchronize the senders.
As far as we know, this assumption is correct with one-to-many SCTP sockets.
Each sent message contains CRC of its payload.
The receiver threads do use a mutex for synchronization, because in SCTP a message may be partially delivered due to lack of kernel buffers.
In this case we are receiving from the socket until MSG_EOR flag is set, which indicates that the message was completely received.
After a message is completely received, the receiver thread is verifying the CRC of the payload.

The Bug
CRC verification fails.
It is important to note that we do not experience failures with sending messages.
Currently there are no more details. We are trying to reproduce the bug in lab with more traces.

Two questions from the customer:

1. They would like to create a reproducer, but don't know how to make partial message delivery happen consistently. Is there something they can do in their test program to make partial delivery more likely?

2. Are there known kernel issues relating to MSG_EOR flag?

Comment 1 Chris Evich 2011-02-21 17:09:28 UTC

Note: 

Still confirming if the CRC is internal to their message or is the protocol CRC that's failing.

They notice CRC failure only when MSG_EOR is set on the last part of a formerly partially delivered message.

Comment 2 Neil Horman 2011-02-21 18:14:18 UTC

Answers to the above questions:

1) Partial delivery can be forced by careful calculation of the receive window.  Set the receiver RCVBUF size to a known value (knowing that the initial rwnd value is going to be min(RCVBUF/2, 1500).  If the receiver then never actually calls sctp_recvmsg, the rwnd will slowly shrink toward zero.  Use small chunks to fill it until the remaining rwnd value is some small number C, then send one large data chunk who's size > C.  That should force you into partial delivery mode.

2) No.

SCTP does include a adler-32 checksum (CRC) in each sctp common header for validation, however, bad checksums to not result in log messages getting created.  Instead the  checksum errors counter gets incremented in /proc/net/sctp/snmp, so that should provide a definitive indicator if its the customer application noting CRC errors or the protocol (my guess will be the former).

I note this bz is filed against 5.4.  I would strongly suggest recreating this on the 5.7 beta kernel before wasting time on a deeper investigation of the problem. In this case, if multi-homing is in use, and messages are traversing multiple packets, I'd be particularly interested in seeing if this is a dup of bz 517504.

Any TCPdumps you can provide along with re-creation code would be appreciated.

Comment 4 Neil Horman 2011-02-21 20:23:19 UTC

yes, thats correct.  Calling recvmsg may actually be optional for them, depending on if they need to actually get data to validate the crc (which I'm guessing they will have to do).  Eitehr way, yes, you enter partial deliver mode when the next chunk of data for an association would overflow its rwnd.  So you can force the condition by not calling recvmsg in the application so that the receive window closes.

Comment 5 RHEL Program Management 2011-06-20 22:33:59 UTC

This request was evaluated by Red Hat Product Management for inclusion in Red Hat Enterprise Linux 5.7 and Red Hat does not plan to fix this issue the currently developed update.

Contact your manager or support representative in case you need to escalate this bug.

Comment 6 Thomas Graf 2011-08-12 10:46:53 UTC

Any news on reproducing this on 5.7?