Description of Problem: I noticed the following in /var/log/messages: recvmsg bug: copied 39AA115 seq 39AA116 Version-Release number of selected component (if applicable): 2.4.9-31smp How Reproducible: This has been noticed on 2 machines so far. One was a short burst that lasted for ~6 seconds, the other machine logged these messages over a span of 5 minutes. Steps to Reproduce: Unknown. Both of these servers push 100Mb/s or more sustained.
Can you extract all of the "recvmsg bug" messages from your logs and attach them to this bugzilla entry? Also, are you using TUX by chance on this machine?
Created attachment 57744 [details] recvmsg files from the first host
Created attachment 57745 [details] recvmsg from the second host
No, we are not using TUX, but we are using vsftpd, which makes heavy use of sendfile() if that makes any difference.
Thanks for the log files and information, they affirmed my suspicion of what this bug might be caused by. I think I know what is wrong, when we directly copy into userspace from the TCP input packet processing, we mishandle the sequence numbers if this happens to be the FIN packet too.
Created attachment 58064 [details] Fix for Andrew's TCP bug.
Please ignore the patch I posted it turns out to be buggy, searching some more.
Good news is that we know what is happening. Bad news is that vsftpd is a buggy pile of crud. From the perspective of the kernel, the situation is harmless. The messages are printed, but the kernel is actually fine. What the application is doing is allowing multiple threads to read from the same socket, one of which is using MSG_PEEK. This is racy and bad. Secondly, default recommended configuration of vsftpd has the 'async_abor_enable' off. This is VERY BAD, it will confuse every FTP client out there. In fact it completely breaks the functionality of ABORT ftp commands. What vsftpd actually does when you use this default is it tries to complete the data transmission and then reads garbage instead of recognizing the ABORT command (clients use URG data to implement asynchronous commands, vsftpd interprets the ABORT as part of the data stream and locks up). So I am going to recommend two things, edit the vsftpd config on these systems and set 'async_abor_enable' to 'YES'. Second, let's get Arjan to build a kernel with the patch I am about to attach. It will print a rate limited debugging message when applications use MSG_PEEK and try to call read on the same socket in parallel like this.
Created attachment 58127 [details] Fix for MSG_PEEK in parallel with normal recvmsg race in TCP.
This bug has been fixed for a long time, I just never got around to closing it.