Red Hat Bugzilla – Bug 64808
kernel TCP error
Last modified: 2007-04-18 12:42:32 EDT
Description of Problem:
I noticed the following in /var/log/messages:
recvmsg bug: copied 39AA115 seq 39AA116
Version-Release number of selected component (if applicable):
This has been noticed on 2 machines so far. One was a short burst that lasted
for ~6 seconds, the other machine logged these messages over a span of 5 minutes.
Steps to Reproduce:
Unknown. Both of these servers push 100Mb/s or more sustained.
Can you extract all of the "recvmsg bug" messages from your
logs and attach them to this bugzilla entry?
Also, are you using TUX by chance on this machine?
Created attachment 57744 [details]
recvmsg files from the first host
Created attachment 57745 [details]
recvmsg from the second host
No, we are not using TUX, but we are using vsftpd, which makes heavy use of
sendfile() if that makes any difference.
Thanks for the log files and information, they affirmed my
suspicion of what this bug might be caused by. I think I
know what is wrong, when we directly copy into userspace from
the TCP input packet processing, we mishandle the sequence
numbers if this happens to be the FIN packet too.
Created attachment 58064 [details]
Fix for Andrew's TCP bug.
Please ignore the patch I posted it turns out to be buggy,
searching some more.
Good news is that we know what is happening. Bad news is that vsftpd
is a buggy pile of crud.
From the perspective of the kernel, the situation is harmless. The messages
are printed, but the kernel is actually fine. What the application
is doing is allowing multiple threads to read from the same socket, one of which
is using MSG_PEEK. This is racy and bad.
Secondly, default recommended configuration of vsftpd has the 'async_abor_enable'
off. This is VERY BAD, it will confuse every FTP client out there. In fact it
completely breaks the functionality of ABORT ftp commands. What vsftpd actually
does when you use this default is it tries to complete the data transmission
and then reads garbage instead of recognizing the ABORT command (clients use
URG data to implement asynchronous commands, vsftpd interprets the ABORT as part
of the data stream and locks up).
So I am going to recommend two things, edit the vsftpd config on these systems
and set 'async_abor_enable' to 'YES'. Second, let's get Arjan to build a
kernel with the patch I am about to attach. It will print a rate limited
debugging message when applications use MSG_PEEK and try to call read on the
same socket in parallel like this.
Created attachment 58127 [details]
Fix for MSG_PEEK in parallel with normal recvmsg race in TCP.
This bug has been fixed for a long time, I just
never got around to closing it.