A note about netperf and the classic TCP_STREAM test - to protect itself from sitting "forever" the receive side of a TCP_STREAM test, the code in recv_tcp_stream() in src/nettest_bsd.c will start a timer equal to the test time plus "PAD_TIME" (four seconds - see src/netlib.h).   The PAD_TIME is to allow netperf's timer to expire before netserver's.  In "normal" operation, netserver's timer should "never" expire - the code running in netserver should get the usual read return of zero to signify that the test has finished and will quite before the timer fires.

If the timer fires, it means that there was no read return of zero before (in this case) 304 seconds.

Likely as not, in this case it simply means that the VMs were so saturated that some of the 1000 (in the context of netperf that is an astronomical number of concurrent netperf TCP_STREAM tests to be running on anything less than a *HUGE* machine) tests could not get their traffic through in less than 304 seconds.

The same sort of thing can happen with a Really Large (tm) TCP window over a Really Slow (tm) link with bare-iron systems.  And with Linux's "autotuning" of the TCP window, a netperf instance without test-specific -s and -S options can grow its window quite large.

Also, it is vitally important to keep in mind that netperf was designed and is implemented as a *benchmark* not as a functional testing tool.  Yes, it does have uses in uncovering bugs, has been used by QA groups since netperf was first released, and this may indeed be a bug, but it is also possible that this is simply a poor use of netperf.

Now, it is little-known, but one can ask a netperf TCP_STREAM test to transfer a fixed quantity of data rather than run for a fixed length of time.  This is accomplished by giving a negative value to the global -l option.

http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002dl_002c-Global-30

For example a netperf asked to transfer 1 GB (GiB for the purists) of data:

raj@tardy:~/netperf2_trunk$ src/netperf -H 192.168.1.3 -l -1G
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.3 (192.168.1.3) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    9.13      941.07   

strictly speaking, the receive side still sets a timer.  This would seem to be a *longstanding* bug in recv_tcp_stream(), but since linux seems to treat the parameter passed to an alarm() call an unsigned quantity, this generally becomes a Really Long (tm) timer.  One that will probably run for even longer than it will take for the thousands of netperfs to complete.  As such, you probably won't get those interrupted system call errors any longer.  Of course, if one or more of the netperf processes go away and the TCP FIN or RST segments don't get seen by the remote netserver processes, they will hang-around for a Really Long Time (tm)...  Netperf/netserver does not set SO_KEEPALIVE on its sockets and makes no use of select/poll/whatnot on the data connections.  Again, netperf is a benchmark, and expects to be run in reasonable benchmarking conditions :)

It seems as if this problem is supposed to happen at some load, and 1000 instances of netperf is OK for RHEL5, especially if the hardware is not a high-end server.
Closing.