Created attachment 634477 [details]
Program to demonstrate splice(2) issue

Description of problem:

splice(2) requires a manual setsockopt(TCP_NODELAY,...) to ensure that
data have been send over the line.  This is a regression (e.g. RHEL6
Kernels send data out after the 200ms TCP_CORK time), is undocumented
and lowers performance

It is probably an upstream kernel problem but I do not have time to
verify it on a vanilla kernel.

E.g. the attached program makes:

$ strace -t ./a.out 192.168.46.142 3128
...
11:43:31 pipe([3, 4])                   = 0
11:43:31 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 5
11:43:31 connect(5, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("192.168.46.142")}, 16) = 0
11:43:31 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
11:43:31 write(4, "test1234", 8)        = 8
11:43:31 splice(3, NULL, 5, NULL, 1024, 0) = 8
11:43:31 read(0, 
"\n", 1)               = 1
11:43:37 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
11:43:37 read(0, 
"\n", 1)               = 1
11:43:41 exit_group(1)                  = ?


Listening on the wire gives

11:43:31.950346 IP 192.168.46.11.58029 > 192.168.46.142.3128: Flags [SEW], seq 286193466, win 14600, options [mss 1460,sackOK,TS val 1223115 ecr 0,nop,wscale 7], length 0
11:43:31.950698 IP 192.168.46.142.3128 > 192.168.46.11.58029: Flags [S.E], seq 2040511295, ack 286193467, win 14480, options [mss 1460,sackOK,TS val 410585279 ecr 1223115,nop,wscale 7], length 0
11:43:31.950763 IP 192.168.46.11.58029 > 192.168.46.142.3128: Flags [.], ack 1, win 115, options [nop,nop,TS val 1223116 ecr 410585279], length 0

11:43:37.778741 IP 192.168.46.11.58029 > 192.168.46.142.3128: Flags [.], seq 1:9, ack 1, win 115, options [nop,nop,TS val 1228944 ecr 410585279], length 8
11:43:37.779079 IP 192.168.46.142.3128 > 192.168.46.11.58029: Flags [.], ack 9, win 114, options [nop,nop,TS val 410585862 ecr 1228944], length 0

11:43:41.259093 IP 192.168.46.11.58029 > 192.168.46.142.3128: Flags [F.], seq 9, ack 1, win 115, options [nop,nop,TS val 1232424 ecr 410585862], length 0
11:43:41.259629 IP 192.168.46.142.3128 > 192.168.46.11.58029: Flags [F.], seq 1, ack 10, win 114, options [nop,nop,TS val 410586210 ecr 1232424], length 0
11:43:41.259676 IP 192.168.46.11.58029 > 192.168.46.142.3128: Flags [.], ack 2, win 115, options [nop,nop,TS val 1232425 ecr 410586210], length 0



As you can see, splice shown in the first part does not send data
within >6s.  Data is send by the setsockopt() call.


Executing same program on an RHEL6.3 kernel, gives

11:50:15.678677 IP 192.168.46.2.57892 > 192.168.46.142.3128: Flags [SEW], seq 1490964025, win 14600, options [mss 1460,sackOK,TS val 4106269449 ecr 0,nop,wscale 7], length 0
11:50:15.678973 IP 192.168.46.142.3128 > 192.168.46.2.57892: Flags [S.E], seq 1495247511, ack 1490964026, win 14480, options [mss 1460,sackOK,TS val 410626211 ecr 4106269449,nop,wscale 7], length 0
11:50:15.678994 IP 192.168.46.2.57892 > 192.168.46.142.3128: Flags [.], ack 1, win 115, options [nop,nop,TS val 4106269449 ecr 410626211], length 0
11:50:15.880401 IP 192.168.46.2.57892 > 192.168.46.142.3128: Flags [P.], seq 1:9, ack 1, win 115, options [nop,nop,TS val 4106269651 ecr 410626211], length 8
11:50:15.880666 IP 192.168.46.142.3128 > 192.168.46.2.57892: Flags [.], ack 9, win 114, options [nop,nop,TS val 410626231 ecr 4106269651], length 0

This is not perfect either (there is still the 200ms TCP_CORK delay
although TCP_NODELAY is set), but data is send-out without the manual
setsockopt().


Version-Release number of selected component (if applicable):

kernel-3.6.3-1.fc17.x86_64 (broken)
kernel-2.6.32-279.5.2.el6.x86_64 (working)


How reproducible:

100%