Created attachment 461169 [details] reproducer code, invoked as "send [ip] [port] 1024 [num_cpus]" Description of problem: Traffic control (tc) with 10Gig ethernet drops packets when txqueuelen set too low. /etc/sysconfig/network-scripts/ifup-eth does not provide a parameter to set this elegantly. Version-Release number of selected component (if applicable): initscripts-8.45.30-2.el5 How reproducible: 100% Steps to Reproduce: 1. Increase write buffers on sending machine with 'sysctl -w et.core.wmem_default=4194304 2. Check current traffic control settings with 'tc -s qdisc ls' and note dropped packets. 3. Compile attachment send.c as 'gcc -o send -m64 -lpthread send.c' 4. On second machine with 10G ethernet, start a listener as 'nc -d -l 5000 < /dev/null' 5. Start application with './send [ip] 5000 1024 [num_cpus]". In our test, num_cpus=7. 6. Re-run 'tc -s qdisc ls' and note that 'dropped' increases significantly. Actual results: [ON RECEIVER] [root@mrgdenprdn1 /root]# /usr/bin/nc -d -l 5000 < /dev/null [ON TRANSMITTER] [root@mrgdenprdn2 ~]# ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:23:7D:5D:E6:E0 inet addr:10.1.23.65 Bcast:10.1.23.255 Mask:255.255.255.0 inet6 addr: fe80::223:7dff:fe5d:e6e0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:330640 errors:0 dropped:0 overruns:0 frame:0 TX packets:1408100059 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:97168394 (92.6 MiB) TX bytes:1506455772470 (1.3 TiB) Interrupt:169 Memory:fd800000-fdffffff [root@mrgdenprdn2 ~]# /sbin/tc -s qdisc ls qdisc pfifo_fast 0: dev eth0 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 1500822564765 bytes 1408090807 pkt (dropped 0, overlimits 0 requeues 571) rate 0bit 0pps backlog 0b 0p requeues 571 [root@mrgdenprdn2 ~] send 10.0.0.1 5000 1024 7 ... lots of output [root@mrgdenprdn2 /root]# /sbin/tc -s qdisc ls qdisc pfifo_fast 0: dev eth0 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 1502438590874 bytes 1409606952 pkt (dropped 4222154, overlimits 0 requeues 572) rate 0bit 0pps backlog 0b 0p requeues 572 Expected results: Transmission control should not be dropping packets, but it is because txqueuelen is too low. Additional info: Proposed patch to /etc/sysconfig/network-scripts/ifup-eth.
Created attachment 461171 [details] Patch Added options for interface (in ifcfg-ethX) configuration: IFCONFIG_OPTS: ability to modify interface options via "ifconfig -s" (including "txqueuelen"). ETHTOOL_C_OPTS: ability to modify ethernet options via "ethtool -C".
While ETHTOOL_OPTS remains as a legacy supported config mechanism, new config really should be done via interface-specific udev rules (that therefore will get automatically applied as soon as the interface appears.)