Bug 481546 - HTB qdisc miscalculates bandwidth with TSO enabled
HTB qdisc miscalculates bandwidth with TSO enabled
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.4
i686 Linux
low Severity medium
: rc
: ---
Assigned To: Thomas Graf
Network QE
http://bugzilla.kernel.org/show_bug.c...
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-01-26 04:09 EST by Kapetanakis Giannis
Modified: 2014-06-18 04:30 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-07-21 06:29:45 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Part of the tc script (944 bytes, application/octet-stream)
2009-01-26 04:12 EST, Kapetanakis Giannis
no flags Details
testing script (1.35 KB, application/octet-stream)
2009-01-28 10:10 EST, Marcela Mašláňová
no flags Details
proposed patch (3.08 KB, patch)
2011-03-15 11:50 EDT, Thomas Graf
no flags Details | Diff

  None (edit)
Description Kapetanakis Giannis 2009-01-26 04:09:11 EST
Hi,

I've just transfered my server from Fedora 8 to Centos 5.2 (up2date)
My traffic shapping/priorization for outgoing traffic script stopped to work.

The ceil in htb class is not respected at all.
tc -s -d class show dev eth0

root class:

class htb 1:1 root rate 1000Mbit ceil 1000Mbit burst 126500b/8 mpu 0b overhead 0b cburst 126500b/8 mpu 0b overhead 0b level 7
 Sent 16143749349 bytes 2547371 pkt (dropped 0, overlimits 0 requeues 0)
 rate 89529Kbit 1747pps backlog 0b 0p requeues 0
 lended: 75 borrowed: 0 giants: 3077938
 tokens: 996 ctokens: 996

internal traffic:

class htb 1:12 parent 1:1 prio 2 quantum 200000 rate 500000Kbit ceil 800000Kbit burst 64062b/8 mpu 0b overhead 0b cburst 101600b/8 mpu 0b overhead 0b level 0
 Sent 376769109 bytes 28585 pkt (dropped 0, overlimits 0 requeues 0)
 rate 75800bit 1pps backlog 0b 0p requeues 0
 lended: 28580 borrowed: 0 giants: 33100
 tokens: 1024 ctokens: 1016

external traffic where the ceil of 40mbit is not respected:

class htb 1:2 parent 1:1 rate 40000Kbit ceil 40000Kbit burst 6600b/8 mpu 0b overhead 0b cburst 6600b/8 mpu 0b overhead 0b level 6
 Sent 15748136104 bytes 2512871 pkt (dropped 0, overlimits 0 requeues 0)
 rate 89562Kbit 1738pps backlog 0b 0p requeues 0
 lended: 879290 borrowed: 0 giants: 3042288
 tokens: 912 ctokens: 912

external subclasses:

class htb 1:25 parent 1:2 leaf 25: prio 5 quantum 31250 rate 25000Kbit ceil 40000Kbit burst 4725b/8 mpu 0b overhead 0b cburst 6600b/8 mpu 0b overhead 0b level 0
 Sent 3425794137 bytes 512016 pkt (dropped 304, overlimits 0 requeues 0)
 rate 23978Kbit 515pps backlog 0b 0p requeues 0
 lended: 510280 borrowed: 1703 giants: 649361
 tokens: 1492 ctokens: 1308


class htb 1:26 parent 1:2 leaf 26: prio 5 quantum 12500 rate 10000Kbit ceil 40000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 6600b/8 mpu 0b overhead 0b level 0
 Sent 12297100303 bytes 1991881 pkt (dropped 1957, overlimits 0 requeues 0)
 rate 65850Kbit 1244pps backlog 0b 0p requeues 0
 lended: 1114201 borrowed: 877554 giants: 1872807
 tokens: -1310 ctokens: 912

class htb 1:27 parent 1:2 leaf 27: prio 5 quantum 6250 rate 5000Kbit ceil 40000Kbit burst 2225b/8 mpu 0b overhead 0b cburst 6600b/8 mpu 0b overhead 0b level 0
 Sent 25427920 bytes 9140 pkt (dropped 0, overlimits 0 requeues 0)
 rate 20608bit 3pps backlog 0b 0p requeues 0
 lended: 9100 borrowed: 33 giants: 8169
 tokens: 1141 ctokens: 1018

Default class is 1:27

Any ideas?

Best Regards
Giannis
Comment 1 Kapetanakis Giannis 2009-01-26 04:10:06 EST
A report in http://bugs.centos.org/view.php?id=3359 also exists

Giannis
Comment 2 Kapetanakis Giannis 2009-01-26 04:12:24 EST
Created attachment 329967 [details]
Part of the tc script
Comment 3 Marcela Mašláňová 2009-01-26 04:17:50 EST
Could you tell me version of iproute? Both Fedora and CentOS?
Comment 4 Kapetanakis Giannis 2009-01-28 05:59:18 EST
Well Fedora is formated but it should have been the latest one
which is iproute-2.6.26-2.fc8

In Centos it is iproute-2.6.18-7.el5

thanks for reply

Giannis
Comment 5 Marcela Mašláňová 2009-01-28 07:16:58 EST
Actually there were changes between those two versions, which were touching the ceil option. If it was intrusive change then I'm surprised that no one else is complaining. I'll post testing patch when I found out what's wrong.
Comment 6 Kapetanakis Giannis 2009-01-28 09:00:50 EST
Would it be safe for me to use
the iproute package from Fedora?

Also maybe not sure if the problem is on the tc program
or the htb in kernel.

Giannis
Comment 7 Marcela Mašláňová 2009-01-28 10:09:29 EST
iproute is connected to kernel. I can't even say if iproute from F-8 will be working with RHEL-5 kernel.

It looks like ceil is working but the iptables must be altered. In attachments you can find your polished testing script. After using sport, which is commented out in script, is shaping working again correctly.
Comment 8 Marcela Mašláňová 2009-01-28 10:10:20 EST
Created attachment 330234 [details]
testing script
Comment 9 Eagle 2009-01-28 16:16:02 EST
Sorry, I haven't RH but I have the same problem as Gianis, ceil does not working for me in 1:6666 class.
Kernel: self builded 2.6.27.10 from Mandriva source with ipp2p and l7 filter support
Iptables: 1.4.2 compiled from latest source (netfilter.org)
Iproute: 2.6.26-1mdv2009.0


class htb 1:666 parent 1:1 rate 10240Kbit ceil 10240Kbit burst 32Kb cburst 1598b 
 Sent 439343367 bytes 4836594 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 47200bit 102pps backlog 0b 0p requeues 0 
 lended: 1480449 borrowed: 0 giants: 0
 tokens: 24959 ctokens: 1179

class htb 1:6666 parent 1:666 leaf 6666: prio 7 rate 128000bit ceil 512000bit burst 32Kb cburst 1599b 
 Sent 439343367 bytes 4836594 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 47200bit 102pps backlog 0b 0p requeues 0 
 lended: 3356145 borrowed: 1480449 giants: 0
 tokens: 1996583 ctokens: 23560
Comment 10 Marcela Mašláňová 2009-01-29 03:45:05 EST
But as you can see above iproute-2.6.26 is working for him. I was testing F-10 iproute v2.6.27, iptables-1.4.1.1 and it's also working. Maybe there was some change in kernel or iptables which had an side effect.
Comment 11 Kapetanakis Giannis 2009-01-30 05:03:49 EST
Hi, Marcela and thanks for your efforts.

I've tried your script without luck.

class htb 1:1 root rate 1000Mbit ceil 1000Mbit burst 126500b/8 mpu 0b overhead 0b cburst 126500b/8 mpu 0b overhead 0b level 7 
 Sent 694706685 bytes 164962 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 49350Kbit 1443pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 176100
 tokens: 1012 ctokens: 1012

class htb 1:2 parent 1:1 rate 20000Kbit ceil 20000Kbit burst 4Kb/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 6 
 Sent 694706685 bytes 164962 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 49306Kbit 1445pps backlog 0b 0p requeues 0 
 lended: 156510 borrowed: 0 giants: 176100
 tokens: -1998 ctokens: -1998

class htb 1:27 parent 1:2 leaf 27: prio 5 quantum 12500 rate 1000Kbit ceil 20000Kbit burst 1725b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 693794782 bytes 164947 pkt (dropped 1627, overlimits 0 requeues 0) 
 rate 49367Kbit 1453pps backlog 0b 57p requeues 0 
 lended: 8380 borrowed: 156510 giants: 92444
 tokens: -3464 ctokens: -2021

class htb 1:25 parent 1:2 leaf 25: prio 3 quantum 125000 rate 10000Kbit ceil 20000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 53616 bytes 72 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 3504bit 0pps backlog 0b 0p requeues 0 
 lended: 72 borrowed: 0 giants: 0
 tokens: 2191 ctokens: 1596


It looks to me that marking is done correct. Also traffic goes
to the appropriate class and is being countered.
The problem seems like htb does not work.

Actually our scripts are almost the same. The only difference is that you
put the flowid handle for 1:27. This is the default class,
so all traffic that is not marked by iptables should go there
by default.

My iptables also look like yours. I'll give you an example of mine:

*mangle
# local traffic
-A POSTROUTING -d 10.0.0.0/255.0.0.0 -j MARK --set-mark 2
-A POSTROUTING -d 10.0.0.0/255.0.0.0 -j RETURN
# web server
-A POSTROUTING -p tcp -m tcp --sport 80 -j MARK --set-mark 6
-A POSTROUTING -p tcp -m tcp --sport 80 -j RETURN
# rest of traffic (ftp server) unmarked goes to 1:27

Most of my mangle table looks like this. I do mark according to 
destination IP or source port of the server.

regards
Giannis
Comment 12 Marcela Mašláňová 2009-01-30 07:21:13 EST
Hello,
did you try my script with uncommented sports? Please use exactly this script and try to download with wget and scp at the same time. Please report the results. It works for me on RHEL.
Comment 13 Kapetanakis Giannis 2009-01-30 08:16:42 EST
With comments (from different download machines) I get
~ 300 Mbps from scp/ssh
~ 100 Mbps from wget/http

class htb 1:27 parent 1:2 leaf 27: prio 5 quantum 12500 rate 1000Kbit ceil 20000Kbit burst 1725b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 3132932221 bytes 98998 pkt (dropped 1687, overlimits 0 requeues 0) 
 rate 288832Kbit 1263pps backlog 0b 22p requeues 0 
 lended: 5181 borrowed: 93795 giants: 81215
 tokens: -6808 ctokens: -1736

class htb 1:25 parent 1:2 leaf 25: prio 3 quantum 125000 rate 10000Kbit ceil 20000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 2280 ctokens: 1640

With comments out (sport mangling on)
~ 300 Mbps from scp/ssh
~ 15 Mbps from wget/http

class htb 1:27 parent 1:2 leaf 27: prio 5 quantum 12500 rate 1000Kbit ceil 20000Kbit burst 1725b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 192156458 bytes 11326 pkt (dropped 179, overlimits 0 requeues 0) 
 rate 17712Kbit 108pps backlog 0b 85p requeues 0 
 lended: 6826 borrowed: 4415 giants: 7467
 tokens: -23054 ctokens: 435

class htb 1:25 parent 1:2 leaf 25: prio 3 quantum 125000 rate 10000Kbit ceil 20000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 2482308602 bytes 99983 pkt (dropped 671, overlimits 0 requeues 0) 
 rate 231129Kbit 1151pps backlog 0b 6p requeues 0 
 lended: 53407 borrowed: 46570 giants: 131507
 tokens: -3252 ctokens: -2219
Comment 14 Kapetanakis Giannis 2009-01-30 08:35:32 EST
In my comment above I used your script as it was.
Even tried the simplest script I could think of,
by globaly limiting outgoing traffic:

--
#!/bin/sh

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1: htb default 27
tc class add dev eth0 parent 1: classid 1:1 htb rate 10mbit ceil 10mbit

tc class add dev eth0 parent 1:1 classid 1:27 htb rate 1mbit ceil 10mbit prio 5
tc qdisc add dev eth0 parent 1:27 handle 27: sfq perturb 10

tc filter add dev eth0 protocol ip parent 1:0 prio 5 handle 7 fw flowid 1:27

iptables -t mangle -F
iptables -t mangle -A POSTROUTING -j MARK --set-mark 7
--

# tc -s -d class show dev eth0
class htb 1:1 root rate 10000Kbit ceil 10000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 2850b/8 mpu 0b overhead 0b level 7 
 Sent 507282332 bytes 72620 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 41216Kbit 730pps backlog 0b 0p requeues 0 
 lended: 65052 borrowed: 0 giants: 81000
 tokens: -1074 ctokens: -1074

class htb 1:27 parent 1:1 leaf 27: prio 5 quantum 12500 rate 1000Kbit ceil 10000Kbit burst 1725b/8 mpu 0b overhead 0b cburst 2850b/8 mpu 0b overhead 0b level 0 
 Sent 506776792 bytes 72653 pkt (dropped 957, overlimits 0 requeues 0) 
 rate 41846Kbit 711pps backlog 0b 33p requeues 0 
 lended: 7568 borrowed: 65052 giants: 44725
 tokens: -15086 ctokens: -1074

Traffic bypasses the 10Mbit ceil and scales up to 40Mbit.
Wondering though why it does not go up to 50mbps which is the router shaping
Comment 15 Kapetanakis Giannis 2009-01-30 09:20:21 EST
:)) success

I just had a flash! About a year ago I had submitted a bug report for the same
server. http://bugzilla.kernel.org/show_bug.cgi?id=9808

I had random kernel crashes related to e1000 and TSO. 
Disabling TSO at that time solved my problem.

Well it did now as well :-)

ethtool -k eth0
ethtool -K eth0 tso off

The shaping seems to work like it used before the transition
to CentOS. Also I've noticed that the giants stopped appearing

was:

class htb 1:2 parent 1:1 rate 40000Kbit ceil 40000Kbit burst 6600b/8 mpu 0b
overhead 0b cburst 6600b/8 mpu 0b overhead 0b level 6
 Sent 15748136104 bytes 2512871 pkt (dropped 0, overlimits 0 requeues 0)
 rate 89562Kbit 1738pps backlog 0b 0p requeues 0
 lended: 879290 borrowed: 0 giants: 3042288  <<<-------------
 tokens: 912 ctokens: 912

is now with TSO disabled:

class htb 1:2 parent 1:1 rate 40000Kbit ceil 40000Kbit burst 6600b/8 mpu 0b overhead 0b cburst 6600b/8 mpu 0b overhead 0b level 6 
 Sent 1539289766 bytes 1045233 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 39646Kbit 3372pps backlog 0b 0p requeues 0 
 lended: 55734 borrowed: 0 giants: 0  <<<--------------
 tokens: -1709 ctokens: -1709

At that time problem had to do with htb-e1000-TSO.
Now seems to me it's totally related.

regards,

Giannis
Comment 16 Levente Farkas 2009-10-29 09:46:14 EDT
the problem still exists in 5.4:-(
Comment 17 Marcela Mašláňová 2009-10-29 11:51:07 EDT
(In reply to comment #16)
> the problem still exists in 5.4:-(  

The problem was with wrong setting (or hardware) as you can see in comment #15. In case your have problems with configuration feel free to contact our support
http://www.redhat.com/support/process/production/
Comment 18 Levente Farkas 2009-10-29 12:21:28 EDT
no the problem was NOT wrong settings! just read the intel guys comment in #15. the suggested solution ie:
ethtool -k eth0
ethtool -K eth0 tso off
is really working, but it's just a workaround and not the solution! it should have to work with TSO on too.
Comment 19 Kapetanakis Giannis 2009-10-29 14:27:31 EDT
Levente is right.

ethtool -K eth0 tso off 
is just a workaround and not a complete solution.

Also have in mind that this is probably fixed in a later kernel version
since that bug was NOT present when the same server/hardware was running Fedora.

best regards,

Giannis
ps. I can live with tso disabled :)
Comment 20 Marcela Mašláňová 2009-10-30 05:17:14 EDT
In this case, it should be reported on kernel. Reassigning.
Comment 21 Jevgeni Maksimov 2010-01-20 13:36:36 EST
I have same problem on CentOS 5.4. On Fedora 5 worked OK.

This solution is NOT fix my problem!
ethtool -K eth0 tso off

Here is my CentOS Bug Tracker:
http://bugs.centos.org/view.php?id=4156
Comment 22 Kapetanakis Giannis 2010-01-21 05:27:42 EST
check https://bugzilla.redhat.com/show_bug.cgi?id=436966

We 're talking about this now.
Comment 23 Jevgeni Maksimov 2010-01-21 06:08:53 EST
I set with HTB 4096Kbit/s for each PC, but i have bigger speed x2-x3 times more.

Hardware:
Tyan Server B7016G24V4H, motherboard S7016GM3NR, CPU: Intel X5550 (Nehalem), 6Gb RAM, 2xNIC Intel 82576 Gigabit.
Installed: CentOS 5.4
Kernel: 2.6.18-164.10.1.el5PAE 0000001 SMP Thu Jan 14 19:55:40 EET 2010 i686 i686 i386 GNU/Linux

Standard kernel drivers trying to. Then installing Intel new drivers.

NIC drivers:
ethtool -i eth0
driver: igb
version: 2.1.1
firmware-version: 1.4-3
bus-info: 0000:09:00.0

===

#Example for dowbload per one PC station (IP hidden):
$TC class add dev $DEV parent 1:${FI}01 classid 1:f701 htb rate 5120Kbit
$TC qdisc add dev $DEV parent 1:f701 handle 0xf701 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:01: match ip dst xxx.xxx.xxx.1 classid 1:f701

# After HTB start:
class htb 1:f701 parent 1:be01 leaf f701: prio 0 rate 5120Kbit ceil 5120Kbit burst 2240b cburst 2240b 
Sent 85217655 bytes 16045 pkt (dropped 0, overlimits 0 requeues 0) 
rate 14449Kbit 332pps backlog 0b 6p requeues 0 
lended: 16039 borrowed: 0 giants: 18590
tokens: -4620 ctokens: -4620

In realy, rate ~3 time more.

What ideas?


Trying this:
#ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: on

# Set hpet timer instead of default - tsc
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

Kernel compiled:
Processor family -> Pentium-4/Celeron(P4-based)/Pentium-4 M/Xeon
Preemption Model -> No Forced Preemption (Server)
CONFIG_HZ_1000=y
CONFIG_HZ=1000


What ideas?

My HTB code:

#!/bin/sh
 
###############################################################################
# variables configuration
###############################################################################
 
TC=/sbin/tc
FI=be
# Free Ip range that not used in our network
# configuration f.e: (x.x.255.x) - ff
# be (x.x.190.x)
 
###############################################################################
# some general initialization
###############################################################################
 
$TC qdisc del dev eth0 root 2>/dev/null
$TC qdisc del dev eth1 root 2>/dev/null
 
# qdisc root
$TC qdisc add dev eth0 root handle 1: htb default ${FI}fe r2q 1
$TC qdisc add dev eth1 root handle 1: htb default ${FI}fe r2q 1
 
###############################################################################
# eth0 (downstream)
###############################################################################
 
# eth0 smotrit v nashu set
# vse IP u glientov GLOBALNYE
 
DEV=eth0
 
# making hash table for fast clients filtering, max table size is 256 entries
# fw filter does`nt support hashing, so we can use hashing of filters
# only on incoming traffic because for outgoung we uze fw filter
$TC filter add dev $DEV parent 1: prio 100 handle 2: protocol ip u32 divisor 256
 
# htb root class
$TC class add dev $DEV parent 1: classid 1:${FI}01 htb rate 1000Mbit
 
# default class ( all unclassified traffic goes here )
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}fe htb rate 3Mbit prio 5
$TC qdisc add dev $DEV parent 1:${FI}fe handle 0x${FI}fe sfq perturb 10
 
# NetWork Server Traffic
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}10 htb rate 25Mbit prio 10
$TC qdisc add dev $DEV parent 1:${FI}10 handle 0x${FI}10 sfq perturb 10
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 match ip src xxx.xxx.127.106/32 classid 1:${FI}10
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 match ip src xxx.xxx.200.49/32 classid 1:${FI}10
 
# For System Subnet
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}11 htb rate 200Mbit prio 8
$TC qdisc add dev $DEV parent 1:${FI}11 handle 0x${FI}11 sfq perturb 10
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 match ip dst xxx.xxx.247.192/26 classid 1:${FI}11
 
# filtering traffic to hashing tables by dst address
$TC filter add dev $DEV protocol ip parent 1: prio 100 u32 ht 800:: \
match ip dst 0/0 \
hashkey mask 0x000000ff at 16 \
link 2:
 
# Clients
 
# example
# d952 - last 2 bytes from IP (*.*.217.82)
# {FI}10  10 or any another number
# ht 2:52: - hash table - 2: value of link value - 52: last byte of IP (*.*.*.82)
 
#<download>
 
# client: xxx.xxx.245.32
$TC class add dev $DEV parent 1:${FI}01 classid 1:f520 htb rate 2560Kbit
$TC qdisc add dev $DEV parent 1:f520 handle 0xf520 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:20: match ip dst xxx.xxx.245.32 classid 1:f520
 
# client: xxx.xxx.245.33
$TC class add dev $DEV parent 1:${FI}01 classid 1:f521 htb rate 2560Kbit
$TC qdisc add dev $DEV parent 1:f521 handle 0xf521 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:21: match ip dst xxx.xxx.245.33 classid 1:f521
 
# client: xxx.xxx.245.72
$TC class add dev $DEV parent 1:${FI}01 classid 1:f548 htb rate 2560Kbit
$TC qdisc add dev $DEV parent 1:f548 handle 0xf548 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:48: match ip dst xxx.xxx.245.72 classid 1:f548
 
# client: xxx.xxx.245.129
$TC class add dev $DEV parent 1:${FI}01 classid 1:f581 htb rate 5120Kbit
$TC qdisc add dev $DEV parent 1:f581 handle 0xf581 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:81: match ip dst xxx.xxx.245.129 classid 1:f581
 
#... and so on
#</download>
 
###############################################################################
# eth1 upstream
###############################################################################
 
# eth1 smotrit na glavnogo provaidera
# vse IP u glientov GLOBALNYE
 
DEV=eth1
 
# making hash table for fast clients filtering, max table size is 256 entries
$TC filter add dev $DEV parent 1: prio 100 handle 2: protocol ip u32 divisor 256
 
# htb root class
$TC class add dev $DEV parent 1: classid 1:${FI}01 htb rate 1000Mbit
 
# default class ( all unclassified traffic goes here )
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}fe htb rate 3Mbit prio 5
$TC qdisc add dev $DEV parent 1:${FI}fe handle 0x${FI}fe sfq perturb 10
 
# For System Subnet
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}11 htb rate 200Mbit prio 8
$TC qdisc add dev $DEV parent 1:${FI}11 handle 0x${FI}11 sfq perturb 10
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 match ip src xxx.xxx.247.192/26 classid 1:${FI}11
 
# filtering traffic to hashing tables by src address
$TC filter add dev $DEV protocol ip parent 1: prio 100 u32 ht 800:: \
match ip dst 0/0 \
hashkey mask 0x000000ff at 12 \
link 2:
 
# Clients
 
#<upload>
 
# client: xxx.xxx.245.32
$TC class add dev $DEV parent 1:${FI}01 classid 1:f520 htb rate 2048Kbit
$TC qdisc add dev $DEV parent 1:f520 handle 0xf520 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:20: match ip src xxx.xxx.245.32 classid 1:f520
 
# client: xxx.xxx.245.33
$TC class add dev $DEV parent 1:${FI}01 classid 1:f521 htb rate 2048Kbit
$TC qdisc add dev $DEV parent 1:f521 handle 0xf521 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:21: match ip src xxx.xxx.245.33 classid 1:f521
 
# client: xxx.xxx.245.72
$TC class add dev $DEV parent 1:${FI}01 classid 1:f548 htb rate 2048Kbit
$TC qdisc add dev $DEV parent 1:f548 handle 0xf548 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:48: match ip src xxx.xxx.245.72 classid 1:f548
 
# client: xxx.xxx.245.129
$TC class add dev $DEV parent 1:${FI}01 classid 1:f581 htb rate 5120Kbit
$TC qdisc add dev $DEV parent 1:f581 handle 0xf581 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:81: match ip src xxx.xxx.245.129 classid 1:f581
 
#... and so on
#</upload>
Comment 24 Jevgeni Maksimov 2010-01-23 14:23:04 EST
Some answers?
Comment 25 Thomas Graf 2011-03-15 08:16:13 EDT
Fixed in upstream commit

commit c9726d6890f7f3a892c879e067c3ed839f61e745
Author: Ranjit Manomohan <ranjitm@google.com>
Date:   Tue Jul 10 22:43:16 2007 -0700

    [NET_SCHED]: Make HTB scheduler work with TSO.
    
    Currently the HTB scheduler does not correctly account for TSO packets
    which causes large inaccuracies in the bandwidth control when using TSO.
    This patch allows the HTB scheduler to work with TSO enabled devices.
    
    Signed-off-by: Ranjit Manomohan <ranjitm@google.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>
Comment 28 RHEL Product and Program Management 2011-03-15 11:49:48 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 29 Thomas Graf 2011-03-15 11:50:54 EDT
Created attachment 485531 [details]
proposed patch
Comment 32 Jarod Wilson 2011-04-04 17:57:36 EDT
Patch(es) available in kernel-2.6.18-255.el5
You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5
Detailed testing feedback is always welcomed.
Comment 35 errata-xmlrpc 2011-07-21 06:29:45 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-1065.html

Note You need to log in before you can comment on or make changes to this bug.