Bug 481546

A report in http://bugs.centos.org/view.php?id=3359 also exists

Giannis

Created attachment 329967 [details]
Part of the tc script

Could you tell me version of iproute? Both Fedora and CentOS?

Well Fedora is formated but it should have been the latest one
which is iproute-2.6.26-2.fc8

In Centos it is iproute-2.6.18-7.el5

thanks for reply

Giannis

Actually there were changes between those two versions, which were touching the ceil option. If it was intrusive change then I'm surprised that no one else is complaining. I'll post testing patch when I found out what's wrong.

Would it be safe for me to use
the iproute package from Fedora?

Also maybe not sure if the problem is on the tc program
or the htb in kernel.

Giannis

iproute is connected to kernel. I can't even say if iproute from F-8 will be working with RHEL-5 kernel.

It looks like ceil is working but the iptables must be altered. In attachments you can find your polished testing script. After using sport, which is commented out in script, is shaping working again correctly.

Created attachment 330234 [details]
testing script

Sorry, I haven't RH but I have the same problem as Gianis, ceil does not working for me in 1:6666 class.
Kernel: self builded 2.6.27.10 from Mandriva source with ipp2p and l7 filter support
Iptables: 1.4.2 compiled from latest source (netfilter.org)
Iproute: 2.6.26-1mdv2009.0


class htb 1:666 parent 1:1 rate 10240Kbit ceil 10240Kbit burst 32Kb cburst 1598b 
 Sent 439343367 bytes 4836594 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 47200bit 102pps backlog 0b 0p requeues 0 
 lended: 1480449 borrowed: 0 giants: 0
 tokens: 24959 ctokens: 1179

class htb 1:6666 parent 1:666 leaf 6666: prio 7 rate 128000bit ceil 512000bit burst 32Kb cburst 1599b 
 Sent 439343367 bytes 4836594 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 47200bit 102pps backlog 0b 0p requeues 0 
 lended: 3356145 borrowed: 1480449 giants: 0
 tokens: 1996583 ctokens: 23560

But as you can see above iproute-2.6.26 is working for him. I was testing F-10 iproute v2.6.27, iptables-1.4.1.1 and it's also working. Maybe there was some change in kernel or iptables which had an side effect.

Hi, Marcela and thanks for your efforts.

I've tried your script without luck.

class htb 1:1 root rate 1000Mbit ceil 1000Mbit burst 126500b/8 mpu 0b overhead 0b cburst 126500b/8 mpu 0b overhead 0b level 7 
 Sent 694706685 bytes 164962 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 49350Kbit 1443pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 176100
 tokens: 1012 ctokens: 1012

class htb 1:2 parent 1:1 rate 20000Kbit ceil 20000Kbit burst 4Kb/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 6 
 Sent 694706685 bytes 164962 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 49306Kbit 1445pps backlog 0b 0p requeues 0 
 lended: 156510 borrowed: 0 giants: 176100
 tokens: -1998 ctokens: -1998

class htb 1:27 parent 1:2 leaf 27: prio 5 quantum 12500 rate 1000Kbit ceil 20000Kbit burst 1725b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 693794782 bytes 164947 pkt (dropped 1627, overlimits 0 requeues 0) 
 rate 49367Kbit 1453pps backlog 0b 57p requeues 0 
 lended: 8380 borrowed: 156510 giants: 92444
 tokens: -3464 ctokens: -2021

class htb 1:25 parent 1:2 leaf 25: prio 3 quantum 125000 rate 10000Kbit ceil 20000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 53616 bytes 72 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 3504bit 0pps backlog 0b 0p requeues 0 
 lended: 72 borrowed: 0 giants: 0
 tokens: 2191 ctokens: 1596


It looks to me that marking is done correct. Also traffic goes
to the appropriate class and is being countered.
The problem seems like htb does not work.

Actually our scripts are almost the same. The only difference is that you
put the flowid handle for 1:27. This is the default class,
so all traffic that is not marked by iptables should go there
by default.

My iptables also look like yours. I'll give you an example of mine:

*mangle
# local traffic
-A POSTROUTING -d 10.0.0.0/255.0.0.0 -j MARK --set-mark 2
-A POSTROUTING -d 10.0.0.0/255.0.0.0 -j RETURN
# web server
-A POSTROUTING -p tcp -m tcp --sport 80 -j MARK --set-mark 6
-A POSTROUTING -p tcp -m tcp --sport 80 -j RETURN
# rest of traffic (ftp server) unmarked goes to 1:27

Most of my mangle table looks like this. I do mark according to 
destination IP or source port of the server.

regards
Giannis

Hello,
did you try my script with uncommented sports? Please use exactly this script and try to download with wget and scp at the same time. Please report the results. It works for me on RHEL.

With comments (from different download machines) I get
~ 300 Mbps from scp/ssh
~ 100 Mbps from wget/http

class htb 1:27 parent 1:2 leaf 27: prio 5 quantum 12500 rate 1000Kbit ceil 20000Kbit burst 1725b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 3132932221 bytes 98998 pkt (dropped 1687, overlimits 0 requeues 0) 
 rate 288832Kbit 1263pps backlog 0b 22p requeues 0 
 lended: 5181 borrowed: 93795 giants: 81215
 tokens: -6808 ctokens: -1736

class htb 1:25 parent 1:2 leaf 25: prio 3 quantum 125000 rate 10000Kbit ceil 20000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 2280 ctokens: 1640

With comments out (sport mangling on)
~ 300 Mbps from scp/ssh
~ 15 Mbps from wget/http

class htb 1:27 parent 1:2 leaf 27: prio 5 quantum 12500 rate 1000Kbit ceil 20000Kbit burst 1725b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 192156458 bytes 11326 pkt (dropped 179, overlimits 0 requeues 0) 
 rate 17712Kbit 108pps backlog 0b 85p requeues 0 
 lended: 6826 borrowed: 4415 giants: 7467
 tokens: -23054 ctokens: 435

class htb 1:25 parent 1:2 leaf 25: prio 3 quantum 125000 rate 10000Kbit ceil 20000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 4Kb/8 mpu 0b overhead 0b level 0 
 Sent 2482308602 bytes 99983 pkt (dropped 671, overlimits 0 requeues 0) 
 rate 231129Kbit 1151pps backlog 0b 6p requeues 0 
 lended: 53407 borrowed: 46570 giants: 131507
 tokens: -3252 ctokens: -2219

In my comment above I used your script as it was.
Even tried the simplest script I could think of,
by globaly limiting outgoing traffic:

--
#!/bin/sh

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1: htb default 27
tc class add dev eth0 parent 1: classid 1:1 htb rate 10mbit ceil 10mbit

tc class add dev eth0 parent 1:1 classid 1:27 htb rate 1mbit ceil 10mbit prio 5
tc qdisc add dev eth0 parent 1:27 handle 27: sfq perturb 10

tc filter add dev eth0 protocol ip parent 1:0 prio 5 handle 7 fw flowid 1:27

iptables -t mangle -F
iptables -t mangle -A POSTROUTING -j MARK --set-mark 7
--

# tc -s -d class show dev eth0
class htb 1:1 root rate 10000Kbit ceil 10000Kbit burst 2850b/8 mpu 0b overhead 0b cburst 2850b/8 mpu 0b overhead 0b level 7 
 Sent 507282332 bytes 72620 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 41216Kbit 730pps backlog 0b 0p requeues 0 
 lended: 65052 borrowed: 0 giants: 81000
 tokens: -1074 ctokens: -1074

class htb 1:27 parent 1:1 leaf 27: prio 5 quantum 12500 rate 1000Kbit ceil 10000Kbit burst 1725b/8 mpu 0b overhead 0b cburst 2850b/8 mpu 0b overhead 0b level 0 
 Sent 506776792 bytes 72653 pkt (dropped 957, overlimits 0 requeues 0) 
 rate 41846Kbit 711pps backlog 0b 33p requeues 0 
 lended: 7568 borrowed: 65052 giants: 44725
 tokens: -15086 ctokens: -1074

Traffic bypasses the 10Mbit ceil and scales up to 40Mbit.
Wondering though why it does not go up to 50mbps which is the router shaping

:)) success

I just had a flash! About a year ago I had submitted a bug report for the same
server. http://bugzilla.kernel.org/show_bug.cgi?id=9808

I had random kernel crashes related to e1000 and TSO. 
Disabling TSO at that time solved my problem.

Well it did now as well :-)

ethtool -k eth0
ethtool -K eth0 tso off

The shaping seems to work like it used before the transition
to CentOS. Also I've noticed that the giants stopped appearing

was:

class htb 1:2 parent 1:1 rate 40000Kbit ceil 40000Kbit burst 6600b/8 mpu 0b
overhead 0b cburst 6600b/8 mpu 0b overhead 0b level 6
 Sent 15748136104 bytes 2512871 pkt (dropped 0, overlimits 0 requeues 0)
 rate 89562Kbit 1738pps backlog 0b 0p requeues 0
 lended: 879290 borrowed: 0 giants: 3042288  <<<-------------
 tokens: 912 ctokens: 912

is now with TSO disabled:

class htb 1:2 parent 1:1 rate 40000Kbit ceil 40000Kbit burst 6600b/8 mpu 0b overhead 0b cburst 6600b/8 mpu 0b overhead 0b level 6 
 Sent 1539289766 bytes 1045233 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 39646Kbit 3372pps backlog 0b 0p requeues 0 
 lended: 55734 borrowed: 0 giants: 0  <<<--------------
 tokens: -1709 ctokens: -1709

At that time problem had to do with htb-e1000-TSO.
Now seems to me it's totally related.

regards,

Giannis

the problem still exists in 5.4:-(

(In reply to comment #16)
> the problem still exists in 5.4:-(  

The problem was with wrong setting (or hardware) as you can see in comment #15. In case your have problems with configuration feel free to contact our support
http://www.redhat.com/support/process/production/

no the problem was NOT wrong settings! just read the intel guys comment in #15. the suggested solution ie:
ethtool -k eth0
ethtool -K eth0 tso off
is really working, but it's just a workaround and not the solution! it should have to work with TSO on too.

Levente is right.

ethtool -K eth0 tso off 
is just a workaround and not a complete solution.

Also have in mind that this is probably fixed in a later kernel version
since that bug was NOT present when the same server/hardware was running Fedora.

best regards,

Giannis
ps. I can live with tso disabled :)

In this case, it should be reported on kernel. Reassigning.

I have same problem on CentOS 5.4. On Fedora 5 worked OK.

This solution is NOT fix my problem!
ethtool -K eth0 tso off

Here is my CentOS Bug Tracker:
http://bugs.centos.org/view.php?id=4156

check https://bugzilla.redhat.com/show_bug.cgi?id=436966

We 're talking about this now.

I set with HTB 4096Kbit/s for each PC, but i have bigger speed x2-x3 times more.

Hardware:
Tyan Server B7016G24V4H, motherboard S7016GM3NR, CPU: Intel X5550 (Nehalem), 6Gb RAM, 2xNIC Intel 82576 Gigabit.
Installed: CentOS 5.4
Kernel: 2.6.18-164.10.1.el5PAE 0000001 SMP Thu Jan 14 19:55:40 EET 2010 i686 i686 i386 GNU/Linux

Standard kernel drivers trying to. Then installing Intel new drivers.

NIC drivers:
ethtool -i eth0
driver: igb
version: 2.1.1
firmware-version: 1.4-3
bus-info: 0000:09:00.0

===

#Example for dowbload per one PC station (IP hidden):
$TC class add dev $DEV parent 1:${FI}01 classid 1:f701 htb rate 5120Kbit
$TC qdisc add dev $DEV parent 1:f701 handle 0xf701 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:01: match ip dst xxx.xxx.xxx.1 classid 1:f701

# After HTB start:
class htb 1:f701 parent 1:be01 leaf f701: prio 0 rate 5120Kbit ceil 5120Kbit burst 2240b cburst 2240b 
Sent 85217655 bytes 16045 pkt (dropped 0, overlimits 0 requeues 0) 
rate 14449Kbit 332pps backlog 0b 6p requeues 0 
lended: 16039 borrowed: 0 giants: 18590
tokens: -4620 ctokens: -4620

In realy, rate ~3 time more.

What ideas?


Trying this:
#ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: on

# Set hpet timer instead of default - tsc
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

Kernel compiled:
Processor family -> Pentium-4/Celeron(P4-based)/Pentium-4 M/Xeon
Preemption Model -> No Forced Preemption (Server)
CONFIG_HZ_1000=y
CONFIG_HZ=1000


What ideas?

My HTB code:

#!/bin/sh
 
###############################################################################
# variables configuration
###############################################################################
 
TC=/sbin/tc
FI=be
# Free Ip range that not used in our network
# configuration f.e: (x.x.255.x) - ff
# be (x.x.190.x)
 
###############################################################################
# some general initialization
###############################################################################
 
$TC qdisc del dev eth0 root 2>/dev/null
$TC qdisc del dev eth1 root 2>/dev/null
 
# qdisc root
$TC qdisc add dev eth0 root handle 1: htb default ${FI}fe r2q 1
$TC qdisc add dev eth1 root handle 1: htb default ${FI}fe r2q 1
 
###############################################################################
# eth0 (downstream)
###############################################################################
 
# eth0 smotrit v nashu set
# vse IP u glientov GLOBALNYE
 
DEV=eth0
 
# making hash table for fast clients filtering, max table size is 256 entries
# fw filter does`nt support hashing, so we can use hashing of filters
# only on incoming traffic because for outgoung we uze fw filter
$TC filter add dev $DEV parent 1: prio 100 handle 2: protocol ip u32 divisor 256
 
# htb root class
$TC class add dev $DEV parent 1: classid 1:${FI}01 htb rate 1000Mbit
 
# default class ( all unclassified traffic goes here )
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}fe htb rate 3Mbit prio 5
$TC qdisc add dev $DEV parent 1:${FI}fe handle 0x${FI}fe sfq perturb 10
 
# NetWork Server Traffic
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}10 htb rate 25Mbit prio 10
$TC qdisc add dev $DEV parent 1:${FI}10 handle 0x${FI}10 sfq perturb 10
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 match ip src xxx.xxx.127.106/32 classid 1:${FI}10
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 match ip src xxx.xxx.200.49/32 classid 1:${FI}10
 
# For System Subnet
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}11 htb rate 200Mbit prio 8
$TC qdisc add dev $DEV parent 1:${FI}11 handle 0x${FI}11 sfq perturb 10
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 match ip dst xxx.xxx.247.192/26 classid 1:${FI}11
 
# filtering traffic to hashing tables by dst address
$TC filter add dev $DEV protocol ip parent 1: prio 100 u32 ht 800:: \
match ip dst 0/0 \
hashkey mask 0x000000ff at 16 \
link 2:
 
# Clients
 
# example
# d952 - last 2 bytes from IP (*.*.217.82)
# {FI}10  10 or any another number
# ht 2:52: - hash table - 2: value of link value - 52: last byte of IP (*.*.*.82)
 
#<download>
 
# client: xxx.xxx.245.32
$TC class add dev $DEV parent 1:${FI}01 classid 1:f520 htb rate 2560Kbit
$TC qdisc add dev $DEV parent 1:f520 handle 0xf520 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:20: match ip dst xxx.xxx.245.32 classid 1:f520
 
# client: xxx.xxx.245.33
$TC class add dev $DEV parent 1:${FI}01 classid 1:f521 htb rate 2560Kbit
$TC qdisc add dev $DEV parent 1:f521 handle 0xf521 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:21: match ip dst xxx.xxx.245.33 classid 1:f521
 
# client: xxx.xxx.245.72
$TC class add dev $DEV parent 1:${FI}01 classid 1:f548 htb rate 2560Kbit
$TC qdisc add dev $DEV parent 1:f548 handle 0xf548 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:48: match ip dst xxx.xxx.245.72 classid 1:f548
 
# client: xxx.xxx.245.129
$TC class add dev $DEV parent 1:${FI}01 classid 1:f581 htb rate 5120Kbit
$TC qdisc add dev $DEV parent 1:f581 handle 0xf581 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:81: match ip dst xxx.xxx.245.129 classid 1:f581
 
#... and so on
#</download>
 
###############################################################################
# eth1 upstream
###############################################################################
 
# eth1 smotrit na glavnogo provaidera
# vse IP u glientov GLOBALNYE
 
DEV=eth1
 
# making hash table for fast clients filtering, max table size is 256 entries
$TC filter add dev $DEV parent 1: prio 100 handle 2: protocol ip u32 divisor 256
 
# htb root class
$TC class add dev $DEV parent 1: classid 1:${FI}01 htb rate 1000Mbit
 
# default class ( all unclassified traffic goes here )
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}fe htb rate 3Mbit prio 5
$TC qdisc add dev $DEV parent 1:${FI}fe handle 0x${FI}fe sfq perturb 10
 
# For System Subnet
$TC class add dev $DEV parent 1:${FI}01 classid 1:${FI}11 htb rate 200Mbit prio 8
$TC qdisc add dev $DEV parent 1:${FI}11 handle 0x${FI}11 sfq perturb 10
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 match ip src xxx.xxx.247.192/26 classid 1:${FI}11
 
# filtering traffic to hashing tables by src address
$TC filter add dev $DEV protocol ip parent 1: prio 100 u32 ht 800:: \
match ip dst 0/0 \
hashkey mask 0x000000ff at 12 \
link 2:
 
# Clients
 
#<upload>
 
# client: xxx.xxx.245.32
$TC class add dev $DEV parent 1:${FI}01 classid 1:f520 htb rate 2048Kbit
$TC qdisc add dev $DEV parent 1:f520 handle 0xf520 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:20: match ip src xxx.xxx.245.32 classid 1:f520
 
# client: xxx.xxx.245.33
$TC class add dev $DEV parent 1:${FI}01 classid 1:f521 htb rate 2048Kbit
$TC qdisc add dev $DEV parent 1:f521 handle 0xf521 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:21: match ip src xxx.xxx.245.33 classid 1:f521
 
# client: xxx.xxx.245.72
$TC class add dev $DEV parent 1:${FI}01 classid 1:f548 htb rate 2048Kbit
$TC qdisc add dev $DEV parent 1:f548 handle 0xf548 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:48: match ip src xxx.xxx.245.72 classid 1:f548
 
# client: xxx.xxx.245.129
$TC class add dev $DEV parent 1:${FI}01 classid 1:f581 htb rate 5120Kbit
$TC qdisc add dev $DEV parent 1:f581 handle 0xf581 pfifo limit 60
$TC filter add dev $DEV parent 1:0 protocol ip prio 100 u32 ht 2:81: match ip src xxx.xxx.245.129 classid 1:f581
 
#... and so on
#</upload>

Some answers?

Fixed in upstream commit

commit c9726d6890f7f3a892c879e067c3ed839f61e745
Author: Ranjit Manomohan <ranjitm>
Date:   Tue Jul 10 22:43:16 2007 -0700

    [NET_SCHED]: Make HTB scheduler work with TSO.
    
    Currently the HTB scheduler does not correctly account for TSO packets
    which causes large inaccuracies in the bandwidth control when using TSO.
    This patch allows the HTB scheduler to work with TSO enabled devices.
    
    Signed-off-by: Ranjit Manomohan <ranjitm>
    Signed-off-by: David S. Miller <davem>

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Created attachment 485531 [details]
proposed patch

Patch(es) available in kernel-2.6.18-255.el5
You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5
Detailed testing feedback is always welcomed.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-1065.html