Bug 453291

Created attachment 310524 [details]
Shell script to trigger the problem.

(In reply to comment #1)
> Created an attachment (id=310524) [edit]
> Shell script to trigger the problem.
> 

Can you post the output messages from the script?

"no longer accepted into the kernel" is not a useful description of the problem.

Created attachment 310957 [details]
Script output when things work (on Fedora 8 system)

This is the output from the test script when run on a system that does not have
the problem (in particular a Fedora 8 system).

I will also attach a copy of the output on a system that has the problem (a
Fedora 9 box.)

Created attachment 310958 [details]
Script output when things do not work (on a Fedora 9 system)

Here is the companion output that shows what happens when the script is run on
a misbehaving system (Fedora 9)

I have attached two output files, these show the script output that happens on a
good system and a bad system.

The script itself contains the expected output and compares it to what it
actually got.

Sorry for not being more clear about what is going on, here is a reprise:

The "tc" command uses netlink to interact with the kernel to build various
packet queues and filters.  These queues and filters are used to to traffic
shaping and also things like netem.

There were some recent kernel changes with respect to netlink that broke the
ability to send some parameters to the sch_netem kernel module.  Those were
fixed.  These changes, and subsequent fixes, may be related to the bug I am
reporting now.

At the current time "tc" seems to be able to properly set up and report traffic
queues and to interact with the netem module.

However, the "tc" command seems to no longer have the ability to establish
filters in the kernel.

Until a few kernel versions ago "tc" was able to establish filters; as of the
current time that ability is gone.

The test script builds a basic traffic control framework and then tries to add a
couple of filters.

That script used to work.  But now several of the filter-building commands fail
and report an error from the kernel.

The kernel says nothing into dmesg and there's nothing in the normal system log
files.

Selinux is present but in "permissive" mode.  No selinux messages are emitted
into its logs.

Seems to work in a 2.6.25.9 kernel...

Yes, I'm also seeing it fixed on both 32 and 64-bit platforms, at least with
Fedora 8.

By-the-way, 2.6.25.10 direct from kernel.org without the RH/Fedora patches still
fails.  I wonder which patch had the silver bullet.

I'll do some further testing later today (Tuesday/July 8) on a Fedora 9/x86-64
box and let you know.

Still fails on Fedora 9 on Linux A192-203-17-213.cavebear.com
2.6.25.9-76.fc9.x86_64 #1 SMP Fri Jun 27 15:58:30 EDT 2008 x86_64 x86_64 x86_64
GNU/Linux

So it seems fixed on the latest Fedora 8 kernel but not on the latest Fedora 9
kernel.

Hmm, maybe the bug is in the tc program and not the kernel?

Could be.

However, the same problem occurs on a small distro I build for embedded systems
using iproute2-2.6.25 (the latest version of tc).  And there hasn't been a
change to 'tc' on the Fedora side through all of this.

My own guess is that some of the recent kernel changes to netlink and its kernel
macros are involved.  Those changes clobbered some of the parameters being sent
from 'tc' to the sch_netem module (this has been fixed) however it would not be
surprising to see the same kind of thing is clobbering the filter parameters
that 'tc' sends via the same mechanisms.

I tend to suspect that the fix is in one of those patches in the Fedora 8 kernel
chain that is not in the Fedora 9 kernel chain.

I ran some tests and got some interesting results to see if I could figure out
whether it is the kernel acting up or the 'tc' command.

On a F8 box I ran both the F8 and F8 binaries for 'tc".  The F8 one worked, the
F9 one showed the problem.

Then on an F9 box I again ran both the F8 and F9 binaries for 'tc'.  Again, the
F8 one worked and the F9 one showed the problem.

On the F8 box its:
root@klack(44): rpm -qf /sbin/tc
iproute-2.6.22-2.fc8

And on the F9:
iproute-2.6.25-1.fc9.x86_64

Hmmm ... on my embedded system, on which I'm seeing the problem, I'm also using
the 2.6.25 version of iproute2.

Looks like this is fixed in iproute2 2.6.26.

Both F8 and F9 should get this version since that kernel version is going to be pushed to them soon.

I tried iproute-2.6.26 on my F-9 box with 2.6.25.11-97.fc9.i686 and no luck. I think it was something from tc also fixed in kernel. I'll push this update of iproute into testing and we'll see whether kernel and iproute fix this issue.

iproute-2.6.26-1.fc9 has been submitted as an update for Fedora 9

I have not had a chance to check out iproute-2.6.26-1.fc9  (in fact I'm having trouble finding it), but my testing of a vanilla version if iproute2 2.6.26 on a vanilla 2.5.26-2 kernel indicates that the problem still exists on that pair.

If I run the following script right out of the netem page (http://www.linuxfoundation.org/en/Net:Netem) the kernel no longer emits error messages but the filters are not set up.  But if I change the script to use a version of 'tc' compiled using the iproute2-2.6.24-rc7 code base then things seem happy.

#!/bin/sh

DEV=eth1
TC=/usr/sbin/tc

echo ${TC} filter del dev ${DEV} prio 3
${TC} filter del dev ${DEV} prio 3
echo ${TC} filter add dev ${DEV} parent 1:0 prio 3 protocol ip u32
${TC} filter add dev ${DEV} parent 1:0 prio 3 protocol ip u32
echo ${TC} filter add dev ${DEV} parent 1:0 prio 3 handle 2: u32 divisor 1
${TC} filter add dev ${DEV} parent 1:0 prio 3 handle 2: u32 divisor 1
echo ${TC} filter add dev ${DEV} parent 1:0 handle ::1 prio 3 u32 match u8 0x40 0xF0 at 0 offset mask 0x0F00 at 0 shift 6 plus 0 ht 800:: link 2:
${TC} filter add dev ${DEV} parent 1:0 handle ::1 prio 3 u32 match u8 0x40 0xF0 at 0 offset mask 0x0F00 at 0 shift 6 plus 0 ht 800:: link 2:
echo ${TC} filter add dev ${DEV} protocol ip parent 1:0 prio 3 u32 match u8 0x11 0xff at 9 flowid 1:4 ht 2:0:
${TC} filter add dev ${DEV} protocol ip parent 1:0 prio 3 u32 match u8 0x11 0xff at 9 flowid 1:4 ht 2:0:

echo ${TC} qdisc show dev ${DEV}
${TC} qdisc show dev ${DEV}
echo ${TC} class show dev ${DEV}
${TC} class show dev ${DEV}
echo ${TC} filter show dev ${DEV}
${TC} filter show dev ${DEV}

The problem is also in rawhide: kernel-2.6.27 and latest iproute2 from upstream git.

iproute-2.6.26-1.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update iproute'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-7583

Ok, this update didn't fix tc.

Could you tell me please which kernel do you have with working iproute2-2.6.24-rc7? I tried different versions of kernel and iproute and now I can't find any functional pair ;-)

/tmp/iproute/devel/iproute-2.6.24/iproute2-2.6.24/tc/tc filter add dev eth0 parent 1:0 prio 3 protocol ip u32
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

Your scripts are missing new mandatory argument "protocol". 

I changed the error message from "We have an error talking to the kernel" to "protocol is required" and I'll send it to upstream soon.

This should be fixed in iproute-2.6.27.

iproute-2.6.27-1.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/iproute-2.6.27-1.fc9

iproute-2.6.27-1.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing-newkey update iproute'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-10904

iproute-2.6.27-1.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.