Bug 1544217 - fc27: Cannot set port mirroring onto two interface [NEEDINFO]
Summary: fc27: Cannot set port mirroring onto two interface
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 27
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL: git://git.engineering.redhat.com/user...
Whiteboard:
Depends On: 1542013
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-11 12:11 UTC by Dan Kenigsberg
Modified: 2018-08-29 14:57 UTC (History)
32 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1542013
Environment:
Last Closed: 2018-08-29 14:57:06 UTC
Type: Bug
Embargoed:
jforbes: needinfo?


Attachments (Terms of Use)

Description Dan Kenigsberg 2018-02-11 12:11:10 UTC
Please fore-port the kernel fix to Fedora, as we see a similar error there.

+++ This bug was initially created as a clone of Bug #1542013 +++

Description of problem:
Can't run VM with port mirroring if another VM with port mirroring is already running on the host.

If trying to run VM with port mirroring vNIC and we have already a running VM with port mirroring running on the host we fail with:

2018-02-05 13:49:02,560+0200 ERROR (jsonrpc/1) [api] FINISH destroy error=(22, 'RTNETLINK answers: Invalid argument', ['/sbin/tc', 'filter', 'replace', 'dev', 'pm1', 'protocol', 'all', 'parent', 'ffff:', 'handle',
 '800::800', 'pref', '49152', 'u32', 'match', 'u8', '0', '0', 'action', 'mirred', 'egress', 'mirror', 'dev', 'vnet0']) (api:127)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 117, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 311, in destroy
    res = self.vm.destroy(gracefulAttempts)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5131, in destroy
    result = self.doDestroy(gracefulAttempts, reason)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5150, in doDestroy
    return self.releaseVm(gracefulAttempts)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5032, in releaseVm
    nic.name)
  File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__
    return callMethod()
  File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda>
    **kwargs)
  File "<string>", line 2, in unsetPortMirroring
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
TrafficControlException: (22, 'RTNETLINK answers: Invalid argument', ['/sbin/tc', 'filter', 'replace', 'dev', 'pm1', 'protocol', 'all', 'parent', 'ffff:', 'handle', '800::800', 'pref', '49152', 'u32', 'match', 'u8', '0', '0', 'action', 'mirred', 'egress', 'mirror', 'dev', 'vnet0'])
2018-02-05 13:49:02,577+0200 INFO  (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("(22, \'RTNETLINK answers: Invalid argument\', [\'/sbin/tc\', \'filter\', \'replace\', \'dev\', \'pm1\', \'protocol\', \'all\', \'parent\', \'ffff:\', \'handle\', \'800::800\', \'pref\', \'49152\', \'u32\', \'match\', \'u8\', \'0\', \'0\', \'action\', \'mirred\', \'egress\', \'mirror\', \'dev\', \'vnet0\'])",)', 'code': 100}} from=::ffff:10.35.163.149,37508 (api:52)

After the vM failed to run, on the host it is reproted as running and reboot required to release it. 

[root@camel-vdsa ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------
 9     V1                             running
 10    V2                             running

VM V2 is failed to run. 

Version-Release number of selected component (if applicable):
vdsm-4.20.17-1.el7ev.x86_64
kernel-3.10.0-830.el7.x86_64

How reproducible:
100

Steps to Reproduce:
1. Create network with port mirroring vNIC profile and attach to the host 
2. Run VM1 with port mirroring vNIC 
3. Try to run VM2 with port mirroring vNIC

Actual results:
Failed with tc error

Expected results:
Should work

--- Additional comment from Red Hat Bugzilla Rules Engine on 2018-02-05 14:18:00 IST ---

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

--- Additional comment from Michael Burman on 2018-02-05 17:11:34 IST ---

This passes on el7.4 (3.10.0-693.el7.x86_64), explodes in el7.5 (3.10.0-830.el7.x86_64)
iproute-3.10.0-87.el7.x86_64 version running on both kernels 


ip link add type veth
ip link add type veth
brctl delbr pm2
brctl addbr pm2

/sbin/tc qdisc add dev pm2 ingress
/sbin/tc filter show dev pm2 parent ffff:
/sbin/tc filter replace dev pm2 protocol all parent ffff: u32 match u8 0 0 action mirred egress mirror dev veth0
/sbin/tc qdisc replace dev pm2 root prio
qd=`/sbin/tc qdisc show dev pm2 |grep '^qdisc prio ' |sed 's/qdisc prio //;s/: .*//'`

/sbin/tc filter show dev pm2 parent "$qd":
/sbin/tc filter replace dev pm2 protocol all parent "$qd": u32 match u8 0 0 action mirred egress mirror dev veth0
/sbin/ip link set dev pm2 promisc on

/sbin/tc qdisc add dev pm2 ingress || :
/sbin/tc filter show dev pm2 parent ffff:
/sbin/tc filter replace dev pm2 protocol all parent ffff: handle 800::800 pref 49152 u32 match u8 0 0 action mirred egress mirror dev veth0 action mirred egress mirror dev veth2


[root@camel-vdsa ~]# bash -ex burman.sh
+ brctl delbr pm2
+ brctl addbr pm2
+ /sbin/tc qdisc add dev pm2 ingress
+ /sbin/tc filter show dev pm2 parent ffff:
+ /sbin/tc filter replace dev pm2 protocol all parent ffff: u32 match u8 0 0 action mirred egress mirror dev veth0
+ /sbin/tc qdisc replace dev pm2 root prio
++ /sbin/tc qdisc show dev pm2
++ grep '^qdisc prio '
++ sed 's/qdisc prio //;s/: .*//'
+ qd=8010
+ /sbin/tc filter show dev pm2 parent 8010:
+ /sbin/tc filter replace dev pm2 protocol all parent 8010: u32 match u8 0 0 action mirred egress mirror dev veth0
+ /sbin/ip link set dev pm2 promisc on
+ /sbin/tc qdisc add dev pm2 ingress
RTNETLINK answers: File exists
+ :
+ /sbin/tc filter show dev pm2 parent ffff:
filter protocol all pref 49152 u32 
filter protocol all pref 49152 u32 fh 800: ht divisor 1 
filter protocol all pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? not_in_hw 
  match 00000000/00000000 at 0
        action order 1: mirred (Egress Mirror to device veth0) pipe
        index 1 ref 1 bind 1
 
+ /sbin/tc filter replace dev pm2 protocol all parent ffff: handle 800::800 pref 49152 u32 match u8 0 0 action mirred egress mirror dev veth0 action mirred egress mirror dev veth2
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

--- Additional comment from Ivan Vecera on 2018-02-08 15:14:56 IST ---

The issue is caused by commit:

commit 24d3dc6d27eae19f422a5e216e25d3a16628d4ff
Author: Or Gerlitz <ogerlitz>
Date:   Thu Feb 16 10:31:15 2017 +0200

    net/sched: cls_u32: Reflect HW offload status
    
    U32 support for the "in hw" offloading flags.
    
    Signed-off-by: Or Gerlitz <ogerlitz>
    Reviewed-by: Amir Vadai <amir>
    Signed-off-by: David S. Miller <davem>

This commit added TCA_CLS_FLAGS_{,NOT}_IN_HW flags to u32 but the conditional in u32_change() is too strict and causes impossibility to replace existing filter:

static int u32_change(struct net *net, struct sk_buff *in_skb,
                      struct tcf_proto *tp, unsigned long base, u32 handle,
                      struct nlattr **tca, void **arg, bool ovr,
                      struct netlink_ext_ack *extack)
{
...
                if (n->flags != flags) {
                        NL_SET_ERR_MSG_MOD(extack, "Key node flags do not match passed flags");
                        return -EINVAL;
                }
...
}

The n->flags contains either ...IN_HW or ...NOT_IN_HW according offloading state. These flags cannot be passed from userspace so the passed flags cannot contain them and the conditional cannot be true.

The upstream is affected as well so I'm going to fix it first.

--- Additional comment from Ivan Vecera on 2018-02-08 17:12:21 IST ---

Upstream patch submitted:

https://patchwork.ozlabs.org/patch/870905/

--- Additional comment from Ivan Vecera on 2018-02-09 15:09:02 IST ---

(In reply to Ivan Vecera from comment #8)
> Upstream patch submitted:
> 
> https://patchwork.ozlabs.org/patch/870905/

Accepted.

Comment 1 Laura Abbott 2018-02-12 17:06:03 UTC
The fix is listed as queued for stable so it should end up in Fedora via that path.

Comment 2 Justin M. Forbes 2018-07-23 14:58:00 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 27 kernel bugs.

Fedora 27 has now been rebased to 4.17.7-100.fc27.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 28, and are still experiencing this issue, please change the version to Fedora 28.

If you experience different issues, please open a new bug report for those.

Comment 3 Justin M. Forbes 2018-08-29 14:57:06 UTC
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 5 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.


Note You need to log in before you can comment on or make changes to this bug.