Hide Forgot
Description of problem: [rhel8.1] VM fail to start if having vNIC profile with port mirroring enabled MainProcess|vm/f1473d49::ERROR::2019-10-23 07:43:13,779::supervdsm_server::96::SuperVdsm.ServerCallback::(wrapper) Error in setPortMirroring Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 94, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/tc/__init__.py", line 83, in setPortMirroring _qdisc_replace_ingress(network) File "/usr/lib/python3.6/site-packages/vdsm/network/tc/__init__.py", line 119, in _qdisc_replace_ingress qdisc.add(dev, 'ingress') File "/usr/lib/python3.6/site-packages/vdsm/network/tc/qdisc.py", line 43, in add _wrapper.process_request(command) File "/usr/lib/python3.6/site-packages/vdsm/network/tc/_wrapper.py", line 44, in process_request raise TrafficControlException(retcode, err, command) vdsm.network.tc._wrapper.TrafficControlException: (2, 'Error: Exclusivity flag on, cannot modify.\n', ['/sbin/tc', 'qdisc', 'add', 'dev', 'ond0985e33f0f94', 'ingress']) MainProcess|jsonrpc/0::DEBUG::2019-10-23 07:43:13,789::supervdsm_server::92::SuperVdsm.ServerCallback::(wrapper) call unsetPortMirroring with ('ond0985e33f0f94', 'vnet1') {} Version-Release number of selected component (if applicable): vdsm-4.40.0-127.gitc628cce.el8ev.x86_64 rhel8.1 How reproducible: 100% Steps to Reproduce: 1. Try to start VM with port mirroring vNIC profile Actual results: Fail with trace error on vdsm Expected results: Should work
Note that the issue reproduced when you have more than 1 VM with port mirroring enabled vNIC. 1 VM can start, the second VM fail to start.
This is not py3 error. The problem here is the difference in tc (probably?) if we have a vm with two interfaces, each has a vnic profile with port mirroring enabled on it - the port mirroring will be enabled twice on the bridge. (The code is not double mirroring proof). A way to reproduce it would be: 1) ip link add br0 type bridge 2) tc qdisc add dev br0 ingress 3) tc qdisc add dev br0 ingress On CentOS we would get an error, but it will not stop and fail the tests: cmdutils.py 130 DEBUG /sbin/tc qdisc add dev target-P1Pf ingress (cwd None) cmdutils.py 138 DEBUG FAILED: <err> = 'RTNETLINK answers: File exists\n'; <rc> = 2 But on fedora 30, we would get the error as described above, and the test fails: root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None) root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot modify.\n'; <rc> = 2 Since looks like this error is only about sending the tc command twice, and does not look like it is hurting any functionality, maybe the right thing to do here is pass the exception. Will consult with Dominik tomorrow.
(In reply to Bell Levin from comment #3) > This is not py3 error. > The problem here is the difference in tc (probably?) > > if we have a vm with two interfaces, each has a vnic profile with port > mirroring enabled on it - the port mirroring will be enabled twice on the > bridge. (The code is not double mirroring proof). > > A way to reproduce it would be: > 1) ip link add br0 type bridge > 2) tc qdisc add dev br0 ingress > 3) tc qdisc add dev br0 ingress > > On CentOS we would get an error, but it will not stop and fail the tests: > > cmdutils.py 130 DEBUG /sbin/tc qdisc add dev target-P1Pf ingress (cwd > None) > cmdutils.py 138 DEBUG FAILED: <err> = 'RTNETLINK answers: File > exists\n'; <rc> = 2 > > > But on fedora 30, we would get the error as described above, and the test > fails: > > root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None) > root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot > modify.\n'; <rc> = 2 > Does this fail RHEL 8.1 like on fedora 30? > > Since looks like this error is only about sending the tc command twice, and > does not look like it is hurting any functionality, maybe the right thing to > do here is pass the exception. > Will consult with Dominik tomorrow.
(In reply to Dominik Holler from comment #4) > (In reply to Bell Levin from comment #3) > > This is not py3 error. > > The problem here is the difference in tc (probably?) > > > > if we have a vm with two interfaces, each has a vnic profile with port > > mirroring enabled on it - the port mirroring will be enabled twice on the > > bridge. (The code is not double mirroring proof). > > > > A way to reproduce it would be: > > 1) ip link add br0 type bridge > > 2) tc qdisc add dev br0 ingress > > 3) tc qdisc add dev br0 ingress > > > > On CentOS we would get an error, but it will not stop and fail the tests: > > > > cmdutils.py 130 DEBUG /sbin/tc qdisc add dev target-P1Pf ingress (cwd > > None) > > cmdutils.py 138 DEBUG FAILED: <err> = 'RTNETLINK answers: File > > exists\n'; <rc> = 2 > > > > > > But on fedora 30, we would get the error as described above, and the test > > fails: > > > > root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None) > > root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot > > modify.\n'; <rc> = 2 > > > > Does this fail RHEL 8.1 like on fedora 30? > Yes, I get exactly the same exception. > > > > > Since looks like this error is only about sending the tc command twice, and > > does not look like it is hurting any functionality, maybe the right thing to > > do here is pass the exception. > > Will consult with Dominik tomorrow.
Verified on - vdsm-4.40.0-164.git38a19bb.el8ev.x86_64 with rhvm-4.4.0-0.9.master.el7.noarch