Bug 1765018 - [rhel8.1] VM fail to start if having vNIC profile with port mirroring enabled
Summary: [rhel8.1] VM fail to start if having vNIC profile with port mirroring enabled
Keywords:
Status: VERIFIED
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.40.0
Hardware: x86_64
OS: Linux
high
high vote
Target Milestone: ovirt-4.4.0
: ---
Assignee: Bell Levin
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On:
Blocks: 1730264
TreeView+ depends on / blocked
 
Reported: 2019-10-24 07:30 UTC by Michael Burman
Modified: 2019-12-16 06:24 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
oVirt Team: Network
mperina: ovirt-4.4?
mperina: blocker?


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 105263 master MERGED net: Add exception handling for double mirroring 2019-12-02 12:39:32 UTC

Description Michael Burman 2019-10-24 07:30:02 UTC
Description of problem:
[rhel8.1] VM fail to start if having vNIC profile with port mirroring enabled

MainProcess|vm/f1473d49::ERROR::2019-10-23 07:43:13,779::supervdsm_server::96::SuperVdsm.ServerCallback::(wrapper) Error in setPortMirroring
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 94, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/__init__.py", line 83, in setPortMirroring
    _qdisc_replace_ingress(network)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/__init__.py", line 119, in _qdisc_replace_ingress
    qdisc.add(dev, 'ingress')
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/qdisc.py", line 43, in add
    _wrapper.process_request(command)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/_wrapper.py", line 44, in process_request
    raise TrafficControlException(retcode, err, command)
vdsm.network.tc._wrapper.TrafficControlException: (2, 'Error: Exclusivity flag on, cannot modify.\n', ['/sbin/tc', 'qdisc', 'add', 'dev', 'ond0985e33f0f94', 'ingress'])
MainProcess|jsonrpc/0::DEBUG::2019-10-23 07:43:13,789::supervdsm_server::92::SuperVdsm.ServerCallback::(wrapper) call unsetPortMirroring with ('ond0985e33f0f94', 'vnet1') {}

Version-Release number of selected component (if applicable):
vdsm-4.40.0-127.gitc628cce.el8ev.x86_64
rhel8.1

How reproducible:
100%

Steps to Reproduce:
1. Try to start VM with port mirroring vNIC profile


Actual results:
Fail with trace error on vdsm

Expected results:
Should work

Comment 2 Michael Burman 2019-10-24 08:54:41 UTC
Note that the issue reproduced when you have more than 1 VM with port mirroring enabled vNIC. 1 VM can start, the second VM fail to start.

Comment 3 Bell Levin 2019-12-01 16:37:19 UTC
This is not py3 error.
The problem here is the difference in tc (probably?)

if we have a vm with two interfaces, each has a vnic profile with port mirroring enabled on it - the port mirroring will be enabled twice on the bridge. (The code is not double mirroring proof).

A way to reproduce it would be:
1) ip link add br0 type bridge
2) tc qdisc add dev br0 ingress
3) tc qdisc add dev br0 ingress

On CentOS we would get an error, but it will not stop and fail the tests:

cmdutils.py    130 DEBUG    /sbin/tc qdisc add dev target-P1Pf ingress (cwd None)
cmdutils.py    138 DEBUG    FAILED: <err> = 'RTNETLINK answers: File exists\n'; <rc> = 2


But on fedora 30, we would get the error as described above, and the test fails:

root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None)
root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot modify.\n'; <rc> = 2


Since looks like this error is only about sending the tc command twice, and does not look like it is hurting any functionality, maybe the right thing to do here is pass the exception.
Will consult with Dominik tomorrow.

Comment 4 Dominik Holler 2019-12-02 07:50:01 UTC
(In reply to Bell Levin from comment #3)
> This is not py3 error.
> The problem here is the difference in tc (probably?)
> 
> if we have a vm with two interfaces, each has a vnic profile with port
> mirroring enabled on it - the port mirroring will be enabled twice on the
> bridge. (The code is not double mirroring proof).
> 
> A way to reproduce it would be:
> 1) ip link add br0 type bridge
> 2) tc qdisc add dev br0 ingress
> 3) tc qdisc add dev br0 ingress
> 
> On CentOS we would get an error, but it will not stop and fail the tests:
> 
> cmdutils.py    130 DEBUG    /sbin/tc qdisc add dev target-P1Pf ingress (cwd
> None)
> cmdutils.py    138 DEBUG    FAILED: <err> = 'RTNETLINK answers: File
> exists\n'; <rc> = 2
> 
> 
> But on fedora 30, we would get the error as described above, and the test
> fails:
> 
> root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None)
> root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot
> modify.\n'; <rc> = 2
> 

Does this fail RHEL 8.1 like on fedora 30?


> 
> Since looks like this error is only about sending the tc command twice, and
> does not look like it is hurting any functionality, maybe the right thing to
> do here is pass the exception.
> Will consult with Dominik tomorrow.

Comment 5 Bell Levin 2019-12-02 08:11:16 UTC
(In reply to Dominik Holler from comment #4)
> (In reply to Bell Levin from comment #3)
> > This is not py3 error.
> > The problem here is the difference in tc (probably?)
> > 
> > if we have a vm with two interfaces, each has a vnic profile with port
> > mirroring enabled on it - the port mirroring will be enabled twice on the
> > bridge. (The code is not double mirroring proof).
> > 
> > A way to reproduce it would be:
> > 1) ip link add br0 type bridge
> > 2) tc qdisc add dev br0 ingress
> > 3) tc qdisc add dev br0 ingress
> > 
> > On CentOS we would get an error, but it will not stop and fail the tests:
> > 
> > cmdutils.py    130 DEBUG    /sbin/tc qdisc add dev target-P1Pf ingress (cwd
> > None)
> > cmdutils.py    138 DEBUG    FAILED: <err> = 'RTNETLINK answers: File
> > exists\n'; <rc> = 2
> > 
> > 
> > But on fedora 30, we would get the error as described above, and the test
> > fails:
> > 
> > root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None)
> > root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot
> > modify.\n'; <rc> = 2
> > 
> 
> Does this fail RHEL 8.1 like on fedora 30?
> 

Yes, I get exactly the same exception.

> 
> > 
> > Since looks like this error is only about sending the tc command twice, and
> > does not look like it is hurting any functionality, maybe the right thing to
> > do here is pass the exception.
> > Will consult with Dominik tomorrow.

Comment 6 Michael Burman 2019-12-16 06:24:58 UTC
Verified on - vdsm-4.40.0-164.git38a19bb.el8ev.x86_64 with rhvm-4.4.0-0.9.master.el7.noarch


Note You need to log in before you can comment on or make changes to this bug.