Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1765018

Summary:	[rhel8.1] VM fail to start if having vNIC profile with port mirroring enabled
Product:	[oVirt] vdsm	Reporter:	Michael Burman <mburman>
Component:	Core	Assignee:	Bell Levin <blevin>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Michael Burman <mburman>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.40.0	CC:	amusil, blevin, bugs, dholler, mperina
Target Milestone:	ovirt-4.4.0	Flags:	mperina: ovirt-4.4? mperina: blocker?
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-05-20 20:01:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Network	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1730264

Description Michael Burman 2019-10-24 07:30:02 UTC

Description of problem:
[rhel8.1] VM fail to start if having vNIC profile with port mirroring enabled

MainProcess|vm/f1473d49::ERROR::2019-10-23 07:43:13,779::supervdsm_server::96::SuperVdsm.ServerCallback::(wrapper) Error in setPortMirroring
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 94, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/__init__.py", line 83, in setPortMirroring
    _qdisc_replace_ingress(network)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/__init__.py", line 119, in _qdisc_replace_ingress
    qdisc.add(dev, 'ingress')
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/qdisc.py", line 43, in add
    _wrapper.process_request(command)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/_wrapper.py", line 44, in process_request
    raise TrafficControlException(retcode, err, command)
vdsm.network.tc._wrapper.TrafficControlException: (2, 'Error: Exclusivity flag on, cannot modify.\n', ['/sbin/tc', 'qdisc', 'add', 'dev', 'ond0985e33f0f94', 'ingress'])
MainProcess|jsonrpc/0::DEBUG::2019-10-23 07:43:13,789::supervdsm_server::92::SuperVdsm.ServerCallback::(wrapper) call unsetPortMirroring with ('ond0985e33f0f94', 'vnet1') {}

Version-Release number of selected component (if applicable):
vdsm-4.40.0-127.gitc628cce.el8ev.x86_64
rhel8.1

How reproducible:
100%

Steps to Reproduce:
1. Try to start VM with port mirroring vNIC profile


Actual results:
Fail with trace error on vdsm

Expected results:
Should work

Comment 2 Michael Burman 2019-10-24 08:54:41 UTC

Note that the issue reproduced when you have more than 1 VM with port mirroring enabled vNIC. 1 VM can start, the second VM fail to start.

Comment 3 Bell Levin 2019-12-01 16:37:19 UTC

This is not py3 error.
The problem here is the difference in tc (probably?)

if we have a vm with two interfaces, each has a vnic profile with port mirroring enabled on it - the port mirroring will be enabled twice on the bridge. (The code is not double mirroring proof).

A way to reproduce it would be:
1) ip link add br0 type bridge
2) tc qdisc add dev br0 ingress
3) tc qdisc add dev br0 ingress

On CentOS we would get an error, but it will not stop and fail the tests:

cmdutils.py    130 DEBUG    /sbin/tc qdisc add dev target-P1Pf ingress (cwd None)
cmdutils.py    138 DEBUG    FAILED: <err> = 'RTNETLINK answers: File exists\n'; <rc> = 2


But on fedora 30, we would get the error as described above, and the test fails:

root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None)
root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot modify.\n'; <rc> = 2


Since looks like this error is only about sending the tc command twice, and does not look like it is hurting any functionality, maybe the right thing to do here is pass the exception.
Will consult with Dominik tomorrow.

Comment 4 Dominik Holler 2019-12-02 07:50:01 UTC

(In reply to Bell Levin from comment #3)
> This is not py3 error.
> The problem here is the difference in tc (probably?)
> 
> if we have a vm with two interfaces, each has a vnic profile with port
> mirroring enabled on it - the port mirroring will be enabled twice on the
> bridge. (The code is not double mirroring proof).
> 
> A way to reproduce it would be:
> 1) ip link add br0 type bridge
> 2) tc qdisc add dev br0 ingress
> 3) tc qdisc add dev br0 ingress
> 
> On CentOS we would get an error, but it will not stop and fail the tests:
> 
> cmdutils.py    130 DEBUG    /sbin/tc qdisc add dev target-P1Pf ingress (cwd
> None)
> cmdutils.py    138 DEBUG    FAILED: <err> = 'RTNETLINK answers: File
> exists\n'; <rc> = 2
> 
> 
> But on fedora 30, we would get the error as described above, and the test
> fails:
> 
> root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None)
> root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot
> modify.\n'; <rc> = 2
> 

Does this fail RHEL 8.1 like on fedora 30?


> 
> Since looks like this error is only about sending the tc command twice, and
> does not look like it is hurting any functionality, maybe the right thing to
> do here is pass the exception.
> Will consult with Dominik tomorrow.

Comment 5 Bell Levin 2019-12-02 08:11:16 UTC

(In reply to Dominik Holler from comment #4)
> (In reply to Bell Levin from comment #3)
> > This is not py3 error.
> > The problem here is the difference in tc (probably?)
> > 
> > if we have a vm with two interfaces, each has a vnic profile with port
> > mirroring enabled on it - the port mirroring will be enabled twice on the
> > bridge. (The code is not double mirroring proof).
> > 
> > A way to reproduce it would be:
> > 1) ip link add br0 type bridge
> > 2) tc qdisc add dev br0 ingress
> > 3) tc qdisc add dev br0 ingress
> > 
> > On CentOS we would get an error, but it will not stop and fail the tests:
> > 
> > cmdutils.py    130 DEBUG    /sbin/tc qdisc add dev target-P1Pf ingress (cwd
> > None)
> > cmdutils.py    138 DEBUG    FAILED: <err> = 'RTNETLINK answers: File
> > exists\n'; <rc> = 2
> > 
> > 
> > But on fedora 30, we would get the error as described above, and the test
> > fails:
> > 
> > root:cmdutils.py:130 /sbin/tc qdisc add dev target-KoNM ingress (cwd None)
> > root:cmdutils.py:138 FAILED: <err> = b'Error: Exclusivity flag on, cannot
> > modify.\n'; <rc> = 2
> > 
> 
> Does this fail RHEL 8.1 like on fedora 30?
> 

Yes, I get exactly the same exception.

> 
> > 
> > Since looks like this error is only about sending the tc command twice, and
> > does not look like it is hurting any functionality, maybe the right thing to
> > do here is pass the exception.
> > Will consult with Dominik tomorrow.

Comment 6 Michael Burman 2019-12-16 06:24:58 UTC

Verified on - vdsm-4.40.0-164.git38a19bb.el8ev.x86_64 with rhvm-4.4.0-0.9.master.el7.noarch

Comment 7 Sandro Bonazzola 2020-05-20 20:01:08 UTC

This bugzilla is included in oVirt 4.4.0 release, published on May 20th 2020.

Since the problem described in this bug report should be
resolved in oVirt 4.4.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.