Bug 1730264

Summary: VMs will fail to start if the vnic profile attached is having port mirroring enabled and have name greater than 15 characters
Product: Red Hat Enterprise Virtualization Manager Reporter: nijin ashok <nashok>
Component: ovirt-engineAssignee: Ales Musil <amusil>
Status: CLOSED ERRATA QA Contact: Michael Burman <mburman>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.4CC: amusil, dholler, edwardh, mburman, pelauter, rdlugyhe
Target Milestone: ovirt-4.4.0Keywords: ZStream
Target Release: 4.4.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, enabling port mirroring on networks whose user-visible name was longer than 15 characters failed. This happened because port mirroring tried to use this long user-visible network name, which was not a valid network name. The current release fixes this issue. Now, instead of the user-visible name, port mirroring uses the VDSM network name. Therefore, you can enable port mirroring for networks whose user-visible name is longer than 15 characters.
Story Points: ---
Clone Of:
: 1744571 (view as bug list) Environment:
Last Closed: 2020-08-04 13:19:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1765018    
Bug Blocks: 1744571    

Description nijin ashok 2019-07-16 11:08:32 UTC
Description of problem:

If the network name is greater than 15 characters, the original name of the network will be  mapped to a different name while creating the bridge.


====
engine=# select name,vdsm_name from network where name='verylongnetworkname';
        name         |    vdsm_name    
---------------------+-----------------
 verylongnetworkname | on5604266439c64
(1 row)
====

However if the port mirroring is enabled, the vdsm still send setPortMirroring request to supervdsm with the "network name" instead of "vdsm_name". Because of this, the VM will fail to start with the error below in vdsm log since the bridge name "verylongnetworkname" doesn't exist.

====
2019-07-16 16:21:17,460+0530 ERROR (vm/b532b3ee) [virt.vm] (vmId='b532b3ee-c5f5-45e5-a1b9-136a7804b04b') The vm start process failed (vm:933)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 867, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2891, in _run
    self._domDependentInit()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2466, in _domDependentInit
    self._vmDependentInit()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2487, in _vmDependentInit
    supervdsm.getProxy().setPortMirroring(network, nic.name)
  File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__
    return callMethod()
  File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>
    **kwargs)
  File "<string>", line 2, in setPortMirroring
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
TrafficControlException: (1, 'Cannot find device "verylongnetwork"\n', ['/sbin/tc', 'qdisc', 'add', 'dev', 'verylongnetworkname', 'ingress'])
=====

Supervdsm log shows error below.

====
MainProcess|vm/b532b3ee::ERROR::2019-07-16 16:21:17,459::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in setPortMirroring
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/tc/__init__.py", line 80, in setPortMirroring
    _qdisc_replace_ingress(network)
  File "/usr/lib/python2.7/site-packages/vdsm/network/tc/__init__.py", line 116, in _qdisc_replace_ingress
    qdisc.add(dev, 'ingress')
  File "/usr/lib/python2.7/site-packages/vdsm/network/tc/qdisc.py", line 43, in add
    _wrapper.process_request(command)
  File "/usr/lib/python2.7/site-packages/vdsm/network/tc/_wrapper.py", line 43, in process_request
    raise TrafficControlException(retcode, err, command)
TrafficControlException: (1, 'Cannot find device "verylongnetwork"\n', ['/sbin/tc', 'qdisc', 'add', 'dev', 'verylongnetworkname', 'ingress'])
====

Version-Release number of selected component (if applicable):

vdsm-4.30.17-1.el7ev.x86_64
rhvm-4.3.4.3-0.1.el7.noarch

How reproducible:

100%

Steps to Reproduce:

1. Create a network with name greater than 15 characters.
2. Enable port mirroring.
3. Attach this network to the VM and start it.

Actual results:

VMs will fail to start if the vnic profile attached is having port mirroring enabled and have name greater than 15 characters

Expected results:

VM should start.

Additional info:

Comment 2 Michael Burman 2019-07-23 06:55:09 UTC
QE managed to reproduce.
Edy, can you please take a look? is it vdsm to handle or is it engine side? 

Thanks,

Comment 3 Edward Haas 2019-07-24 10:26:42 UTC
(In reply to Michael Burman from comment #2)
> QE managed to reproduce.
> Edy, can you please take a look? is it vdsm to handle or is it engine side? 
> 
> Thanks,

Looks like Engine side. The long-name should never reach VDSM in the current solution.

Comment 8 Daniel Gur 2019-08-28 13:14:33 UTC
sync2jira

Comment 9 Daniel Gur 2019-08-28 13:19:35 UTC
sync2jira

Comment 11 Michael Burman 2019-10-23 12:02:05 UTC
I have no idea how this is possible, but the issue reproduced on rhv-4.4.0
4.4.0-0.4.master.el7
vdsm-4.40.0-127.gitc628cce.el8ev.x86_64
rhel8.1

MainProcess|vm/f1473d49::ERROR::2019-10-23 07:43:13,779::supervdsm_server::96::SuperVdsm.ServerCallback::(wrapper) Error in setPortMirroring
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 94, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/__init__.py", line 83, in setPortMirroring
    _qdisc_replace_ingress(network)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/__init__.py", line 119, in _qdisc_replace_ingress
    qdisc.add(dev, 'ingress')
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/qdisc.py", line 43, in add
    _wrapper.process_request(command)
  File "/usr/lib/python3.6/site-packages/vdsm/network/tc/_wrapper.py", line 44, in process_request
    raise TrafficControlException(retcode, err, command)
vdsm.network.tc._wrapper.TrafficControlException: (2, 'Error: Exclusivity flag on, cannot modify.\n', ['/sbin/tc', 'qdisc', 'add', 'dev', 'ond0985e33f0f94', 'ingress'])
MainProcess|jsonrpc/0::DEBUG::2019-10-23 07:43:13,789::supervdsm_server::92::SuperVdsm.ServerCallback::(wrapper) call unsetPortMirroring with ('ond0985e33f0f94', 'vnet1') {}

Comment 12 Dominik Holler 2019-10-23 15:39:22 UTC
This is a new issue, probably rhel8 related:
the initial reported error was:
TrafficControlException: (1, 'Cannot find device "verylongnetwork"\n', ['/sbin/tc', 'qdisc', 'add', 'dev', 'verylongnetworkname', 'ingress'])
the new error is:
TrafficControlException: (2, 'Error: Exclusivity flag on, cannot modify.\n', ['/sbin/tc', 'qdisc', 'add', 'dev', 'ond0985e33f0f94', 'ingress'])

But never the less we should fix this.

Comment 13 Dominik Holler 2019-10-23 15:58:53 UTC
Networks with short names are affected, too.

Comment 14 Ales Musil 2019-10-24 06:25:44 UTC
That is different issue and IIRC we had discussion some time ago that tc is not working as vdsm expects.

Comment 15 Michael Burman 2019-10-24 07:10:53 UTC
(In reply to Ales Musil from comment #14)
> That is different issue and IIRC we had discussion some time ago that tc is
> not working as vdsm expects.

ack
Do you want new bug for that which will block this one?

Comment 16 Ales Musil 2019-10-24 07:14:30 UTC
Yes please, to keep it separate.

Comment 18 RHV bug bot 2019-12-13 13:15:59 UTC
WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 19 Michael Burman 2019-12-16 06:41:58 UTC
Verified on - vdsm-4.40.0-164.git38a19bb.el8ev.x86_64 with rhvm-4.4.0-0.9.master.el7.noarch

Comment 20 RHV bug bot 2019-12-20 17:45:31 UTC
WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 21 RHV bug bot 2020-01-08 14:47:47 UTC
WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 22 RHV bug bot 2020-01-08 15:17:24 UTC
WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 23 RHV bug bot 2020-01-24 19:49:30 UTC
WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 31 errata-xmlrpc 2020-08-04 13:19:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: RHV Manager (ovirt-engine) 4.4 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3247