Bug 847518

Summary: 3.1 [Port mirroring] Failed to start second VM with port mirroring enabled while another VM with port mirroring enabled is already running
Product: Red Hat Enterprise Linux 6 Reporter: GenadiC <gcheresh>
Component: vdsmAssignee: Dan Kenigsberg <danken>
Status: CLOSED ERRATA QA Contact: GenadiC <gcheresh>
Severity: high Docs Contact:
Priority: high    
Version: 6.3CC: abaron, bazulay, iheim, ilvovsky, lpeer, ykaul
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: network
Fixed In Version: vdsm-4.9.6-32.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-04 19:05:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsm log none

Description GenadiC 2012-08-12 14:42:07 UTC
Created attachment 603788 [details]
vdsm log

Description of problem:
Enable port mirroring on one VM and run it. Then enable port mirroring on another VM and try to run it. It will fail with "The vm start process failed" error message

Steps to Reproduce:
1. Enable port mirroring on several NICs of one VM and run it
2. After some time enable port mirroring on several NICs of another VM and run it

Actual results:
The second VM will fail to run

Expected results:
The second VM is supposed to run with port mirroring enabled

Additional info:
MainProcess|Thread-277295::DEBUG::2012-08-12 17:23:45,675::__init__::1164::Storage.Misc.excCmd::(_log) '/sbin/tc qdisc add dev rhevm ingress' (cwd None)
MainProcess|Thread-277295::DEBUG::2012-08-12 17:23:45,703::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = 'RTNETLINK answers: File exists\n'; <rc> = 2
MainProcess|Thread-277295::ERROR::2012-08-12 17:23:45,704::supervdsmServer::61::SuperVdsm.ServerCallback::(wrapper) Error in setPortMirroring
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 59, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 202, in setPortMirroring
    tc.setPortMirroring(networkName, ifaceName)
  File "/usr/share/vdsm/tc.py", line 36, in setPortMirroring
    qdisc_add_ingress(network)
  File "/usr/share/vdsm/tc.py", line 57, in qdisc_add_ingress
    _process_request(command)
  File "/usr/share/vdsm/tc.py", line 52, in _process_request
    raise TrafficControlException(retcode, msg)
TrafficControlException: (2, 'error executing command "/sbin/tc qdisc add dev rhevm ingress" error: RTNETLINK answers: File exists\n')
Thread-277295::DEBUG::2012-08-12 17:23:45,704::vm::581::vm.Vm::(_startUnderlyingVm) vmId=`1108cc0e-dd17-4944-9a58-9e59a29f1a2a`::_ongoingCreations released
Thread-277295::ERROR::2012-08-12 17:23:45,705::vm::605::vm.Vm::(_startUnderlyingVm) vmId=`1108cc0e-dd17-4944-9a58-9e59a29f1a2a`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 571, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/libvirtvm.py", line 1379, in _run
    self._domDependentInit()
  File "/usr/share/vdsm/libvirtvm.py", line 1265, in _domDependentInit
    supervdsm.getProxy().setPortMirroring(network, nic.name)
  File "/usr/share/vdsm/supervdsm.py", line 59, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 57, in <lambda>
    callMethod = lambda : getattr(self._supervdsmProxy._svdsm, self._funcName)(*args, **kwargs)
  File "<string>", line 2, in setPortMirroring
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
TrafficControlException: (2, 'error executing command "/sbin/tc qdisc add dev rhevm ingress" error: RTNETLINK answers: File exists\n')
Thread-277295::DEBUG::2012-08-12 17:23:45,783::vm::921::vm.Vm::(setDownStatus) vmId=`1108cc0e-dd17-4944-9a58-9e59a29f1a2a`::Changed state to Down: (2, 'error executing command "
/sbin/tc qdisc add dev rhevm ingress" error: RTNETLINK answers: File exists\n')

Comment 3 Dan Kenigsberg 2012-08-13 08:40:30 UTC
We need to add refernce counting to the bridge's promisc mode, and to its ingress rule. Furthermore, only the per-target filter should be removed when a monitorring VM is taken down, not the complete rule.

Comment 9 Dan Kenigsberg 2012-08-23 19:23:42 UTC
There's a bunch of patches dealing with port-mirroring issue:

http://gerrit.ovirt.org/7425
http://gerrit.ovirt.org/7299

and their friends.

Comment 11 GenadiC 2012-09-06 12:18:16 UTC
Verified in SI17

Comment 15 errata-xmlrpc 2012-12-04 19:05:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1508.html