Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 918412

Summary: cannot plug VNIC with port mirroring enabled on running VM
Product: Red Hat Enterprise Virtualization Manager Reporter: Martin Pavlik <mpavlik>
Component: vdsmAssignee: Antoni Segura Puimedon <asegurap>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Martin Pavlik <mpavlik>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.2.0CC: acathrow, bazulay, danken, gklein, hateya, iheim, jkt, lpeer, mpavlik, myakove, Rhev-m-bugs, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: network
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-02 11:32:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreport-LogCollector-admin-20130306091226-6399.tar
none
screenshot 1
none
log_collector_2
none
log_collector_3 none

Description Martin Pavlik 2013-03-06 08:18:56 UTC
Created attachment 705802 [details]
sosreport-LogCollector-admin-20130306091226-6399.tar

Description of problem:
If VM is launched with unplugged VNIC which has port mirroring enabled, the VNIC cannot be plugged on running VM. 

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Manager Version: 3.2.0-10.10.beta1.el6ev 

How reproducible:
100%

Steps to Reproduce:
1) Create VM - with 1 VNIC connected to rhevm, link up, port mirroring enabled, unplugged
2) Start VM, wait until VM is UP
3) go to VNIC edit dialog -> plug the VNIC
  
Actual results:
Error while executing action Edit VM Interface properties: Failed to activate VM Network Interface.

Expected results:
Interface activated

Additional info:

2013-03-06 09:09:25,321 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-2) [2824eabf] Failed in HotPlugNicVDS method
2013-03-06 09:09:25,321 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-2) [2824eabf] Error code ACTIVATE_NIC_FAILED and error message VDSGenericException: VDSErrorException: Failed to HotPlugNicVDS, error = internal error ifname "vnet0" not in key map
2013-03-06 09:09:25,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-2) [2824eabf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand return value 
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=49, mMessage=internal error ifname "vnet0" not in key map]]
2013-03-06 09:09:25,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-2) [2824eabf] HostName = dell-05
2013-03-06 09:09:25,322 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8702-2) [2824eabf] Command HotPlugNicVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugNicVDS, error = internal error ifname "vnet0" not in key map
2013-03-06 09:09:25,322 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (ajp-/127.0.0.1:8702-2) [2824eabf] FINISH, HotPlugNicVDSCommand, log id: 597de01e
2013-03-06 09:09:25,322 ERROR [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-2) [2824eabf] Command org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugNicVDS, error = internal error ifname "vnet0" not in key map

Comment 1 Martin Pavlik 2013-03-06 08:19:24 UTC
Created attachment 705803 [details]
screenshot 1

Comment 2 Martin Pavlik 2013-03-07 14:49:43 UTC
when running another test I noticed that it is also not possible to add a new VNIC with port mirroring enabled

it seems like a typo somewhere in code

look at this log line: parse_action: bad value (17:mirred)!

seems like there is "mirred" instead of mirrored

1) create new VM with no VNICs with OS suppurting hotplug
2) run the VM, wait until it is UP
3) create new VNIC, connected to rhevm, link up, plugged, port mirroring enabled

full log collector attached as log_collector_2

2013-03-07 15:44:29,672 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-13) [6fb8c6cc] Failed in HotPlugNicVDS method
2013-03-07 15:44:29,672 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-13) [6fb8c6cc] Error code ACTIVATE_NIC_FAILED and error message VDSGenericException: VDSErrorException: Failed to HotPlugNicVDS, error = Cannot find device "if109"
bad action parsing
parse_action: bad value (17:mirred)!
Illegal "action"

2013-03-07 15:44:29,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-13) [6fb8c6cc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand return value 
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=49, mMessage=Cannot find device "if109"
bad action parsing
parse_action: bad value (17:mirred)!
Illegal "action"
]]
2013-03-07 15:44:29,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-13) [6fb8c6cc] HostName = dell-05
2013-03-07 15:44:29,672 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8702-13) [6fb8c6cc] Command HotPlugNicVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugNicVDS, error = Cannot find device "if109"
bad action parsing
parse_action: bad value (17:mirred)!
Illegal "action"

2013-03-07 15:44:29,672 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (ajp-/127.0.0.1:8702-13) [6fb8c6cc] FINISH, HotPlugNicVDSCommand, log id: 5e568eb3
2013-03-07 15:44:29,673 ERROR [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-13) [6fb8c6cc] Command org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugNicVDS, error = Cannot find device "if109"
bad action parsing
parse_action: bad value (17:mirred)!
Illegal "action"

Comment 3 Martin Pavlik 2013-03-07 14:56:56 UTC
Created attachment 706655 [details]
log_collector_2

Comment 4 Martin Pavlik 2013-03-08 12:02:47 UTC
Seems like this is coming from vdsm see vdms.log attached as log_collector_2 , this problem also fails migration of VMs with port mirror enabled

MainProcess|Thread-117381::ERROR::2013-03-07 15:44:29,375::supervdsmServer::81::SuperVdsm.ServerCallback::(wrapper) Error in setPortMirroring
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 79, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 285, in setPortMirroring
    tc.setPortMirroring(networkName, ifaceName)
  File "/usr/share/vdsm/tc.py", line 72, in setPortMirroring
    _addTarget(network, QDISC_INGRESS, target)
  File "/usr/share/vdsm/tc.py", line 48, in _addTarget
    filter_replace(network, parent, filt)
  File "/usr/share/vdsm/tc.py", line 130, in filter_replace
    _process_request(command)
  File "/usr/share/vdsm/tc.py", line 100, in _process_request
    raise TrafficControlException(retcode, err, command)
TrafficControlException: (1, 'Cannot find device "if109"\nbad action parsing\nparse_action: bad value (17:mirred)!\nIllegal "action"\n', ['/sbin/tc', 'filter', 'replace', 'dev', 'rhevm', 'parent', 'ffff:', 'prio', '49152', 'handle', '800::800', 'protocol', 'ip', 'u32', 'match', 'u8', '0', '0', 'action', 'mirred', 'egress', 'mirror', 'dev', 'if109', 'action', 'mirred', 'egress', 'mirror', 'dev', 'vnet1', 'action', 'mirred', 'egress', 'mirror', 'dev', u'vnet2'])

Comment 5 Dan Kenigsberg 2013-03-11 12:27:55 UTC
Martin, 'dev if109' is what 'tc' reports when if cannot find the source bridge. The interesting issue is how your system has got to this condition.

Does this bug reproduce after the host is rebooted?
How? The key it to understand how you get into

 /sbin/tc filter show dev rhevm parent ffff:

reporting ifXXX devices.

Comment 6 Martin Pavlik 2013-04-18 09:27:22 UTC
Dan, I probably found a reason of this behavior, this problem appears if you assign some network with display role, but you do not assign an IP to the host interface attached to this network.

in steps:

1) create logical network (bridgeless in my case) and attach to your cluster
2) assign the network with display role
3) attach the network to host/hosts with IP set to none
4) try to run VM

I would suggest some warning in GUI, that it is not recommended to use none IP in case of display network, or even block "none IP" option if display role is set.

log collector attached as log_collector_3

GUI error: VM RHEL64 is down. Exit message: internal error ifname "vnet0" not in key map.

vdsm.log:
Thread-56058::DEBUG::2013-04-18 09:05:37,395::vm::676::vm.Vm::(_startUnderlyingVm) vmId=`e536ea57-36c4-40c3-bea7-5062299667a1`::_ongoingCreations released
Thread-56058::ERROR::2013-04-18 09:05:37,395::vm::700::vm.Vm::(_startUnderlyingVm) vmId=`e536ea57-36c4-40c3-bea7-5062299667a1`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 662, in _startUnderlyingVm
  File "/usr/share/vdsm/libvirtvm.py", line 1524, in _run
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 104, in wrapper
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2645, in createXML
libvirtError: internal error ifname "vnet0" not in key map
Thread-56058::DEBUG::2013-04-18 09:05:37,402::vm::1064::vm.Vm::(setDownStatus) vmId=`e536ea57-36c4-40c3-bea7-5062299667a1`::Changed state to Down: internal error ifname "vnet0" not in key map

Comment 7 Martin Pavlik 2013-04-18 09:30:02 UTC
Created attachment 737251 [details]
log_collector_3

Comment 8 Dan Kenigsberg 2013-07-21 14:40:29 UTC
Martin, the description of comment 6 is not in line with the bug summary. Do you currently experience any problem with port mirroring or the `tc` tool? If not, what you describe is a dup of bug 955429.

Would you please either close this bug or supply a reproduction for the
  Cannot find device "if109"
issue?