Bug 1011887

Summary: vmDestroy is repeatedly called after starting VM with multicast MAC address and port mirroring on its NIC
Product: Red Hat Enterprise Virtualization Manager Reporter: tgeft
Component: vdsmAssignee: Ondřej Svoboda <osvoboda>
Status: CLOSED CURRENTRELEASE QA Contact: Meni Yakove <myakove>
Severity: medium Docs Contact:
Priority: low    
Version: 3.3.0CC: bazulay, danken, lpeer, mburman, mpoledni, myakove, nobody, osvoboda, srevivo, ylavi
Target Milestone: ovirt-3.6.0-rc3Keywords: CodeChange
Target Release: 3.6.0Flags: ylavi: Triaged+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-20 01:39:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs (updated with correct files)
none
VDSM (4.14.1-108.giteb16cad.el6) and oVirt engine (3.4.0-0.11.beta3.fc20) logs none

Description tgeft 2013-09-25 10:39:24 UTC 
Description of problem:
When starting a VM that has a plugged NIC containing port mirroring and a multicast MAC address (see below) the VM fails to start but vmDestory constantly fails and is endlessly called again.

Version-Release number of selected component (if applicable):
vdsm-4.12.0-156.git6e499d6.el6ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare a shut down VM with a plugged NIC.
2. Select a profile with port mirroring for the NIC.
2. Edit the NIC's MAC address to a multicast address.
3. Start the VM.

Actual results:
An error is given about the VM failing to start because of the multicast address on its NIC. However, this message is constantly repeated as vmDestroy keeps failing and so the log gets flooded with "Message: VM VM1 is down. Exit message: XML error: expected unicast mac address, found multicast '01:1a:4a:16:88:54'".

After deleting the VM, a new one is automatically created instead of it with engine log showing "VM external-VM1 was created by <UNKNOWN>", where the name of the original VM is "VM1". The same error message keeps appearing for this VM as vmDestroy continues to fail and the behavior is repeated when you try to delete the new VM.

Restarting VDSM looks like the only way to stop the behavior.

Expected results:
The VM should fail to start with only one error message and vmDestory shouldn't keep getting called. (Also, deleting the VM should work normally).

Additional info:
Multi cast MAC address is defined by having the most significant bit of the most significant byte set to 1. In other words it's an address of the form
XA:XX:XX:XX:XX:XX where A's parity bit is one, such as 01:1a:4a:16:88:56.
Comment 1 tgeft 2013-09-25 10:40:25 UTC 
Created attachment 802721 [details]
incorrect file
Comment 2 Dan Kenigsberg 2013-09-28 22:41:07 UTC 
Please provide vdsm version and vdsm.log + supervdsm.log as for every other bug opened on the vdsm component.

The fact that we can ask a VM to start with a multicast address is (another) Engine bug imo.

I believe that the very same vmDestroy bug exists in rhev-3.2, hence this bug should not block the rhev-3.3 release.
Comment 3 tgeft 2013-09-29 09:33:58 UTC 
Created attachment 804651 [details]
logs (updated with correct files)
Comment 5 Ondřej Svoboda 2014-02-25 17:39:35 UTC 
Created attachment 867568 [details]
VDSM (4.14.1-108.giteb16cad.el6) and oVirt engine (3.4.0-0.11.beta3.fc20) logs
Comment 6 Ondřej Svoboda 2014-02-25 17:52:56 UTC 
The exception from the original vdsm.log no longer occurs in the current (master) version of VDSM so the 'destroy' call finishes.

Thread-241::ERROR::2013-09-24 19:04:43,237::BindingXMLRPC::993::vds::(wrapper) unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 979, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/BindingXMLRPC.py", line 211, in vmDestroy
    return vm.destroy()
  File "/usr/share/vdsm/API.py", line 323, in destroy
    res = v.destroy()
  File "/usr/share/vdsm/vm.py", line 4326, in destroy
    response = self.releaseVm()
  File "/usr/share/vdsm/vm.py", line 4252, in releaseVm
    supervdsm.getProxy().unsetPortMirroring(network, nic.name)
AttributeError: 'NetworkInterfaceDevice' object has no attribute 'name'

According to the current logs, port mirroring (probably causing the bug originally) is ignored:

Thread-1886::DEBUG::2014-02-25 17:19:33,039::vm::1259::vm.Vm::(__init__) vmId=`47d76bfb-a180-455c-8ffa-71856d9c6ebc`::Ignoring param (portMirroring, ['net']) in NetworkInterfaceDevice

The issue is gone in my case, but I will need to investigate further.
Comment 7 Dan Kenigsberg 2014-02-25 21:10:36 UTC 
You are right Ondrej. commit 8225ae9aeebe8 broke portMirroring in the master branch, which should be fixed asap by adding it as __slot__ to NetworkInterfaceDevice. Then, please re-check this issue.
Comment 8 Dan Kenigsberg 2015-10-25 11:14:13 UTC 
Hopefully fixed with bug 1254713.
Comment 9 Michael Burman 2015-10-26 12:14:22 UTC 
Verified on - 3.6.0.2-0.1.el6

Blocked by engine from entering multicast MAC addresses :
Blocked in step 3 from steps to reproduce ^^
01:1a:4a:16:88:54 --> 'This field must contain a valid unicast MAC addresses.'