Bug 1011887 - vmDestroy is repeatedly called after starting VM with multicast MAC address and port mirroring on its NIC
Summary: vmDestroy is repeatedly called after starting VM with multicast MAC address a...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.3.0
Hardware: x86_64
OS: Linux
low
medium
Target Milestone: ovirt-3.6.0-rc3
: 3.6.0
Assignee: Ondřej Svoboda
QA Contact: Meni Yakove
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-25 10:39 UTC by tgeft
Modified: 2017-07-10 12:03 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-04-20 01:39:08 UTC
oVirt Team: Network
Target Upstream Version:
Embargoed:
ylavi: Triaged+


Attachments (Terms of Use)
logs (updated with correct files) (68.59 KB, application/zip)
2013-09-29 09:33 UTC, tgeft
no flags Details
VDSM (4.14.1-108.giteb16cad.el6) and oVirt engine (3.4.0-0.11.beta3.fc20) logs (6.73 KB, application/x-zip-compressed)
2014-02-25 17:39 UTC, Ondřej Svoboda
no flags Details

Description tgeft 2013-09-25 10:39:24 UTC
Description of problem:
When starting a VM that has a plugged NIC containing port mirroring and a multicast MAC address (see below) the VM fails to start but vmDestory constantly fails and is endlessly called again.

Version-Release number of selected component (if applicable):
vdsm-4.12.0-156.git6e499d6.el6ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare a shut down VM with a plugged NIC.
2. Select a profile with port mirroring for the NIC.
2. Edit the NIC's MAC address to a multicast address.
3. Start the VM.

Actual results:
An error is given about the VM failing to start because of the multicast address on its NIC. However, this message is constantly repeated as vmDestroy keeps failing and so the log gets flooded with "Message: VM VM1 is down. Exit message: XML error: expected unicast mac address, found multicast '01:1a:4a:16:88:54'".

After deleting the VM, a new one is automatically created instead of it with engine log showing "VM external-VM1 was created by <UNKNOWN>", where the name of the original VM is "VM1". The same error message keeps appearing for this VM as vmDestroy continues to fail and the behavior is repeated when you try to delete the new VM.

Restarting VDSM looks like the only way to stop the behavior.

Expected results:
The VM should fail to start with only one error message and vmDestory shouldn't keep getting called. (Also, deleting the VM should work normally).

Additional info:
Multi cast MAC address is defined by having the most significant bit of the most significant byte set to 1. In other words it's an address of the form
XA:XX:XX:XX:XX:XX where A's parity bit is one, such as 01:1a:4a:16:88:56.

Comment 1 tgeft 2013-09-25 10:40:25 UTC
Created attachment 802721 [details]
incorrect file

Comment 2 Dan Kenigsberg 2013-09-28 22:41:07 UTC
Please provide vdsm version and vdsm.log + supervdsm.log as for every other bug opened on the vdsm component.

The fact that we can ask a VM to start with a multicast address is (another) Engine bug imo.

I believe that the very same vmDestroy bug exists in rhev-3.2, hence this bug should not block the rhev-3.3 release.

Comment 3 tgeft 2013-09-29 09:33:58 UTC
Created attachment 804651 [details]
logs (updated with correct files)

Comment 5 Ondřej Svoboda 2014-02-25 17:39:35 UTC
Created attachment 867568 [details]
VDSM (4.14.1-108.giteb16cad.el6) and oVirt engine (3.4.0-0.11.beta3.fc20) logs

Comment 6 Ondřej Svoboda 2014-02-25 17:52:56 UTC
The exception from the original vdsm.log no longer occurs in the current (master) version of VDSM so the 'destroy' call finishes.

Thread-241::ERROR::2013-09-24 19:04:43,237::BindingXMLRPC::993::vds::(wrapper) unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 979, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/BindingXMLRPC.py", line 211, in vmDestroy
    return vm.destroy()
  File "/usr/share/vdsm/API.py", line 323, in destroy
    res = v.destroy()
  File "/usr/share/vdsm/vm.py", line 4326, in destroy
    response = self.releaseVm()
  File "/usr/share/vdsm/vm.py", line 4252, in releaseVm
    supervdsm.getProxy().unsetPortMirroring(network, nic.name)
AttributeError: 'NetworkInterfaceDevice' object has no attribute 'name'

According to the current logs, port mirroring (probably causing the bug originally) is ignored:

Thread-1886::DEBUG::2014-02-25 17:19:33,039::vm::1259::vm.Vm::(__init__) vmId=`47d76bfb-a180-455c-8ffa-71856d9c6ebc`::Ignoring param (portMirroring, ['net']) in NetworkInterfaceDevice

The issue is gone in my case, but I will need to investigate further.

Comment 7 Dan Kenigsberg 2014-02-25 21:10:36 UTC
You are right Ondrej. commit 8225ae9aeebe8 broke portMirroring in the master branch, which should be fixed asap by adding it as __slot__ to NetworkInterfaceDevice. Then, please re-check this issue.

Comment 8 Dan Kenigsberg 2015-10-25 11:14:13 UTC
Hopefully fixed with bug 1254713.

Comment 9 Michael Burman 2015-10-26 12:14:22 UTC
Verified on - 3.6.0.2-0.1.el6

Blocked by engine from entering multicast MAC addresses :
Blocked in step 3 from steps to reproduce ^^
01:1a:4a:16:88:54 --> 'This field must contain a valid unicast MAC addresses.'


Note You need to log in before you can comment on or make changes to this bug.