Bug 1533778
| Summary: | Hotplug VM vNIC with port mirroring failed | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] vdsm | Reporter: | Meni Yakove <myakove> | ||||
| Component: | Core | Assignee: | Francesco Romani <fromani> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Mor <mkalfon> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 4.20.11 | CC: | bugs, danken, fromani, lveyde, mburman, michal.skrivanek, myakove, ylavi | ||||
| Target Milestone: | ovirt-4.2.2 | Keywords: | Automation, Regression | ||||
| Target Release: | --- | Flags: | rule-engine:
ovirt-4.2+
ylavi: blocker- ylavi: exception+ |
||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | vdsm v4.20.15 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-03-29 11:19:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Network | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1542013 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Meni Yakove
2018-01-12 08:17:45 UTC
I assume it works on a RHEL 7.4 based host? So it's somewhat of a 7.5 issue? Meni, can you try port mirroring of multiple VMs without hotplug? If hotplug is failing due to bug 1533762, you cannot expect port-mirroring to work. I am asking for multiple VMs because the failing command tries to set it on two: /sbin/tc filter replace dev pm_net_1 protocol all parent 800c: u32 match u8 0 0 action mirred egress mirror dev vnet5 (In reply to Yaniv Kaul from comment #1) > I assume it works on a RHEL 7.4 based host? So it's somewhat of a 7.5 issue? It has worked in 7.4 but I guess that since then VDSM was changed as well. (In reply to Dan Kenigsberg from comment #2) > Meni, can you try port mirroring of multiple VMs without hotplug? port mirroring on a fresh VM works just fine. > > If hotplug is failing due to bug 1533762, you cannot expect port-mirroring > to work. bug 1533762 is limited to Empty vNICs; here the vNIC is connected. > I am asking for multiple VMs because the failing command tries to > set it on two: > /sbin/tc filter replace dev pm_net_1 protocol all parent 800c: u32 match > u8 0 0 action mirred egress mirror dev vnet5 I'll try and report back. /sbin/tc filter replace dev pm_net_1 protocol all parent ffff: handle 800::800 pref 49152 u32 match u8 0 0 action mirred egress mirror dev vnet5 action mirred egress mirror dev vnet7 Cannot find device "vnet7" bad action parsing parse_action: bad value (5:mirred)! Illegal "action" Seems like we try to update vnet7 but this vnet doesn't exist in the host. brctl show bridge name bridge id STP enabled interfaces ;vdsmdummy; 8000.000000000000 no ovirtmgmt 8000.002590c6d982 no enp1s0f0 vnet0 vnet1 vnet2 vnet3 vnet4 pm_net_1 8000.002590c6d983 no enp1s0f1.162 vnet11 vnet13 vnet5 vnet9 pm_net_2 8000.00e0ed33c092 no bond01.163 vnet10 vnet12 vnet14 vnet6 vnet8 I guess that vnet7 is the vNIC that we hot-unpluged. Before hotplug: brctl show bridge name bridge id STP enabled interfaces ;vdsmdummy; 8000.000000000000 no ovirtmgmt 8000.002590c6d982 no enp1s0f0 vnet0 vnet1 vnet2 vnet3 vnet4 pm_net_1 8000.002590c6d983 no enp1s0f1.162 vnet11 vnet13 vnet5 vnet7 vnet9 pm_net_2 8000.00e0ed33c092 no bond01.163 vnet10 vnet12 vnet14 vnet6 vnet8 After hot-unplug: brctl show bridge name bridge id STP enabled interfaces ;vdsmdummy; 8000.000000000000 no ovirtmgmt 8000.002590c6d982 no enp1s0f0 vnet0 vnet1 vnet2 vnet3 vnet4 pm_net_1 8000.002590c6d983 no enp1s0f1.162 vnet11 vnet13 vnet5 vnet9 pm_net_2 8000.00e0ed33c092 no bond01.163 vnet10 vnet12 vnet14 vnet6 vnet8 This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP. http://gerrit.ovirt.org/84866 merged -> MODIFIED The fix is not in the current build vdsm-4.20.14-1.el7ev.x86_64
/usr/lib/python2.7/site-packages/vdsm/virt/vm.py:
with self.setLinkAndNetwork(netDev, netConf, linkValue, network,
custom, specParams):
with self.updatePortMirroring(netConf, netsToMirror):
return {'status': doneCode, 'vmList': self.status()}
with vdsm-4.20.17-1.el7ev.x86_64 we still fail with the same error. sure, because tag 4.20.17 doesn't contain the code. It shouldn't have been moved to 4.2.1 and the bot shouldn't move it to ON_QA (In reply to Michal Skrivanek from comment #10) > sure, because tag 4.20.17 doesn't contain the code. It shouldn't have been > moved to 4.2.1 and the bot shouldn't move it to ON_QA Would you point me to that code? The only patch mentioned in this bug is https://gerrit.ovirt.org/#/c/84866/ and it is in v4.20.15 Oh, my bad, I misread the comments, I've meant 4.20.14, the one from comment #8...now I read it again I do not know what I was thinking....my apologies. I wonder if the warning during unplug is relevant, as it looks like perhaps teh nic wasn't removed from the internal device list and then we try set portMirroring on non-existent device. 2018-01-12 10:05:29,610+0200 WARN (libvirt/events) [virt.vm] (vmId='72b85ef7-dbd0-4a07-9afe-a3b90e2317fb') Removed device not found in conf: net1 (vm:5956) This is the call that fails:
2018-01-12 10:05:31,759+0200 INFO (jsonrpc/7) [api.virt] START hotplugNic(params={'xml': '<?xml version="1.0" encoding="UTF-8"?><hotplug><devices><interface type="bridge"><model type="virtio"></model><link state="up"></link><source bridge="pm_net_1"></source><address bus="0x00" domain="0x0000" function="0x0" slot="0x0a" type="pci"></address><mac address="00:1a:4a:16:20:23"></mac><filterref filter="vdsm-no-mac-spoofing"></filterref><bandwidth></bandwidth></interface></devices><metadata xmlns:ovirt-vm="http://ovirt.org/vm/1.0"><ovirt-vm:vm><ovirt-vm:device mac_address="00:1a:4a:16:20:23"><ovirt-vm:portMirroring><ovirt-vm:network>pm_net_1</ovirt-vm:network></ovirt-vm:portMirroring><ovirt-vm:custom></ovirt-vm:custom></ovirt-vm:device></ovirt-vm:vm></metadata></hotplug>', 'nic': {'nicModel': 'pv', 'macAddr': '00:1a:4a:16:20:23', 'linkActive': 'true', 'network': 'pm_net_1', 'filterParameters': [], 'filter': 'vdsm-no-mac-spoofing', 'specParams': {'inbound': {}, 'outbound': {}}, 'deviceId': 'eba87202-035f-45e6-8201-f0c0f1d4e251', 'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x0a'}, 'device': 'bridge', 'type': 'interface', 'portMirroring': ['pm_net_1']}, 'vmId': '72b85ef7-dbd0-4a07-9afe-a3b90e2317fb'})
I find suspicious that we are trying to mirror on the same network on which the vNIC seems attached to:
network='pm_net_1'
portMirroring=['pm_net_1']
(same in XML)
Dan, is this a supported operation?
patch https://gerrit.ovirt.org/87011 will not fix this issue. I tried to reproduce with Vdsm master snapshot 2fe542a9c Engine 4.2.1.5 libvirt-daemon-3.2.0-14.el7_4.7.x86_64 libvirt-python-3.2.0-3.el7_4.1.x86_64 qemu-kvm-ev-2.9.0-16.el7_4.13.1.x86_64 tried the following: 1. run VM, hotplug one NIC with port mirroring -> OK 2. run VM, hotplug 6 (SIX) NICs withOUT port mirroring, then one with port mirroring -> OK Notes: 1. during tests, I killed QEMU and Vdsm failed to clean up, hence patch 87011 2. hotUNplug failed, known libvirt issue So, does it reproduce on every box or just on some? Does it work if yoi just run one VM, let it go UP, then attach one NIC with port mirroring? Michael, maybe you can reproduce this bug manually, as the automated reproduction is too convoluted to follow? (In reply to Dan Kenigsberg from comment #15) > Michael, maybe you can reproduce this bug manually, as the automated > reproduction is too convoluted to follow? I didn't managed to reproduce it manually, only with automation. Meni? We saw this only in automation. Francesco, I have setup to reproduce. please content me. Looks like it is actually a kernel bug I believe that the ovirt-side bug has been fixed, however, we now depend on kernel bug 1542013 no doc_text needed, should Just Work INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Open patch attached] For more info please contact: infra please retest with kernel-3.10.0-854.el7 ovirt-engine-4.2.2.1-0.1.el7.noarch kernel 3.10.0-855.el7.x86_64 This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |