Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1680450

Summary: VDSM does not handle device removal from within the Guest
Product: Red Hat Enterprise Virtualization Manager Reporter: Germano Veit Michel <gveitmic>
Component: vdsmAssignee: Milan Zamazal <mzamazal>
Status: CLOSED ERRATA QA Contact: Roni <reliezer>
Severity: high Docs Contact:
Priority: medium    
Version: 4.2.8CC: dfodor, dholler, gveitmic, lleistne, lsurette, mtessun, mzamazal, rbarry, reliezer, srevivo, vicente.balaguer, ycui
Target Milestone: ovirt-4.3.3Keywords: TestOnly
Target Release: 4.3.0Flags: lsvaty: testing_plan_complete-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: VerificationWeek
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-08 12:36:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
win10 getInfo after uninstalling vNIC
none
win10 dumpxml plug/unplug/uninstall none

Description Germano Veit Michel 2019-02-25 04:05:40 UTC
Description of problem:

If the user removes a device from within the Guest, VDSM keeps reporting the device to the engine even if the device does not exist anymore.

2019-02-25 14:01:13,064+1000 INFO  (libvirt/events) [virt.vm] (vmId='eb658a5a-9121-4473-9627-3d6c7535e9b4') Device removal reported: ua-18a7700e-0787-40f2-a735-3be53369ca36 (vm:6177)
2019-02-25 14:01:13,064+1000 WARN  (libvirt/events) [virt.vm] (vmId='eb658a5a-9121-4473-9627-3d6c7535e9b4') Removed device not found in conf: ua-18a7700e-0787-40f2-a735-3be53369ca36 (vm:6187)
2019-02-25 14:01:13,064+1000 WARN  (libvirt/events) [virt.vm] (vmId='eb658a5a-9121-4473-9627-3d6c7535e9b4') Removed device not found in devices: ua-18a7700e-0787-40f2-a735-3be53369ca36 (vm:6197)

Version-Release number of selected component (if applicable):
vdsm-4.20.43-1.el7ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Remove a vNIC from within the Windows Guest
   * Right click on "Eject" icon on the taskbar next to the clock
   * eject the Virtio Network Adapter

Actual results:
1) At this point even the vnet device on the host is done, and its not present on the libvirt XML too.
2) VDSM continues reporting the device, and so does the engine. Here is the getInfo from VDSM API:

        {
            "nicModel": "pv", 
            "macAddr": "00:1a:4a:61:18:29", 
            "linkActive": true, 
            "network": "rhevm", 
            "filterParameters": [], 
            "alias": "ua-18a7700e-0787-40f2-a735-3be53369ca36", 
            "filter": "vdsm-no-mac-spoofing", 
            "specParams": {}, 
            "address": {
                "function": "0x0", 
                "bus": "0x00", 
                "domain": "0x0000", 
                "type": "pci", 
                "slot": "0x03"
            }, 
            "device": "bridge", 
            "type": "interface"
        },

Expected results:
Engine shows the device as unplugged or something to tell the Admin that the device was removed from within the Guest.

Comment 2 Milan Zamazal 2019-03-05 15:51:26 UTC
Vdsm < 4.3 doesn't detect device removal from within the guest. Although Engine >= 4.2 doesn't rely on device info from Vdsm and uses libvirt XML to process devices, Vdsm < 4.3 doesn't update the provided libvirt XML on device removal from within the guest and thus the change may be detected by Engine only later or not at all.

Device removal was reworked in Vdsm 4.3 and the given scenario works for me properly in 4.3. That means the NIC disappears in Vdsm and Engine shows it as unplugged.

Comment 3 Milan Zamazal 2019-03-05 15:56:26 UTC
Germano, does it work for you in 4.3? If yes, can we close the bug?

Comment 4 Germano Veit Michel 2019-03-06 01:53:22 UTC
(In reply to Milan Zamazal from comment #3)
> Germano, does it work for you in 4.3? If yes, can we close the bug?

Thanks Milan,

I'll need some time to test this on 4.3 as I had to re-install my env with 4.2.8. I might be able to test it later this week.

Still, I think the correct procedure here is to QE and then include in 4.3 GA errata, as this is a downstream bug, not close the BZ now. Right?

Comment 5 Ryan Barry 2019-03-06 02:05:00 UTC
Yes, but only because it has a customer case

Comment 6 Germano Veit Michel 2019-03-06 02:32:41 UTC
Thanks Ryan, so should this be moved to ON_QA?

Do you still need me to test it?

Comment 7 Ryan Barry 2019-03-06 02:42:29 UTC
No need -- QE is necessary for VERIFIED, and I'll take Milan's word for it.

If you have the time, a second positive result always helps rule it out, though.

Comment 9 Germano Veit Michel 2019-03-06 03:48:57 UTC
Works for me on 4.3:
  ovirt-engine-4.3.0.4-1.el7.noarch
  vdsm-4.30.8-1.el7.x86_64

VDSM:
2019-03-06 13:41:26,942+1000 INFO  (libvirt/events) [virt.vm] (vmId='f18ab596-3f36-4d3c-b5fc-d4ebfb9ba716') Device removal reported: ua-09ad9be8-71e3-440d-8135-2f35cec758a5 (vm:6018)

ENGINE:
Takes a while to update (next DumpXmlsVDSCommand). And its not specifically logged in engine.log or audit_log, I think it should be.

Comment 10 Roni 2019-03-07 16:19:31 UTC
Created attachment 1541901 [details]
win10 getInfo after uninstalling vNIC

Comment 11 Roni 2019-03-07 16:26:30 UTC
- vdsm-4.30.10-1.el7ev.x86_64
- Win10
After uninstalling the vNIC from Device Manager, getInfo still return its details
see attached vdsm-client VM getInfo "vmID"="<VM ID>" details

Comment 12 Milan Zamazal 2019-03-07 17:00:03 UTC
Roni, on our setup uninstalling vNIC from Device Manager doesn't remove the interface from the VM, it's still present in its domain XML (virsh -r dumpxml win10). The device must be ejected as described in the bug description in order to get the device removed.

Can you check that the device got actually removed from the domain XML in your reproducer? If not then it's working as expected -- the device is still there.

Comment 13 Roni 2019-03-08 06:44:20 UTC
Milan, yes the device still exists at the dumpxml
After unplug, it disappears as you expected
so actually it's ok, but when uninstalling the NIC
it is not available at the guest OS and not seeing by running ipconfig
but it still appears at the Engine as active
don't you think this should be reflected at the Engine

Comment 14 Roni 2019-03-08 07:11:09 UTC
Created attachment 1542055 [details]
win10 dumpxml plug/unplug/uninstall

Comment 15 Milan Zamazal 2019-03-08 08:07:55 UTC
(In reply to Roni from comment #13)
> Milan, yes the device still exists at the dumpxml
> After unplug, it disappears as you expected
> so actually it's ok, but when uninstalling the NIC
> it is not available at the guest OS and not seeing by running ipconfig
> but it still appears at the Engine as active
> don't you think this should be reflected at the Engine

In theory, it would be nice to have that. But that requires that a guest is technically able to pass information about device removal to the host and that it actually does that. It works when ejecting device on Windows, but apparently doesn't work when uninstalling the device on either Windows or RHEL. And AFAIK we don't support that functionality.

If the functionality was really needed, a separate RFE should be created. Such a change would require changes in one or more of the guest OS, device drivers, guest agent, and QEMU. If it worked in all of that, it should work in RHV without further changes. It's out of scope of this bug, so putting it back to MODIFIED.

Comment 16 RHV bug bot 2019-03-29 11:14:52 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 17 Roni 2019-04-01 15:07:32 UTC
Verified: 4.3.2-0.1.el7

Comment 19 errata-xmlrpc 2019-05-08 12:36:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1077

Comment 20 Daniel Gur 2019-08-28 13:13:18 UTC
sync2jira

Comment 21 Daniel Gur 2019-08-28 13:17:31 UTC
sync2jira