This is a libvirt counterpart to consume QEMU's work. +++ This bug was initially created as a clone of Bug #1790899 +++ Description of problem: We have a customer that is using Windows guests with libvirt (CNV) with virtio drivers loaded on the guest. Non-admin users are seeing the options to 'safely remove ...' the NIC, disks, etc. They'd like this functionality disabled. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Install a Windows 10 guest with virtio drivers 2. Login as a non-admin user 3. Select 'Eject SAS Controller' from taskbar Actual results: Controller is removed Expected results: There should be an option to prevent non-admin users from such tasks. Additional info: 1. Microsoft indicate that this has to be done in the driver, as opposed to registry changes to prevent it 2. In VMWare there is an option, devices.hotplug = “false”, that currently works for the customer
There was some upstream discussion: https://www.redhat.com/archives/libvir-list/2020-February/msg00110.html
I have patches ready to test that expose Julia's new flag in libvirt, and will post them upstream when I verify they work. This will allow a management application to easily disable hotplug for an entire guest (by simply adding "many" pcie-root-ports with <target hotplug='off'/> when the config is created). disabling for a single device will be more problematic, since that requires associating a particular device with a particular pcie-root-port, and currently the only way to do that is by exactly specifying PCI addresses for all devices, which management apps don't want to do (and libvirt also doesn't want them to do :-))
(In reply to Laine Stump from comment #3) > disabling for a single device > will be more problematic, since that requires associating a particular > device with a particular pcie-root-port, and currently the only way to do > that is by exactly specifying PCI addresses for all devices, which > management apps don't want to do (and libvirt also doesn't want them to do > :-)) While it doesn't look like we need to solve this problem for KubeVirt's immediate customer request, I don't think we can continue to punt this problem down the road. The need for applications to be able to have fine grained control over device address assignment (and PCI topology in general) has been clear for a while and demand is growing. Libvirt can never solve all application needs in this area with its current API design approach which aims to avoid policy based decisions. Thus we need a new conceptual approach for this problem, which I have proposed here: https://www.redhat.com/archives/libvir-list/2020-March/msg00078.html
I now have patches that add a hotplug='off' attribute to a pcie-root-port's XML, e.g.: <controller type='pci' model='pcie-root-port'> <target hotplug='off'/> </controller> I've tested this with a qemu build that has Julia's patch described in Bug 1790899 (upstream commit 530a0963184e57e71a5b538e9161f115df533e96), and it has the desired effect - when all the pcie-root-ports have hotplug='off', a Windows guest will no longer show the option to "safely eject" all the PCI devices. I haven't posted the patches yet because right now libvirt itself will still attempt to plug/unplug a device into controllers that have hotplug='off', and just rely on QEMU to fail, but libvirt itself should be prohibiting this. As soon as I've added a patch for that to the series, I will post it upstream. Note that this new attribute is a necessary building block for the new API/library Daniel mentions in Comment 7, and it can also be used on its own in the short term.
patches posted upstream: https://www.redhat.com/archives/libvir-list/2020-April/msg00254.html
pushed upstream, will be in libvirt-6.3.0: commit cbd4ab4cc6945e610b0a3061fcf3517755d65dcf Author: Laine Stump <laine> Date: Thu Feb 27 15:22:59 2020 -0500 qemu: new capabilities flag pcie-root-port.hotplug commit 78f4d5e6f188a9f0f8d6da6b1fe78b9f4172d9ad Author: Laine Stump <laine> Date: Tue Mar 3 12:23:52 2020 -0500 conf: new attribute "hotplug" for pci controllers commit 2d3cf60328c138f7a8fd5905eb345d5f48227ff8 Author: Laine Stump <laine> Date: Tue Mar 3 22:22:14 2020 -0500 qemu: hook up pcie-root-port hotplug='off' option commit 389811d517c270c4f916fce4b3d64f90331244d7 Author: Laine Stump <laine> Date: Thu Mar 5 15:17:41 2020 -0500 docs: mention hotplug='off' in news.xml commit c296a846ad7de3066a17404f4f407d3e1b6d7935 Author: Laine Stump <laine> Date: Sun Mar 22 22:32:49 2020 -0400 conf: add new PCI_CONNECT flag AUTOASSIGN commit 7c98f5e3652e19e4eb015d290c1eed2f1b58ee72 Author: Laine Stump <laine> Date: Sun Apr 5 17:16:55 2020 -0400 conf/qemu: s/VIR_PCI_CONNECT_HOTPLUGGABLE/VIR_PCI_CONNECT_AUTOASSIGN/g commit fcdf87d3ef14de9dfb0acaf4b4445e1580dfc629 Author: Laine Stump <laine> Date: Sun Apr 5 18:01:43 2020 -0400 conf: simplify logic when checking for AUTOASSIGN PCI addresses commit aa15e9259f1f246e69fb9742581ced720c88695d Author: Laine Stump <laine> Date: Sun Apr 5 22:40:37 2020 -0400 qemu/conf: set HOTPLUGGABLE connect flag during PCI address set init commit a283189f8c64882681ea99259ccfc8d1b8e524dd Author: Laine Stump <laine> Date: Sun Apr 5 22:57:43 2020 -0400 conf: check HOTPLUGGABLE connect flag when validating a PCI address commit 7118bdee1550b6022e7362402ca8204add4cf80b Author: Laine Stump <laine> Date: Sun Apr 5 23:44:16 2020 -0400 conf: during PCI hotplug, require that the controller support hotplug
Hi laine, Please help to check the scenario below, Thank you! hot unplug interface connected to pcie-root-port with hotplug=off will report succeed, but it should report error: 1. # virsh dumpxml rhel <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10' hotplug='off'/> ... <interface type='network'> <mac address='52:54:00:16:ee:be'/> <source network='default' portid='a98c367e-0bed-4d47-93fc-de3eeb4a829f' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> ... 2. # ps aux | grep qemu ... -device pcie-root-port,port=0x10,chassis=1,id=pci.1,hotplug=off,bus=pcie.0,multifunction=on,addr=0x2 ... 3. Detach-interface report succeed: # virsh detach-interface rhel network 52:54:00:16:ee:be Interface detached successfully 4. Check by domiflist, and check on guest os, the interface still exists, which is expected. # virsh domiflist rhel Interface Type Source Model MAC ------------------------------------------------------------- vnet0 network default virtio 52:54:00:16:ee:be
Hi laine, another question need your confirmation: After testing, I found the <hotplug> attribute do not inherit from the upstream controllers, right? For example, one VM with the topology as like this: pcie-root-port -> pcie-switch-upstream-port -> pcie-switch-downstream-port.If the pcie-root-port is set with hotplug=off, while the pcie-switch-downstream-port is set with hotplug=on, the devices connect with the pcie-switch-downstream-port will still be able to hotplug/unplug.
1) You are correct about the unplug scenario - libvirt should check the controller of the device that is being unplugged and fail the device detach if hotplug is disabled, but I forgot to do that in the original series. (qemu also originally missed that case, but recently I saw a patch for qemu; the guest will still refuse the unplug request though, which is why the device still shows up in the guest device list). 2) You are correct that the hotplug attribute is not inherited from an upstream controller. This is correct behavior.
Pushed this upstream to fix the bug noted by yalzhang in Comment 16 commit c0e04c2e62957fe872b5bc3d89d5b1d95f10450c Author: Laine Stump <laine> Date: Thu May 14 14:01:47 2020 -0400 qemu: prevent attempts to detach a device on a controller with hotplug='off'
Test on libvirt-6.0.0-22.module+el8.2.1, the result is as expected 1. vm xml validation -> PASS 2. hot-plug/unplug interface to pcie-root-port with hotplug=on/off -> PASS 3. hot-plug/unplug interface to pcie-swtich-downstream-port with hotplug=on/off -> PASS except bug 1832850 about downstream port controller 4. restart libvirtd, save/restore with the hotplug=off setting on controolers -> PASS 5. hotplug / unplug 2 times -> PASS except bug 1832850 about downstream port controller 6. managedsave -> PASS Details for scenario 2: 1. Prepare vm xml with below setting and no interface exists: <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x15' hotplug='on'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x16' hotplug='off'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-switch-upstream-port'> <model name='x3130-upstream'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='9' model='pcie-switch-downstream-port'> <model name='xio3130-downstream'/> <target chassis='9' port='0x0' hotplug='on'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='10' model='pcie-switch-downstream-port'> <model name='xio3130-downstream'/> <target chassis='10' port='0x1' hotplug='off'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x01' function='0x0'/> </controller> 2. prepare interface xml for hotplug: (pcie-root-port with hotplug=on) # cat interface6.xml <interface type='network'> <source network='default'/> <mac address='52:54:00:4e:58:01'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </interface> (pcie-root-port with hotplug=off) # cat interface7.xml <interface type='network'> <mac address='52:54:00:1d:85:58'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </interface> (pcie-switch-downstream-port with hotplug=on) # cat interface9.xml <interface type='network'> <source network='default'/> <mac address='52:54:00:12:b3:84'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </interface> (pcie-switch-downstream-port with hotplug=off) # cat interface10.xml <interface type='network'> <source network='default'/> <mac address='52:54:00:90:d1:20'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </interface> 3. hotplug: # virsh attach-device rhel interface6.xml Device attached successfully # virsh attach-device rhel interface7.xml error: Failed to attach device from interface7.xml error: XML error: The device at PCI address 0000:07:00.0 requires hotplug capability, but the PCI controller with index='7' doesn't support hotplug # virsh attach-device rhel interface9.xml Device attached successfully # virsh attach-device rhel interface10.xml error: Failed to attach device from interface10.xml error: XML error: The device at PCI address 0000:0a:00.0 requires hotplug capability, but the PCI controller with index='10' doesn't support hotplug check on the guest and check the xml, the interface6 and interface9 are attached successfully, and interface7 and interface10 do not exists in the xml and guest os. 4. hot-unplug: 1)destroy the vm and cold-plug all the interface below: # virsh attach-device rhel interface6.xml --config Device attached successfully # virsh attach-device rhel interface7.xml --config Device attached successfully # virsh attach-device rhel interface9.xml --config Device attached successfully # virsh attach-device rhel interface10.xml --config Device attached successfully # virsh domiflist rhel Interface Type Source Model MAC ------------------------------------------------------------- vnet0 network default virtio 52:54:00:4e:58:01 vnet1 network default virtio 52:54:00:1d:85:58 vnet2 network default virtio 52:54:00:12:b3:84 vnet3 network default virtio 52:54:00:90:d1:20 2) try to detach the interface one by one: [root@hp-dl385g10-16 tmp]# virsh detach-device rhel interface6.xml Device detached successfully # virsh detach-device rhel interface7.xml error: Failed to detach device from interface7.xml error: operation failed: cannot hot unplug interface device with PCI guest address: 0000:07:00.0 - not allowed by controller # virsh detach-device rhel interface9.xml Device detached successfully # virsh detach-device rhel interface10.xml error: Failed to detach device from interface10.xml error: operation failed: cannot hot unplug interface device with PCI guest address: 0000:0a:00.0 - not allowed by controller # virsh domiflist rhel Interface Type Source Model MAC ------------------------------------------------------------- vnet1 network default virtio 52:54:00:1d:85:58 vnet3 network default virtio 52:54:00:90:d1:20
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3172