+++ This bug was initially created as a clone of Bug #517464 +++ libvirt supports KVM PCI device assignment since RHEL 5.4, but we do not support hotplug. Patches to add this support have been committed upstream now: http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=7636ef4630 http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=0c5b7b93a3
@Intel We need to confirm that there is commitment to test for the resolution of this request during the RHEL 5.5 test phase, if it is accepted into the release. Please post a confirmation before Oct 16th, 2009, including the contact information for testing engineers.
Dexuan, Would you response comment #2? Thanks.
Add ddugger to CC.
Well, there's no virDomainSaveStatus() in 0.6.3, so even if you added XML_INTERNAL_STATUS it would never be used IMHO, it's fine to just drop the domain_conf.c changes from the second patch - it just means that if you re-start libvirtd after hot-plugging a device, you won't be able to hot-unplug the device. I don't think libvirtd restart is reliable in 0.6.3, anyway
Created attachment 376092 [details] initial refactoring patch Re-factor the hostdev hotplug code so that we can easily add PCI hostdev hotplug to qemudDomainAttachHostDevice(). Based on http://libvirt.org/git/?p=libvirt.git;a=commitdiff_plain;h=7636ef4630
Created attachment 376094 [details] Add host PCI device hotplug support This patch adds the code needed for hotplug, it's based on http://libvirt.org/git/?p=libvirt.git;a=commitdiff_plain;h=0c5b7b93a3 but changed to add missing parts, and remove device assignment since we always run libvirtd as root on RHEL 5.x
libvirt-0.6.3-23.el5 has been built in dist-5E-qu-candidate with the patches, Daniel
This bug still exist with libvirt-0.6.3-28.el5 on RHEL5u5,libvirt don't support PCI device hotplug. Version-Release number of selected component (if applicable): [root@dhcp-66-70-62 ~]# uname -a Linux dhcp-66-70-62.nay.redhat.com 2.6.18-183.el5 #1 SMP Mon Dec 21 18:37:42 EST 2009 x86_64 x86_64 x86_64 GNU/Linux [root@dhcp-66-70-62 ~]# lsmod|grep kvm kvm_intel 86664 0 kvm 223648 2 ksm,kvm_intel [root@dhcp-66-70-62 libvirt]# rpm -qa|grep libvirt libvirt-0.6.3-28.el5 libvirt-debuginfo-0.6.3-28.el5 libvirt-python-0.6.3-28.el5 [root@dhcp-66-70-62 ~]# rpm -q virt-manager kvm virt-manager-0.6.1-11.el5 kvm-83-140.el5 Step to Reproduce: Using virt-manager tool to test: * Remove all NICs from a guest: Run virt-manager, open an existing shutoff status guest and go to the Hardware tab,then choose "NIC :XX:XX:XX" and click "Remove" button * Start the guest * Attach an interface to the guest Run virt-manager, open an running guest and go to the Hardware tab,then click "Add hardware", choose "Network" and click "Forward",choose "Virtual Network" default config,then click "Forward" and "Finish" virt-manager will prompt a dialog,which content display "Are you sure you want to add this device? This device could not be attached to the running machine ..." If you choose "yes' button and log into guest using "lspci -t -v",you can't see the attached NIC device.and you will see the attached NIC device if rebooting guest.
(In reply to comment #13) > Step to Reproduce: > Using virt-manager tool to test: > * Remove all NICs from a guest: > Run virt-manager, open an existing shutoff status guest and go to the > Hardware tab,then choose "NIC :XX:XX:XX" and click "Remove" button > * Start the guest > * Attach an interface to the guest > Run virt-manager, open an running guest and go to the Hardware tab,then > click "Add hardware", choose "Network" and click "Forward",choose > "Virtual Network" default config,then click "Forward" and "Finish" Wait, this doesn't make any sense. This bug is about PCI device assignment, yet in this last step you are trying to add a *virtual* network device. What you actually want to do is go to "Add hardware", choose "Physical host device", hit "Forward", choose the device from the list, hit "Forward", and then hit "Finish". That will test out this bug. Note that in order for hotplug to actually work inside the guest, you need to do "modprobe acpiphp" before you do any of this. If what I outline above doesn't work with libvirt-0.6.3-28.el5, please try out the test packages mentioned in Comment #8 in BZ 500217. I tested exactly this scenario with those test packages, and it seems to work for me. Chris Lalancette
Hi,Chris,I changed the above last testing step according to your comment content,and tried libvirt-0.6.3-29 and you provide packages, but the result is failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe acpiphp" successfully,it seems the kernel bug: [root@dhcp-66-70-62 ~]# modprobe acpiphp FATAL: Error inserting acpiphp (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such device [root@dhcp-66-70-62 ~]# ls /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/ acpiphp_ibm.ko acpiphp.ko fakephp.ko pciehp.ko shpchp.ko
Created attachment 382428 [details] dialog information
(In reply to comment #15) > Hi,Chris,I changed the above last testing step according to your comment > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe > acpiphp" successfully,it seems the kernel bug: > [root@dhcp-66-70-62 ~]# modprobe acpiphp > FATAL: Error inserting acpiphp > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such > device Maybe guest acpi is off by default here?? IIRC, in RHEL5.4 Xen, hvm guest's acpi is off by default. You could double check this. > [root@dhcp-66-70-62 ~]# ls > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/ > acpiphp_ibm.ko acpiphp.ko fakephp.ko pciehp.ko shpchp.ko
(In reply to comment #15) > Hi,Chris,I changed the above last testing step according to your comment > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe > acpiphp" successfully,it seems the kernel bug: > > [root@dhcp-66-70-62 ~]# modprobe acpiphp > FATAL: Error inserting acpiphp > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such > device > > [root@dhcp-66-70-62 ~]# ls > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/ > acpiphp_ibm.ko acpiphp.ko fakephp.ko pciehp.ko shpchp.ko Hm, this is odd. Did you do this *inside* the guest? You need to do it in the guest, not on the host. If you did do it in the guest, you should check that ACPI is enabled in the guest like Dexuan mentions. The fact that it won't attach is another problem. What happens if you use virsh attach-device to attach the device instead of virt-manager? Does that work? If not, does that give you any additional error information? Chris Lalancette
(In reply to comment #17) > (In reply to comment #15) > > Hi,Chris,I changed the above last testing step according to your comment > > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is > > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe > > acpiphp" successfully,it seems the kernel bug: > > [root@dhcp-66-70-62 ~]# modprobe acpiphp > > FATAL: Error inserting acpiphp > > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such > > device > Maybe guest acpi is off by default here?? > IIRC, in RHEL5.4 Xen, hvm guest's acpi is off by default. > You could double check this. Dexuan,thanks for your comment,and I will check it. > > > [root@dhcp-66-70-62 ~]# ls > > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/ > > acpiphp_ibm.ko acpiphp.ko fakephp.ko pciehp.ko shpchp.ko
(In reply to comment #18) > (In reply to comment #15) > > Hi,Chris,I changed the above last testing step according to your comment > > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is > > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe > > acpiphp" successfully,it seems the kernel bug: > > > > [root@dhcp-66-70-62 ~]# modprobe acpiphp > > FATAL: Error inserting acpiphp > > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such > > device > > > > [root@dhcp-66-70-62 ~]# ls > > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/ > > acpiphp_ibm.ko acpiphp.ko fakephp.ko pciehp.ko shpchp.ko > > Hm, this is odd. Did you do this *inside* the guest? You need to do it in the > guest, not on the host. If you did do it in the guest, you should check that > ACPI is enabled in the guest like Dexuan mentions. Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded successfully. > > The fact that it won't attach is another problem. What happens if you use > virsh attach-device to attach the device instead of virt-manager? Does that > work? If not, does that give you any additional error information? > > Chris Lalancette Actually,the above Comment 14 confused me,I think that host device is assigned directly to a guest,this process is pci passthrough not hotplug,they are completely different concept.I want to know the feature whether is pci passthrough,so that I retest it using correct steps.
Created attachment 382926 [details] steps to reproduce
(In reply to comment #20) > (In reply to comment #18) > > (In reply to comment #15) > > > Hi,Chris,I changed the above last testing step according to your comment > > > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is > > > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe > > > acpiphp" successfully,it seems the kernel bug: > > > > > > [root@dhcp-66-70-62 ~]# modprobe acpiphp > > > FATAL: Error inserting acpiphp > > > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such > > > device > > > > > > [root@dhcp-66-70-62 ~]# ls > > > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/ > > > acpiphp_ibm.ko acpiphp.ko fakephp.ko pciehp.ko shpchp.ko > > > > Hm, this is odd. Did you do this *inside* the guest? You need to do it in the > > guest, not on the host. If you did do it in the guest, you should check that > > ACPI is enabled in the guest like Dexuan mentions. > Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded > successfully. OK, good. So you have to do this step before you do the attach-device step. > > > > The fact that it won't attach is another problem. What happens if you use > > virsh attach-device to attach the device instead of virt-manager? Does that > > work? If not, does that give you any additional error information? > > > > Chris Lalancette > Actually,the above Comment 14 confused me,I think that host device is assigned > directly to a guest,this process is pci passthrough not hotplug,they are > completely different concept.I want to know the feature whether is pci > passthrough,so that I retest it using correct steps. In Comment #13, you tried to attach a Virtual Network device, *not* a PCI passthrough device. That's the part that didn't make sense. Additionally, PCI passthrough and hotplug are not completely different, they are related. What we are trying to test here is whether PCI passthrough works, both at domain boot time and also when doing a hotplug/hot-unplug. The steps you outline in Comment #21 look reasonable, except you didn't show me what's in hostdev.xml, so I can't tell for sure. Please show me what's in hostdev.xml so I can confirm. Finally, the error messages you see in the steps in Comment #21 look related to the problems in BZ 500217. Let's try to figure that one out first, then we'll get back to this. Chris Lalancette
(In reply to comment #22) > (In reply to comment #20) > > (In reply to comment #18) > > > (In reply to comment #15) > > > > Hi,Chris,I changed the above last testing step according to your comment > > > > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is > > > > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe > > > > acpiphp" successfully,it seems the kernel bug: > > > > > > > > [root@dhcp-66-70-62 ~]# modprobe acpiphp > > > > FATAL: Error inserting acpiphp > > > > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such > > > > device > > > > > > > > [root@dhcp-66-70-62 ~]# ls > > > > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/ > > > > acpiphp_ibm.ko acpiphp.ko fakephp.ko pciehp.ko shpchp.ko > > > > > > Hm, this is odd. Did you do this *inside* the guest? You need to do it in the > > > guest, not on the host. If you did do it in the guest, you should check that > > > ACPI is enabled in the guest like Dexuan mentions. > > Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded > > successfully. > > OK, good. So you have to do this step before you do the attach-device step. > > > > > > > The fact that it won't attach is another problem. What happens if you use > > > virsh attach-device to attach the device instead of virt-manager? Does that > > > work? If not, does that give you any additional error information? > > > > > > Chris Lalancette > > Actually,the above Comment 14 confused me,I think that host device is assigned > > directly to a guest,this process is pci passthrough not hotplug,they are > > completely different concept.I want to know the feature whether is pci > > passthrough,so that I retest it using correct steps. > > In Comment #13, you tried to attach a Virtual Network device, *not* a PCI > passthrough device. That's the part that didn't make sense. > > Additionally, PCI passthrough and hotplug are not completely different, they > are related. What we are trying to test here is whether PCI passthrough works, > both at domain boot time and also when doing a hotplug/hot-unplug. The steps > you outline in Comment #21 look reasonable, except you didn't show me what's in > hostdev.xml, so I can't tell for sure. Please show me what's in hostdev.xml so > I can confirm. Hi,Chris,the hostdev.xml looks like as following: [root@dhcp-66-70-62 ~]# cat hostdev.xml <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> </source> </hostdev> > > Finally, the error messages you see in the steps in Comment #21 look related to > the problems in BZ 500217. Let's try to figure that one out first, then we'll > get back to this. > > Chris Lalancette
Hi,Chris,the PCI passthrough will work if we degrade libvirt version to 0.6.3-23.el5,but it failed on libvirt-0.6.3-29.el5/libvirt-0.6.3-30.el5, so this may be regression bug(see attachment).
Created attachment 384885 [details] libvirt-0.6.3-23.el5 host information
Created attachment 384886 [details] libvirt-0.6.3-23.el5 guest information
Created attachment 384888 [details] libvirt-0.6.3-30.el5 host information
This bug has been fixed with libvirt-0.6.3-31.el5 on rhel5u5,set the bug status to VERIFIED.(see the attachment)
Created attachment 387479 [details] libvirt-0.6.3-31.el5 host information
Created attachment 387480 [details] libvirt-0.6.3-31.el5 guest information
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0205.html