Bug 517465 - Add support to libvirt for KVM PCI device assignment hotplug
Summary: Add support to libvirt for KVM PCI device assignment hotplug
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: libvirt
Version: 5.5
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: 5.5
Assignee: Daniel Veillard
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 516837 532386 533941 557291
TreeView+ depends on / blocked
 
Reported: 2009-08-14 08:04 UTC by Mark McLoughlin
Modified: 2010-03-30 08:09 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-03-30 08:09:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
initial refactoring patch (3.73 KB, patch)
2009-12-04 14:28 UTC, Daniel Veillard
no flags Details | Diff
Add host PCI device hotplug support (17.43 KB, patch)
2009-12-04 14:31 UTC, Daniel Veillard
no flags Details | Diff
dialog information (1.11 MB, image/png)
2010-01-08 09:55 UTC, Alex Jia
no flags Details
steps to reproduce (1.82 KB, text/plain)
2010-01-11 07:44 UTC, Alex Jia
no flags Details
libvirt-0.6.3-23.el5 host information (5.28 KB, text/plain)
2010-01-17 08:10 UTC, Alex Jia
no flags Details
libvirt-0.6.3-23.el5 guest information (200.37 KB, image/png)
2010-01-17 08:11 UTC, Alex Jia
no flags Details
libvirt-0.6.3-30.el5 host information (5.17 KB, text/plain)
2010-01-17 08:15 UTC, Alex Jia
no flags Details
libvirt-0.6.3-31.el5 host information (5.55 KB, text/plain)
2010-01-29 03:14 UTC, Alex Jia
no flags Details
libvirt-0.6.3-31.el5 guest information (77.80 KB, image/png)
2010-01-29 03:15 UTC, Alex Jia
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0205 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2010-03-29 12:27:37 UTC

Description Mark McLoughlin 2009-08-14 08:04:19 UTC
+++ This bug was initially created as a clone of Bug #517464 +++

libvirt supports KVM PCI device assignment since RHEL 5.4, but we do not support hotplug.

Patches to add this support have been committed upstream now:

  http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=7636ef4630
  http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=0c5b7b93a3

Comment 2 Chris Ward 2009-10-13 15:48:47 UTC
@Intel

We need to confirm that there is commitment to test 
for the resolution of this request during the RHEL 5.5 test
phase, if it is accepted into the release. 

Please post a confirmation before Oct 16th, 2009, 
including the contact information for testing engineers.

Comment 3 Jane Lv 2009-10-15 03:14:00 UTC
Dexuan,

Would you response comment #2?  Thanks.

Comment 4 Dexuan Cui 2009-11-02 06:38:32 UTC
Add ddugger to CC.

Comment 7 Mark McLoughlin 2009-11-30 15:17:03 UTC
Well, there's no virDomainSaveStatus() in 0.6.3, so even if you added XML_INTERNAL_STATUS it would never be used

IMHO, it's fine to just drop the domain_conf.c changes from the second patch - it just means that if you re-start libvirtd after hot-plugging a device, you won't be able to hot-unplug the device. I don't think libvirtd restart is reliable in 0.6.3, anyway

Comment 9 Daniel Veillard 2009-12-04 14:28:35 UTC
Created attachment 376092 [details]
initial refactoring patch

Re-factor the hostdev hotplug code so that we can easily add PCI
hostdev hotplug to qemudDomainAttachHostDevice(). Based on
http://libvirt.org/git/?p=libvirt.git;a=commitdiff_plain;h=7636ef4630

Comment 10 Daniel Veillard 2009-12-04 14:31:07 UTC
Created attachment 376094 [details]
Add host PCI device hotplug support

This patch adds the code needed for hotplug, it's based on 
http://libvirt.org/git/?p=libvirt.git;a=commitdiff_plain;h=0c5b7b93a3
but changed to add missing parts, and remove device assignment since
we always run libvirtd as root on RHEL 5.x

Comment 11 Daniel Veillard 2009-12-04 15:36:44 UTC
libvirt-0.6.3-23.el5 has been built in dist-5E-qu-candidate with
the patches,

Daniel

Comment 13 Alex Jia 2009-12-30 09:56:09 UTC
This bug still exist with libvirt-0.6.3-28.el5 on RHEL5u5,libvirt don't support
PCI device hotplug.

Version-Release number of selected component (if applicable):
[root@dhcp-66-70-62 ~]# uname -a
Linux dhcp-66-70-62.nay.redhat.com 2.6.18-183.el5 #1 SMP Mon Dec 21 18:37:42
EST 2009 x86_64 x86_64 x86_64 GNU/Linux

[root@dhcp-66-70-62 ~]# lsmod|grep kvm
kvm_intel              86664  0 
kvm                   223648  2 ksm,kvm_intel

[root@dhcp-66-70-62 libvirt]# rpm -qa|grep libvirt
libvirt-0.6.3-28.el5
libvirt-debuginfo-0.6.3-28.el5
libvirt-python-0.6.3-28.el5

[root@dhcp-66-70-62 ~]# rpm -q virt-manager kvm
virt-manager-0.6.1-11.el5
kvm-83-140.el5  


Step to Reproduce:
Using virt-manager tool to test:
  * Remove all NICs from a guest:
    Run virt-manager, open an existing shutoff status guest and go to the   
    Hardware tab,then choose "NIC :XX:XX:XX" and click "Remove" button
  * Start the guest 
  * Attach an interface to the guest
    Run virt-manager, open an running guest and go to the Hardware tab,then 
    click "Add hardware", choose "Network" and click "Forward",choose 
    "Virtual Network" default config,then click "Forward" and "Finish"

virt-manager will prompt a dialog,which content display "Are you sure you want
to add this device? This device could not be attached to the running machine ..."

If you choose "yes' button and log into guest using "lspci -t -v",you can't see the attached NIC device.and you will see the attached NIC device if rebooting guest.

Comment 14 Chris Lalancette 2010-01-07 19:02:59 UTC
(In reply to comment #13)
> Step to Reproduce:
> Using virt-manager tool to test:
>   * Remove all NICs from a guest:
>     Run virt-manager, open an existing shutoff status guest and go to the   
>     Hardware tab,then choose "NIC :XX:XX:XX" and click "Remove" button
>   * Start the guest 
>   * Attach an interface to the guest
>     Run virt-manager, open an running guest and go to the Hardware tab,then 
>     click "Add hardware", choose "Network" and click "Forward",choose 
>     "Virtual Network" default config,then click "Forward" and "Finish"

Wait, this doesn't make any sense.  This bug is about PCI device assignment, yet in this last step you are trying to add a *virtual* network device.  What you actually want to do is go to "Add hardware", choose "Physical host device", hit "Forward", choose the device from the list, hit "Forward", and then hit "Finish".  That will test out this bug.  Note that in order for hotplug to actually work inside the guest, you need to do "modprobe acpiphp" before you do any of this.

If what I outline above doesn't work with libvirt-0.6.3-28.el5, please try out the test packages mentioned in Comment #8 in BZ 500217.  I tested exactly this scenario with those test packages, and it seems to work for me.

Chris Lalancette

Comment 15 Alex Jia 2010-01-08 09:52:20 UTC
Hi,Chris,I changed the above last testing step according to your comment content,and tried libvirt-0.6.3-29 and you provide packages, but the result is failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe acpiphp" successfully,it seems the kernel bug:

[root@dhcp-66-70-62 ~]# modprobe acpiphp
FATAL: Error inserting acpiphp (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such device

[root@dhcp-66-70-62 ~]# ls /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko

Comment 16 Alex Jia 2010-01-08 09:55:09 UTC
Created attachment 382428 [details]
dialog information

Comment 17 Dexuan Cui 2010-01-08 10:15:41 UTC
(In reply to comment #15)
> Hi,Chris,I changed the above last testing step according to your comment
> content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> acpiphp" successfully,it seems the kernel bug:
> [root@dhcp-66-70-62 ~]# modprobe acpiphp
> FATAL: Error inserting acpiphp
> (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> device
Maybe guest acpi is off by default here?? 
IIRC, in RHEL5.4 Xen, hvm guest's acpi is off by default.
You could double check this.

> [root@dhcp-66-70-62 ~]# ls
> /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko

Comment 18 Chris Lalancette 2010-01-08 21:59:37 UTC
(In reply to comment #15)
> Hi,Chris,I changed the above last testing step according to your comment
> content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> acpiphp" successfully,it seems the kernel bug:
> 
> [root@dhcp-66-70-62 ~]# modprobe acpiphp
> FATAL: Error inserting acpiphp
> (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> device
> 
> [root@dhcp-66-70-62 ~]# ls
> /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko  

Hm, this is odd.  Did you do this *inside* the guest?  You need to do it in the guest, not on the host.  If you did do it in the guest, you should check that ACPI is enabled in the guest like Dexuan mentions.

The fact that it won't attach is another problem.  What happens if you use virsh attach-device to attach the device instead of virt-manager?  Does that work?  If not, does that give you any additional error information?

Chris Lalancette

Comment 19 Alex Jia 2010-01-11 05:55:05 UTC
(In reply to comment #17)
> (In reply to comment #15)
> > Hi,Chris,I changed the above last testing step according to your comment
> > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> > acpiphp" successfully,it seems the kernel bug:
> > [root@dhcp-66-70-62 ~]# modprobe acpiphp
> > FATAL: Error inserting acpiphp
> > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> > device
> Maybe guest acpi is off by default here?? 
> IIRC, in RHEL5.4 Xen, hvm guest's acpi is off by default.
> You could double check this.
Dexuan,thanks for your comment,and I will check it.
> 
> > [root@dhcp-66-70-62 ~]# ls
> > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> > acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko

Comment 20 Alex Jia 2010-01-11 07:25:24 UTC
(In reply to comment #18)
> (In reply to comment #15)
> > Hi,Chris,I changed the above last testing step according to your comment
> > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> > acpiphp" successfully,it seems the kernel bug:
> > 
> > [root@dhcp-66-70-62 ~]# modprobe acpiphp
> > FATAL: Error inserting acpiphp
> > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> > device
> > 
> > [root@dhcp-66-70-62 ~]# ls
> > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> > acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko  
> 
> Hm, this is odd.  Did you do this *inside* the guest?  You need to do it in the
> guest, not on the host.  If you did do it in the guest, you should check that
> ACPI is enabled in the guest like Dexuan mentions.
Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded successfully.
> 
> The fact that it won't attach is another problem.  What happens if you use
> virsh attach-device to attach the device instead of virt-manager?  Does that
> work?  If not, does that give you any additional error information?
> 
> Chris Lalancette   
Actually,the above Comment 14 confused me,I think that host device is assigned directly to a guest,this process is pci passthrough not hotplug,they are completely different concept.I want to know the feature whether is pci passthrough,so that I retest it using correct steps.

Comment 21 Alex Jia 2010-01-11 07:44:28 UTC
Created attachment 382926 [details]
steps to reproduce

Comment 22 Chris Lalancette 2010-01-11 15:26:50 UTC
(In reply to comment #20)
> (In reply to comment #18)
> > (In reply to comment #15)
> > > Hi,Chris,I changed the above last testing step according to your comment
> > > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> > > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> > > acpiphp" successfully,it seems the kernel bug:
> > > 
> > > [root@dhcp-66-70-62 ~]# modprobe acpiphp
> > > FATAL: Error inserting acpiphp
> > > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> > > device
> > > 
> > > [root@dhcp-66-70-62 ~]# ls
> > > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> > > acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko  
> > 
> > Hm, this is odd.  Did you do this *inside* the guest?  You need to do it in the
> > guest, not on the host.  If you did do it in the guest, you should check that
> > ACPI is enabled in the guest like Dexuan mentions.
> Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded
> successfully.

OK, good.  So you have to do this step before you do the attach-device step.

> > 
> > The fact that it won't attach is another problem.  What happens if you use
> > virsh attach-device to attach the device instead of virt-manager?  Does that
> > work?  If not, does that give you any additional error information?
> > 
> > Chris Lalancette   
> Actually,the above Comment 14 confused me,I think that host device is assigned
> directly to a guest,this process is pci passthrough not hotplug,they are
> completely different concept.I want to know the feature whether is pci
> passthrough,so that I retest it using correct steps.    

In Comment #13, you tried to attach a Virtual Network device, *not* a PCI passthrough device.  That's the part that didn't make sense.

Additionally, PCI passthrough and hotplug are not completely different, they are related.  What we are trying to test here is whether PCI passthrough works, both at domain boot time and also when doing a hotplug/hot-unplug.  The steps you outline in Comment #21 look reasonable, except you didn't show me what's in hostdev.xml, so I can't tell for sure.  Please show me what's in hostdev.xml so I can confirm.

Finally, the error messages you see in the steps in Comment #21 look related to the problems in BZ 500217.  Let's try to figure that one out first, then we'll get back to this.

Chris Lalancette

Comment 23 Alex Jia 2010-01-15 02:29:26 UTC
(In reply to comment #22)
> (In reply to comment #20)
> > (In reply to comment #18)
> > > (In reply to comment #15)
> > > > Hi,Chris,I changed the above last testing step according to your comment
> > > > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> > > > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> > > > acpiphp" successfully,it seems the kernel bug:
> > > > 
> > > > [root@dhcp-66-70-62 ~]# modprobe acpiphp
> > > > FATAL: Error inserting acpiphp
> > > > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> > > > device
> > > > 
> > > > [root@dhcp-66-70-62 ~]# ls
> > > > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> > > > acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko  
> > > 
> > > Hm, this is odd.  Did you do this *inside* the guest?  You need to do it in the
> > > guest, not on the host.  If you did do it in the guest, you should check that
> > > ACPI is enabled in the guest like Dexuan mentions.
> > Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded
> > successfully.
> 
> OK, good.  So you have to do this step before you do the attach-device step.
> 
> > > 
> > > The fact that it won't attach is another problem.  What happens if you use
> > > virsh attach-device to attach the device instead of virt-manager?  Does that
> > > work?  If not, does that give you any additional error information?
> > > 
> > > Chris Lalancette   
> > Actually,the above Comment 14 confused me,I think that host device is assigned
> > directly to a guest,this process is pci passthrough not hotplug,they are
> > completely different concept.I want to know the feature whether is pci
> > passthrough,so that I retest it using correct steps.    
> 
> In Comment #13, you tried to attach a Virtual Network device, *not* a PCI
> passthrough device.  That's the part that didn't make sense.
> 
> Additionally, PCI passthrough and hotplug are not completely different, they
> are related.  What we are trying to test here is whether PCI passthrough works,
> both at domain boot time and also when doing a hotplug/hot-unplug.  The steps
> you outline in Comment #21 look reasonable, except you didn't show me what's in
> hostdev.xml, so I can't tell for sure.  Please show me what's in hostdev.xml so
> I can confirm.
Hi,Chris,the hostdev.xml looks like as following:
 [root@dhcp-66-70-62 ~]# cat hostdev.xml 
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
    </hostdev>

> 
> Finally, the error messages you see in the steps in Comment #21 look related to
> the problems in BZ 500217.  Let's try to figure that one out first, then we'll
> get back to this.
> 
> Chris Lalancette

Comment 24 Alex Jia 2010-01-17 07:56:38 UTC
Hi,Chris,the PCI passthrough will work if we degrade libvirt version to 0.6.3-23.el5,but it failed on libvirt-0.6.3-29.el5/libvirt-0.6.3-30.el5,
so this may be regression bug(see attachment).

Comment 25 Alex Jia 2010-01-17 08:10:34 UTC
Created attachment 384885 [details]
libvirt-0.6.3-23.el5 host information

Comment 26 Alex Jia 2010-01-17 08:11:54 UTC
Created attachment 384886 [details]
libvirt-0.6.3-23.el5 guest information

Comment 27 Alex Jia 2010-01-17 08:15:53 UTC
Created attachment 384888 [details]
libvirt-0.6.3-30.el5 host information

Comment 32 Alex Jia 2010-01-29 03:13:46 UTC
This bug has been fixed with libvirt-0.6.3-31.el5 on rhel5u5,set the bug status
to VERIFIED.(see the attachment)

Comment 33 Alex Jia 2010-01-29 03:14:29 UTC
Created attachment 387479 [details]
libvirt-0.6.3-31.el5 host information

Comment 34 Alex Jia 2010-01-29 03:15:50 UTC
Created attachment 387480 [details]
libvirt-0.6.3-31.el5 guest information

Comment 37 errata-xmlrpc 2010-03-30 08:09:50 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0205.html


Note You need to log in before you can comment on or make changes to this bug.