Bug 517465 - Add support to libvirt for KVM PCI device assignment hotplug
Add support to libvirt for KVM PCI device assignment hotplug
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: libvirt (Show other bugs)
5.5
All Linux
medium Severity medium
: rc
: 5.5
Assigned To: Daniel Veillard
Virtualization Bugs
: FutureFeature, OtherQA
Depends On:
Blocks: 516837 532386 533941 557291
  Show dependency treegraph
 
Reported: 2009-08-14 04:04 EDT by Mark McLoughlin
Modified: 2010-03-30 04:09 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-03-30 04:09:50 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
initial refactoring patch (3.73 KB, patch)
2009-12-04 09:28 EST, Daniel Veillard
no flags Details | Diff
Add host PCI device hotplug support (17.43 KB, patch)
2009-12-04 09:31 EST, Daniel Veillard
no flags Details | Diff
dialog information (1.11 MB, image/png)
2010-01-08 04:55 EST, Alex Jia
no flags Details
steps to reproduce (1.82 KB, text/plain)
2010-01-11 02:44 EST, Alex Jia
no flags Details
libvirt-0.6.3-23.el5 host information (5.28 KB, text/plain)
2010-01-17 03:10 EST, Alex Jia
no flags Details
libvirt-0.6.3-23.el5 guest information (200.37 KB, image/png)
2010-01-17 03:11 EST, Alex Jia
no flags Details
libvirt-0.6.3-30.el5 host information (5.17 KB, text/plain)
2010-01-17 03:15 EST, Alex Jia
no flags Details
libvirt-0.6.3-31.el5 host information (5.55 KB, text/plain)
2010-01-28 22:14 EST, Alex Jia
no flags Details
libvirt-0.6.3-31.el5 guest information (77.80 KB, image/png)
2010-01-28 22:15 EST, Alex Jia
no flags Details

  None (edit)
Description Mark McLoughlin 2009-08-14 04:04:19 EDT
+++ This bug was initially created as a clone of Bug #517464 +++

libvirt supports KVM PCI device assignment since RHEL 5.4, but we do not support hotplug.

Patches to add this support have been committed upstream now:

  http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=7636ef4630
  http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=0c5b7b93a3
Comment 2 Chris Ward 2009-10-13 11:48:47 EDT
@Intel

We need to confirm that there is commitment to test 
for the resolution of this request during the RHEL 5.5 test
phase, if it is accepted into the release. 

Please post a confirmation before Oct 16th, 2009, 
including the contact information for testing engineers.
Comment 3 Jane Lv 2009-10-14 23:14:00 EDT
Dexuan,

Would you response comment #2?  Thanks.
Comment 4 Dexuan Cui 2009-11-02 01:38:32 EST
Add ddugger to CC.
Comment 7 Mark McLoughlin 2009-11-30 10:17:03 EST
Well, there's no virDomainSaveStatus() in 0.6.3, so even if you added XML_INTERNAL_STATUS it would never be used

IMHO, it's fine to just drop the domain_conf.c changes from the second patch - it just means that if you re-start libvirtd after hot-plugging a device, you won't be able to hot-unplug the device. I don't think libvirtd restart is reliable in 0.6.3, anyway
Comment 9 Daniel Veillard 2009-12-04 09:28:35 EST
Created attachment 376092 [details]
initial refactoring patch

Re-factor the hostdev hotplug code so that we can easily add PCI
hostdev hotplug to qemudDomainAttachHostDevice(). Based on
http://libvirt.org/git/?p=libvirt.git;a=commitdiff_plain;h=7636ef4630
Comment 10 Daniel Veillard 2009-12-04 09:31:07 EST
Created attachment 376094 [details]
Add host PCI device hotplug support

This patch adds the code needed for hotplug, it's based on 
http://libvirt.org/git/?p=libvirt.git;a=commitdiff_plain;h=0c5b7b93a3
but changed to add missing parts, and remove device assignment since
we always run libvirtd as root on RHEL 5.x
Comment 11 Daniel Veillard 2009-12-04 10:36:44 EST
libvirt-0.6.3-23.el5 has been built in dist-5E-qu-candidate with
the patches,

Daniel
Comment 13 Alex Jia 2009-12-30 04:56:09 EST
This bug still exist with libvirt-0.6.3-28.el5 on RHEL5u5,libvirt don't support
PCI device hotplug.

Version-Release number of selected component (if applicable):
[root@dhcp-66-70-62 ~]# uname -a
Linux dhcp-66-70-62.nay.redhat.com 2.6.18-183.el5 #1 SMP Mon Dec 21 18:37:42
EST 2009 x86_64 x86_64 x86_64 GNU/Linux

[root@dhcp-66-70-62 ~]# lsmod|grep kvm
kvm_intel              86664  0 
kvm                   223648  2 ksm,kvm_intel

[root@dhcp-66-70-62 libvirt]# rpm -qa|grep libvirt
libvirt-0.6.3-28.el5
libvirt-debuginfo-0.6.3-28.el5
libvirt-python-0.6.3-28.el5

[root@dhcp-66-70-62 ~]# rpm -q virt-manager kvm
virt-manager-0.6.1-11.el5
kvm-83-140.el5  


Step to Reproduce:
Using virt-manager tool to test:
  * Remove all NICs from a guest:
    Run virt-manager, open an existing shutoff status guest and go to the   
    Hardware tab,then choose "NIC :XX:XX:XX" and click "Remove" button
  * Start the guest 
  * Attach an interface to the guest
    Run virt-manager, open an running guest and go to the Hardware tab,then 
    click "Add hardware", choose "Network" and click "Forward",choose 
    "Virtual Network" default config,then click "Forward" and "Finish"

virt-manager will prompt a dialog,which content display "Are you sure you want
to add this device? This device could not be attached to the running machine ..."

If you choose "yes' button and log into guest using "lspci -t -v",you can't see the attached NIC device.and you will see the attached NIC device if rebooting guest.
Comment 14 Chris Lalancette 2010-01-07 14:02:59 EST
(In reply to comment #13)
> Step to Reproduce:
> Using virt-manager tool to test:
>   * Remove all NICs from a guest:
>     Run virt-manager, open an existing shutoff status guest and go to the   
>     Hardware tab,then choose "NIC :XX:XX:XX" and click "Remove" button
>   * Start the guest 
>   * Attach an interface to the guest
>     Run virt-manager, open an running guest and go to the Hardware tab,then 
>     click "Add hardware", choose "Network" and click "Forward",choose 
>     "Virtual Network" default config,then click "Forward" and "Finish"

Wait, this doesn't make any sense.  This bug is about PCI device assignment, yet in this last step you are trying to add a *virtual* network device.  What you actually want to do is go to "Add hardware", choose "Physical host device", hit "Forward", choose the device from the list, hit "Forward", and then hit "Finish".  That will test out this bug.  Note that in order for hotplug to actually work inside the guest, you need to do "modprobe acpiphp" before you do any of this.

If what I outline above doesn't work with libvirt-0.6.3-28.el5, please try out the test packages mentioned in Comment #8 in BZ 500217.  I tested exactly this scenario with those test packages, and it seems to work for me.

Chris Lalancette
Comment 15 Alex Jia 2010-01-08 04:52:20 EST
Hi,Chris,I changed the above last testing step according to your comment content,and tried libvirt-0.6.3-29 and you provide packages, but the result is failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe acpiphp" successfully,it seems the kernel bug:

[root@dhcp-66-70-62 ~]# modprobe acpiphp
FATAL: Error inserting acpiphp (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such device

[root@dhcp-66-70-62 ~]# ls /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko
Comment 16 Alex Jia 2010-01-08 04:55:09 EST
Created attachment 382428 [details]
dialog information
Comment 17 Dexuan Cui 2010-01-08 05:15:41 EST
(In reply to comment #15)
> Hi,Chris,I changed the above last testing step according to your comment
> content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> acpiphp" successfully,it seems the kernel bug:
> [root@dhcp-66-70-62 ~]# modprobe acpiphp
> FATAL: Error inserting acpiphp
> (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> device
Maybe guest acpi is off by default here?? 
IIRC, in RHEL5.4 Xen, hvm guest's acpi is off by default.
You could double check this.

> [root@dhcp-66-70-62 ~]# ls
> /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko
Comment 18 Chris Lalancette 2010-01-08 16:59:37 EST
(In reply to comment #15)
> Hi,Chris,I changed the above last testing step according to your comment
> content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> acpiphp" successfully,it seems the kernel bug:
> 
> [root@dhcp-66-70-62 ~]# modprobe acpiphp
> FATAL: Error inserting acpiphp
> (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> device
> 
> [root@dhcp-66-70-62 ~]# ls
> /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko  

Hm, this is odd.  Did you do this *inside* the guest?  You need to do it in the guest, not on the host.  If you did do it in the guest, you should check that ACPI is enabled in the guest like Dexuan mentions.

The fact that it won't attach is another problem.  What happens if you use virsh attach-device to attach the device instead of virt-manager?  Does that work?  If not, does that give you any additional error information?

Chris Lalancette
Comment 19 Alex Jia 2010-01-11 00:55:05 EST
(In reply to comment #17)
> (In reply to comment #15)
> > Hi,Chris,I changed the above last testing step according to your comment
> > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> > acpiphp" successfully,it seems the kernel bug:
> > [root@dhcp-66-70-62 ~]# modprobe acpiphp
> > FATAL: Error inserting acpiphp
> > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> > device
> Maybe guest acpi is off by default here?? 
> IIRC, in RHEL5.4 Xen, hvm guest's acpi is off by default.
> You could double check this.
Dexuan,thanks for your comment,and I will check it.
> 
> > [root@dhcp-66-70-62 ~]# ls
> > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> > acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko
Comment 20 Alex Jia 2010-01-11 02:25:24 EST
(In reply to comment #18)
> (In reply to comment #15)
> > Hi,Chris,I changed the above last testing step according to your comment
> > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> > acpiphp" successfully,it seems the kernel bug:
> > 
> > [root@dhcp-66-70-62 ~]# modprobe acpiphp
> > FATAL: Error inserting acpiphp
> > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> > device
> > 
> > [root@dhcp-66-70-62 ~]# ls
> > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> > acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko  
> 
> Hm, this is odd.  Did you do this *inside* the guest?  You need to do it in the
> guest, not on the host.  If you did do it in the guest, you should check that
> ACPI is enabled in the guest like Dexuan mentions.
Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded successfully.
> 
> The fact that it won't attach is another problem.  What happens if you use
> virsh attach-device to attach the device instead of virt-manager?  Does that
> work?  If not, does that give you any additional error information?
> 
> Chris Lalancette   
Actually,the above Comment 14 confused me,I think that host device is assigned directly to a guest,this process is pci passthrough not hotplug,they are completely different concept.I want to know the feature whether is pci passthrough,so that I retest it using correct steps.
Comment 21 Alex Jia 2010-01-11 02:44:28 EST
Created attachment 382926 [details]
steps to reproduce
Comment 22 Chris Lalancette 2010-01-11 10:26:50 EST
(In reply to comment #20)
> (In reply to comment #18)
> > (In reply to comment #15)
> > > Hi,Chris,I changed the above last testing step according to your comment
> > > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> > > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> > > acpiphp" successfully,it seems the kernel bug:
> > > 
> > > [root@dhcp-66-70-62 ~]# modprobe acpiphp
> > > FATAL: Error inserting acpiphp
> > > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> > > device
> > > 
> > > [root@dhcp-66-70-62 ~]# ls
> > > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> > > acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko  
> > 
> > Hm, this is odd.  Did you do this *inside* the guest?  You need to do it in the
> > guest, not on the host.  If you did do it in the guest, you should check that
> > ACPI is enabled in the guest like Dexuan mentions.
> Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded
> successfully.

OK, good.  So you have to do this step before you do the attach-device step.

> > 
> > The fact that it won't attach is another problem.  What happens if you use
> > virsh attach-device to attach the device instead of virt-manager?  Does that
> > work?  If not, does that give you any additional error information?
> > 
> > Chris Lalancette   
> Actually,the above Comment 14 confused me,I think that host device is assigned
> directly to a guest,this process is pci passthrough not hotplug,they are
> completely different concept.I want to know the feature whether is pci
> passthrough,so that I retest it using correct steps.    

In Comment #13, you tried to attach a Virtual Network device, *not* a PCI passthrough device.  That's the part that didn't make sense.

Additionally, PCI passthrough and hotplug are not completely different, they are related.  What we are trying to test here is whether PCI passthrough works, both at domain boot time and also when doing a hotplug/hot-unplug.  The steps you outline in Comment #21 look reasonable, except you didn't show me what's in hostdev.xml, so I can't tell for sure.  Please show me what's in hostdev.xml so I can confirm.

Finally, the error messages you see in the steps in Comment #21 look related to the problems in BZ 500217.  Let's try to figure that one out first, then we'll get back to this.

Chris Lalancette
Comment 23 Alex Jia 2010-01-14 21:29:26 EST
(In reply to comment #22)
> (In reply to comment #20)
> > (In reply to comment #18)
> > > (In reply to comment #15)
> > > > Hi,Chris,I changed the above last testing step according to your comment
> > > > content,and tried libvirt-0.6.3-29 and you provide packages, but the result is
> > > > failed,a dialog is raised by virt-manager(see attachment).and I can't "modprobe
> > > > acpiphp" successfully,it seems the kernel bug:
> > > > 
> > > > [root@dhcp-66-70-62 ~]# modprobe acpiphp
> > > > FATAL: Error inserting acpiphp
> > > > (/lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/acpiphp.ko): No such
> > > > device
> > > > 
> > > > [root@dhcp-66-70-62 ~]# ls
> > > > /lib/modules/2.6.18-183.el5/kernel/drivers/pci/hotplug/
> > > > acpiphp_ibm.ko  acpiphp.ko  fakephp.ko  pciehp.ko  shpchp.ko  
> > > 
> > > Hm, this is odd.  Did you do this *inside* the guest?  You need to do it in the
> > > guest, not on the host.  If you did do it in the guest, you should check that
> > > ACPI is enabled in the guest like Dexuan mentions.
> > Hi,Chris,I recheck acpiphp module inside the guest,the module can be loaded
> > successfully.
> 
> OK, good.  So you have to do this step before you do the attach-device step.
> 
> > > 
> > > The fact that it won't attach is another problem.  What happens if you use
> > > virsh attach-device to attach the device instead of virt-manager?  Does that
> > > work?  If not, does that give you any additional error information?
> > > 
> > > Chris Lalancette   
> > Actually,the above Comment 14 confused me,I think that host device is assigned
> > directly to a guest,this process is pci passthrough not hotplug,they are
> > completely different concept.I want to know the feature whether is pci
> > passthrough,so that I retest it using correct steps.    
> 
> In Comment #13, you tried to attach a Virtual Network device, *not* a PCI
> passthrough device.  That's the part that didn't make sense.
> 
> Additionally, PCI passthrough and hotplug are not completely different, they
> are related.  What we are trying to test here is whether PCI passthrough works,
> both at domain boot time and also when doing a hotplug/hot-unplug.  The steps
> you outline in Comment #21 look reasonable, except you didn't show me what's in
> hostdev.xml, so I can't tell for sure.  Please show me what's in hostdev.xml so
> I can confirm.
Hi,Chris,the hostdev.xml looks like as following:
 [root@dhcp-66-70-62 ~]# cat hostdev.xml 
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
    </hostdev>

> 
> Finally, the error messages you see in the steps in Comment #21 look related to
> the problems in BZ 500217.  Let's try to figure that one out first, then we'll
> get back to this.
> 
> Chris Lalancette
Comment 24 Alex Jia 2010-01-17 02:56:38 EST
Hi,Chris,the PCI passthrough will work if we degrade libvirt version to 0.6.3-23.el5,but it failed on libvirt-0.6.3-29.el5/libvirt-0.6.3-30.el5,
so this may be regression bug(see attachment).
Comment 25 Alex Jia 2010-01-17 03:10:34 EST
Created attachment 384885 [details]
libvirt-0.6.3-23.el5 host information
Comment 26 Alex Jia 2010-01-17 03:11:54 EST
Created attachment 384886 [details]
libvirt-0.6.3-23.el5 guest information
Comment 27 Alex Jia 2010-01-17 03:15:53 EST
Created attachment 384888 [details]
libvirt-0.6.3-30.el5 host information
Comment 32 Alex Jia 2010-01-28 22:13:46 EST
This bug has been fixed with libvirt-0.6.3-31.el5 on rhel5u5,set the bug status
to VERIFIED.(see the attachment)
Comment 33 Alex Jia 2010-01-28 22:14:29 EST
Created attachment 387479 [details]
libvirt-0.6.3-31.el5 host information
Comment 34 Alex Jia 2010-01-28 22:15:50 EST
Created attachment 387480 [details]
libvirt-0.6.3-31.el5 guest information
Comment 37 errata-xmlrpc 2010-03-30 04:09:50 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0205.html

Note You need to log in before you can comment on or make changes to this bug.