Bug 611715 - qemu-kvm gets no responsive when do hot-unplug pass-through device
Summary: qemu-kvm gets no responsive when do hot-unplug pass-through device
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
(Show other bugs)
Version: 6.0
Hardware: All Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Alex Williamson
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Keywords:
Depends On: 618788
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-07-06 09:27 UTC by juzhang
Modified: 2013-01-09 22:49 UTC (History)
6 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.95.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-11-10 21:26:05 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

Description juzhang 2010-07-06 09:27:10 UTC
Description of problem:
qemu-kvm gets no responsive  when do  hot-unplug pass-through device

Version-Release number of selected component (if applicable):
#rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.90.el6.x86_64
#uname -r
2.6.32-37.el6.x86_64


How reproducible:


Steps to Reproduce:
1. Unbind device from host kernel driver (PCI device 28:00.0)
1.1 lspci | grep Ethernet
lspci | grep Ethernet
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5764M Gigabit
Ethernet PCIe (rev 10)
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection
(rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection
(rev 01)
28:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) (rev 06)
1.2. #lspci -n | grep 28:00.0
28:00.0 0200: 8086:10b9 (rev 06)
1.3 echo "8086 10b9" >/sys/bus/pci/drivers/pci-stub/new_id
1.4 echo 0000:28:00.0 >/sys/bus/pci/devices/0000\:28\:00.0/driver/unbind
1.5 echo 0000:28:00.0 >/sys/bus/pci/drivers/pci-stub/bind
2. Boot guest with virtual and physical NIC card.
/usr/libexec/qemu-kvm -m 4G -smp 4 -drive file=/root/zhangjunyi/rhel6.32.raw,if=none,id=drive-virtio-disk0,boot=on,cache=none,format=raw -device virtio-blk-pci,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:11:22:45:66:97 -device virtio-balloon-pci -cpu qemu64,+sse2,+x2apic -monitor stdio -vnc :10 -qmp tcp:0:4445,server,nowait -drive file=/root/zhangjunyi/boot.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,drive=drive-ide0-1-0 -boot c -pcidevice host=28:00.0,id=zjy
3.hot-unplug physical NIC card host=28:00.0
(qemu) device_del zjy

Actual results:
qemu-kvm process gets no responsive.

Expected results:
physical NIC card host=28:00.0 is hot-unplug successful.

Additional info:
#top
 05:08:40 up 24 min,  3 users,  load average: 1.00, 0.95, 0.63
Tasks: 220 total,   2 running, 218 sleeping,   0 stopped,   0 zombie
Cpu(s):  6.6%us,  6.5%sy,  0.0%ni, 86.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  24654040k total,  5160116k used, 19493924k free,    16632k buffers
Swap:  4194296k total,        0k used,  4194296k free,   266924k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                         
 4141 root      20   0 4615m 4.0g 3512 R 100.2 17.2  16:47.00 qemu-kvm

Please note: qemu-kvm process  %CPU is 100.2

Please Note:
If Boot guest without physical NIC card, add physical NIC card after guest booted via command line.physical NIC card host=28:00.0 can be hot-unplug successful.the following is steps in details.

1. Boot guest with virtual
#/usr/libexec/qemu-kvm -m 4G -smp 4 -drive file=/root/zhangjunyi/rhel6.32.raw,if=none,id=drive-virtio-disk0,boot=on,cache=none,format=raw -device virtio-blk-pci,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:11:22:45:66:97 -device virtio-balloon-pci -cpu qemu64,+sse2,+x2apic -monitor stdio -vnc :10 -qmp tcp:0:4445,server,nowait -drive file=/root/zhangjunyi/boot.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,drive=drive-ide0-1-0 -boot c
2. hot-plug pass-through device
(qemu) device_add pci-assign,host=28:00.0,id=zjy
3.after step2, device was hot-plug successful.
(qemu) info pc
 Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc000 [0xc00f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1013:00b8
      BAR0: 32 bit prefetchable memory at 0xf0000000 [0xf1ffffff].
      BAR1: 32 bit memory at 0xf2000000 [0xf2000fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf2020000 [0xf2020fff].
      id "virtio-disk0"
  Bus  0, device   4, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 11.
      BAR0: I/O at 0xc080 [0xc09f].
      BAR1: 32 bit memory at 0xf2021000 [0xf2021fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id "net0"
  Bus  0, device   5, function 0:
    RAM controller: PCI device 1af4:1002
      IRQ 10.
      BAR0: I/O at 0xc0a0 [0xc0bf].
      id ""
  Bus  0, device   6, function 0:
    Ethernet controller: PCI device 8086:10b9
      IRQ 3.
      BAR0: 32 bit memory at 0xe0000000 [0xe001ffff].
      BAR1: 32 bit memory at 0xe0020000 [0xe003ffff].
      BAR2: I/O at 0x1000 [0x101f].
      BAR6: 32 bit prefetchable memory at 0xffffffffffffffff [0x0001fffe].
      id "zjy"

4. hot-unplug pass-through device
(qemu) device_del zjy
5.after step4, device was hot-unplug successful and qemu-kvm process is fine.
(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc000 [0xc00f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1013:00b8
      BAR0: 32 bit prefetchable memory at 0xf0000000 [0xf1ffffff].
      BAR1: 32 bit memory at 0xf2000000 [0xf2000fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf2020000 [0xf2020fff].
      id "virtio-disk0"
  Bus  0, device   4, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 11.
      BAR0: I/O at 0xc080 [0xc09f].
      BAR1: 32 bit memory at 0xf2021000 [0xf2021fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id "net0"
  Bus  0, device   5, function 0:
    RAM controller: PCI device 1af4:1002
      IRQ 10.
      BAR0: I/O at 0xc0a0 [0xc0bf].
      id ""

Comment 2 Don Dutile 2010-07-06 15:32:23 UTC
What does info pc show when pass-through device is passed to guest at boot time?

Comment 3 juzhang 2010-07-07 01:32:04 UTC
Boot guest with virtual and physical NIC card.

[root@dhcp-91-99 ~]# /usr/libexec/qemu-kvm -m 4G -smp 4 -drive file=/root/zhangjunyi/rhel6.32.raw,if=none,id=drive-virtio-disk0,boot=on,cache=none,format=raw -device virtio-blk-pci,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:11:22:45:66:97 -device virtio-balloon-pci -cpu qemu64,+sse2,+x2apic -monitor stdio -vnc :10 -qmp tcp:0:4445,server,nowait -drive file=/root/zhangjunyi/boot.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,drive=drive-ide0-1-0 -boot c -pcidevice host=28:00.0,id=zjy
device: zjy: driver="pci-assign" host="28:00.0"
QEMU 0.12.1 monitor - type 'help' for more information
(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc000 [0xc00f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1013:00b8
      BAR0: 32 bit prefetchable memory at 0xf0000000 [0xf1ffffff].
      BAR1: 32 bit memory at 0xf2000000 [0xf2000fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf2020000 [0xf2020fff].
      id "virtio-disk0"
  Bus  0, device   4, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 11.
      BAR0: I/O at 0xc080 [0xc09f].
      BAR1: 32 bit memory at 0xf2021000 [0xf2021fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id "net0"
  Bus  0, device   5, function 0:
    RAM controller: PCI device 1af4:1002
      IRQ 10.
      BAR0: I/O at 0xc0a0 [0xc0bf].
      id ""
  Bus  0, device   6, function 0:
    Ethernet controller: PCI device 8086:10b9
      IRQ 10.
      BAR0: 32 bit memory at 0xf2040000 [0xf205ffff].
      BAR1: 32 bit memory at 0xf2060000 [0xf207ffff].
      BAR2: I/O at 0xc0c0 [0xc0df].
      BAR6: 32 bit prefetchable memory at 0xffffffffffffffff [0x0001fffe].
      id "zjy"

Comment 4 Alex Williamson 2010-07-07 01:50:57 UTC
I see there's also an 82576 NIC in your system.  Does the same problem happen if you try to pass through either the physical or virtual functions of that device?

Comment 5 juzhang 2010-07-07 02:44:39 UTC
Yes a dual-port 82576 NIC in my system.
I tried two type of NIC(82576(physical) and e1000e),both hits this problem.

Comment 6 Alex Williamson 2010-07-07 03:57:32 UTC
Ok, I can reproduce with an 82576 physical function using your command line.  Can you confirm that the problem doesn't occur if you replace the "-monitor stdio" with "-monitor tcp::5678,server,nowait"?  You'll then need to "telnet localhost 5678" and issue the device_del command there.

Comment 7 juzhang 2010-07-07 04:33:27 UTC
I Using 82576 physical testing and using "tcp:0:5678,server,nowait" replace the stdio, can't hit this issue again.


/usr/libexec/qemu-kvm -m 4G -smp 4 -drive file=/root/zhangjunyi/rhel6.32.raw,if=none,id=drive-virtio-disk0,boot=on,cache=none,format=raw -device virtio-blk-pci,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:11:22:45:66:97 -device virtio-balloon-pci -cpu qemu64,+sse2,+x2apic -monitor tcp:0:5678,server,nowait -vnc :10 -qmp tcp:0:4445,server,nowait -drive file=/root/zhangjunyi/boot.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,drive=drive-ide0-1-0 -boot c -pcidevice host=03:00.1,id=zjy

Would you please tell me why?it's different bug?

Comment 8 Alex Williamson 2010-07-07 18:30:05 UTC
The problem is we're accidentally closing file descriptor zero on devices that have a PCI option rom.  With -monitor stdio, we close the file descriptor the monitor is using and things break.  The fix is pretty simple.

Comment 12 juzhang 2010-07-29 02:15:40 UTC
Tested on qemu-kvm-0.12.1.2-2.99.el6.this issue blocked by Bug 618788 - device-assignment hangs with kvm_run: Bad address.I will retest this issue once bz618788 is fixed.

When boot the guest,the qemu monitor emit the following messages" and the guest stops.
(qemu)kvm_run: Bad address
kvm_run returned -14

Comment 13 juzhang 2010-08-03 06:35:18 UTC
Verified on qemu-kvm-0.12.1.2-2.106.el6,passed.

Using the steps as same as comment0.
1.boot guest
#/usr/libexec/qemu-kvm -m 4G -smp 4 -drive file=/root/zhangjunyi/RHEL-Server-6.0-64-virtio.qcow2,if=none,id=drive-virtio-disk0,boot=on,cache=none,format=qcow2 -device virtio-blk-pci,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:11:22:45:66:97 -device virtio-balloon-pci -cpu qemu64,+sse2,+x2apic -monitor stdio -vnc :10 -qmp tcp:0:4445,server,nowait -boot c -pcidevice host=03:00.0,id=zjy
2.hot-unplug physical NIC card host=03:00.0 using qemu and qmp monitor
(qemu) device_del zjy
{"execute":"device_del","arguments":{"id":"zjy"}}


After step 2,physical NIC card host=03:00.0 is hot-unplug successful.

Comment 14 releng-rhel@redhat.com 2010-11-10 21:26:05 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.