Bug 685147
| Summary: | guest with assigned nic got kernel panic when send system_reset signal in QEMU monitor | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Chao Yang <chayang> | ||||||||
| Component: | qemu-kvm | Assignee: | Alex Williamson <alex.williamson> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 6.1 | CC: | ajia, alex.williamson, ddutile, juzhang, michen, mkenneth, tburke, virt-maint | ||||||||
| Target Milestone: | rc | Keywords: | Triaged | ||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | qemu-kvm-0.12.1.2-2.152.el6 | Doc Type: | Bug Fix | ||||||||
| Doc Text: |
Cause:
Assigned devices may continue DMA operations across a reset of the VM.
Consequence:
Guest memory buffers are reset across reboot, resulting in continuing DMA operations possibly overwriting guest memory.
Fix:
Assigned devices are reset on VM reset.
Result:
Assigned device is quiesced and does not continue to DMA across a VM reset.
|
Story Points: | --- | ||||||||
| Clone Of: | |||||||||||
| : | 689002 (view as bug list) | Environment: | |||||||||
| Last Closed: | 2011-05-19 11:24:01 UTC | Type: | --- | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 689002 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 484451 [details]
boot guest with 82576 assigned
I mark this bug as a testblocker, cause it blocks device assignment/ MSI tests. how reproducible: 4/4 I tested 4 times on a fresh installed rhel6.1 32bit os with following cli: # /usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -m 4096 -smp 4 -cpu cpu64-rhel6 -name rhel6.1-32 -uuid `uuidgen` -rtc base=localtime,clock=vm,driftfix=slew -no-kvm-pit-reinjection -boot c -drive file=/root/image/rhel6.1-32.qcow2,if=none,id=drive-virtio-0-0,media=disk,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-virtio-0-0,id=virt0-0-0 -netdev tap,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:40:31:61:23 -usb -device usb-tablet,id=input1 -vnc :0 -monitor stdio -balloon none -device pci-assign,host=02:00.0,id=bcm5709 -serial unix:/root/seri,server,nowait # nc -U /root/seri NMI watchdog disabled for cpu0: unable to create perf event: -2 NMI watchdog disabled for cpu1: unable to create perf event: -2 NMI watchdog disabled for cpu2: unable to create perf event: -2 NMI watchdog disabled for cpu3: unable to create perf event: -2 � Welcome to Red Hat Enterprise Linux Server Starting udev: bnx2 0000:00:05.0: pci_enable_pcie_error_reporting failed 0xfffffffb BUG: unable to handle kernel NULL pointer dereference at 0000000000000074 IP: [<ffffffff811475ec>] valid_swaphandles+0x6c/0x160 PGD 115f25067 PUD 116654067 PMD 0 Oops: 0000 [#1] SMP last sysfs file: /sys/devices/virtual/block/dm-1/dev CPU 0 Modules linked in: bnx2(+) i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: scsi_wait_scan] Modules linked in: bnx2(+) i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: scsi_wait_scan] Pid: 681, comm: blkid Not tainted 2.6.32-122.el6.x86_64 #1 KVM RIP: 0010:[<ffffffff811475ec>] [<ffffffff811475ec>] valid_swaphandles+0x6c/0x160 RSP: 0000:ffff880119e51c28 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 00000000001da920 RCX: 0000000000000003 RDX: 0000000000000000 RSI: ffff880119e51c90 RDI: ffffffff81ee5130 RBP: ffff880119e51c68 R08: ffff880118588860 R09: 0000000000000008 R10: 0000000000000051 R11: 0000000000000246 R12: 00000000001da921 R13: 00000000001da920 R14: 0000000000000000 R15: ffff880119e51c90 FS: 00007fb4a1f3c740(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000074 CR3: 0000000114fb5000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process blkid (pid: 681, threadinfo ffff880119e50000, task ffff880116eb2040) Stack: ffff880119e51c78 ffffffff00000003 ffff880119e51c88 ffff880115f2ca20 <0> 00000000000200da ffff880115f2ca20 00007fb4a190b668 00007fb4a190b668 <0> ffff880119e51cc8 ffffffff81146c5c ffff880119e51cd8 28000000001da921 Call Trace: [<ffffffff81146c5c>] swapin_readahead+0x2c/0xc0 [<ffffffff8118a739>] ? touch_atime+0x69/0x170 [<ffffffff811377db>] handle_pte_fault+0x70b/0xb40 [<ffffffff8110ea70>] ? generic_file_aio_read+0x380/0x700 [<ffffffff81137dfd>] handle_mm_fault+0x1ed/0x2b0 [<ffffffff81170d1a>] ? do_sync_read+0xfa/0x140 [<ffffffff810412d9>] __do_page_fault+0x139/0x480 [<ffffffff8121060b>] ? selinux_file_permission+0xfb/0x150 [<ffffffff814dedae>] do_page_fault+0x3e/0xa0 [<ffffffff814dc155>] page_fault+0x25/0x30 Code: 3b 48 c7 c7 30 51 ee 81 4d 89 e5 4c 8b 34 c5 40 51 ee 81 89 4d c8 49 d3 ed 49 d3 e5 4d 85 ed 49 0f 45 dd e8 47 46 39 00 8b 4d c8 <41> 8b 56 74 b8 01 00 00 00 d3 e0 48 98 49 01 c5 49 39 d5 4c 0f RIP [<ffffffff811475ec>] valid_swaphandles+0x6c/0x160 RSP <ffff880119e51c28> CR2: 0000000000000074 ---[ end trace 0942b589781074d8 ]--- Kernel panic - not syncing: Fatal exception Pid: 681, comm: blkid Tainted: G D ---------------- 2.6.32-122.el6.x86_64 #1 Call Trace: [<ffffffff814d8d99>] ? panic+0x78/0x143 [<ffffffff814dcde4>] ? oops_end+0xe4/0x100 [<ffffffff81040a8b>] ? no_context+0xfb/0x260 [<ffffffff81040d15>] ? __bad_area_nosemaphore+0x125/0x1e0 [<ffffffff81247a5e>] ? generic_make_request+0x21e/0x5b0 [<ffffffff81040e3e>] ? bad_area+0x4e/0x60 [<ffffffff81041563>] ? __do_page_fault+0x3c3/0x480 [<ffffffff8100987e>] ? __switch_to+0x26e/0x320 [<ffffffff810370d8>] ? pvclock_clocksource_read+0x58/0xd0 [<ffffffff814dedae>] ? do_page_fault+0x3e/0xa0 [<ffffffff814dc155>] ? page_fault+0x25/0x30 [<ffffffff811475ec>] ? valid_swaphandles+0x6c/0x160 [<ffffffff811475e9>] ? valid_swaphandles+0x69/0x160 [<ffffffff81146c5c>] ? swapin_readahead+0x2c/0xc0 [<ffffffff8118a739>] ? touch_atime+0x69/0x170 [<ffffffff811377db>] ? handle_pte_fault+0x70b/0xb40 [<ffffffff8110ea70>] ? generic_file_aio_read+0x380/0x700 [<ffffffff81137dfd>] ? handle_mm_fault+0x1ed/0x2b0 [<ffffffff81170d1a>] ? do_sync_read+0xfa/0x140 [<ffffffff810412d9>] ? __do_page_fault+0x139/0x480 [<ffffffff8121060b>] ? selinux_file_permission+0xfb/0x150 [<ffffffff814dedae>] ? do_page_fault+0x3e/0xa0 [<ffffffff814dc155>] ? page_fault+0x25/0x30 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 63297 root 20 0 4700m 4.0g 3044 S 107.2 0.9 1:07.13 qemu-kvm 63365 root 20 0 15808 2120 956 R 2.0 0.0 0:00.26 top Created attachment 485703 [details]
lspci -vvv info of BCM5709
guest was assigned BCM5709 card for the four times tests.
I've reproduced this. It looks like the card is continuing to DMA to memory after the system is reset. Upstream has added a reset function to write to the command register of the card to stop DMA, but it doesn't seem to be sufficient. I've sand boxed a fix that writes to the pci sysfs reset file from the reset function to do the same kind of hard device reset as when it's first assigned. This appears to fix the problem. I can reproduce the bug with the above test environment(qemu-kvm-0.12.1.2-2.150.el6.x86_64, NICs is BCM5709 with MSI capability)
And the bug has been verified on rhel6.1(2.6.32-122.el6.x86_64) with qemu-kvm-0.12.1.2-2.152.el6.x86_64.
From libvirt point of view, I can't execute reset action for the device by virsh command:
# virsh nodedev-dettach pci_0000_01_00_1
Device pci_0000_01_00_1 dettached
# ls /sys/bus/pci/drivers/bnx2/
0000:02:00.0 0000:02:00.1 bind module new_id remove_id uevent unbind
# ls /sys/bus/pci/drivers/pci-stub/
0000:01:00.0 0000:01:00.1 0000:09:00.0 bind new_id remove_id uevent unbind
# virsh nodedev-reset pci_0000_01_00_1
error: Failed to reset device pci_0000_01_00_1
error: this function is not supported by the connection driver: Unable to reset PCI device 0000:01:00.1: this function is not supported by the connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus reset
And it's also fail to directly hot-plug/cold-plug the device to guest by virt-manager, the same error information will be raise:
Error starting domain: this function is not supported by the connection driver: Unable to reset PCI device 0000:01:00.1: this function is not supported by the connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus reset
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 45, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/engine.py", line 956, in asyncfunc
vm.startup()
File "/usr/share/virt-manager/virtManager/domain.py", line 1048, in startup
self._backend.create()
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 325, in create
if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: this function is not supported by the connection driver: Unable to reset PCI device 0000:01:00.1: this function is not supported by the connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus reset
# uname -r
2.6.32-122.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.152.el6.x86_64
# rpm -q libvirt
libvirt-0.8.7-14.el6.x86_64
# rpm -q virt-manager
virt-manager-0.8.6-3.el6.noarch
# rpm -q python-virtinst
python-virtinst-0.500.5-2.el6.noarch
(In reply to comment #14) > I can reproduce the bug with the above test > environment(qemu-kvm-0.12.1.2-2.150.el6.x86_64, NICs is BCM5709 with MSI > capability) > > And the bug has been verified on rhel6.1(2.6.32-122.el6.x86_64) with > qemu-kvm-0.12.1.2-2.152.el6.x86_64. > > From libvirt point of view, I can't execute reset action for the device by > virsh command: > # virsh nodedev-dettach pci_0000_01_00_1 > Device pci_0000_01_00_1 dettached > > # ls /sys/bus/pci/drivers/bnx2/ > 0000:02:00.0 0000:02:00.1 bind module new_id remove_id uevent unbind > > # ls /sys/bus/pci/drivers/pci-stub/ > 0000:01:00.0 0000:01:00.1 0000:09:00.0 bind new_id remove_id uevent > unbind > > # virsh nodedev-reset pci_0000_01_00_1 > error: Failed to reset device pci_0000_01_00_1 > error: this function is not supported by the connection driver: Unable to reset > PCI device 0000:01:00.1: this function is not supported by the connection > driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus > reset I don't understand what you're trying to do here. The original bug was shown by issuing a system_reset on the qemu monitor. What you're trying to do is a completely different test. If you want to verify using libvirt, you can do so using the qemu-monitor-command option of virsh with the --hmp option for system_reset. Note that you'll need the libvirt fix for bz689002 if testing via libvirt. > And it's also fail to directly hot-plug/cold-plug the device to guest by > virt-manager, the same error information will be raise: I don't understand what you're trying to do here. > Error starting domain: this function is not supported by the connection driver: > Unable to reset PCI device 0000:01:00.1: this function is not supported by the > connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not > doing bus reset > > Traceback (most recent call last): > File "/usr/share/virt-manager/virtManager/asyncjob.py", line 45, in > cb_wrapper > callback(asyncjob, *args, **kwargs) > File "/usr/share/virt-manager/virtManager/engine.py", line 956, in asyncfunc > vm.startup() > File "/usr/share/virt-manager/virtManager/domain.py", line 1048, in startup > self._backend.create() > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 325, in create > if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) > libvirtError: this function is not supported by the connection driver: Unable > to reset PCI device 0000:01:00.1: this function is not supported by the > connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not > doing bus reset > > # uname -r > 2.6.32-122.el6.x86_64 > > # rpm -q qemu-kvm > qemu-kvm-0.12.1.2-2.152.el6.x86_64 > > # rpm -q libvirt > libvirt-0.8.7-14.el6.x86_64 > > # rpm -q virt-manager > virt-manager-0.8.6-3.el6.noarch > > # rpm -q python-virtinst > python-virtinst-0.500.5-2.el6.noarch move it to verified based on commnet#14
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Cause:
Assigned devices may continue DMA operations across a reset of the VM.
Consequence:
Guest memory buffers are reset across reboot, resulting in continuing DMA operations possibly overwriting guest memory.
Fix:
Assigned devices are reset on VM reset.
Result:
Assigned device is quiesced and does not continue to DMA across a VM reset.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html |
Created attachment 484450 [details] boot with BCM5709 assigned Description of problem: Boot a rhel6.1 32 bit guest with a nic with MSI capability, guest got kernel panic when send system_reset in QEMU monitor. Version-Release number of selected component (if applicable): # uname -r 2.6.32-122.el6.x86_64 # rpm -qa|grep qemu qemu-kvm-0.12.1.2-2.150.el6.x86_64 qemu-img-0.12.1.2-2.150.el6.x86_64 qemu-kvm-tools-0.12.1.2-2.150.el6.x86_64 gpxe-roms-qemu-0.9.7-6.7.el6.noarch qemu-kvm-debuginfo-0.12.1.2-2.150.el6.x86_64 CLI: # /usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -m 4096 -smp 4 -cpu cpu64-rhel6 -name rhel6.1-32 -uuid `uuidgen` -rtc base=localtime,clock=vm,driftfix=slew -no-kvm-pit-reinjection -boot c -drive file=/root/image/rhel6.1-32-virtio.qcow2,if=none,id=drive-virtio-0-0,media=disk,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-virtio-0-0,id=virt0-0-0 -net none -usb -device usb-tablet,id=input1 -vnc :0 -monitor stdio -balloon none -device pci-assign,host=02:00.0,id=BC,addr=0x6 -serial unix:/root/moni,server,nowait # lspci|grep Eth 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) How reproducible: always Steps to Reproduce: 1.Boot guest using cli above 2.Open a terminal, and ping remote host 3.send system_reset in monitor Actual results: Guest hangs, send system_reset again, then got kernel panic Expected results: guest should not get kernel panic. Additional info: Tried with BCM5709 and 82576, both hit this issue.