RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 685147 - guest with assigned nic got kernel panic when send system_reset signal in QEMU monitor
Summary: guest with assigned nic got kernel panic when send system_reset signal in QEM...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Alex Williamson
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 689002
TreeView+ depends on / blocked
 
Reported: 2011-03-15 12:32 UTC by Chao Yang
Modified: 2013-01-09 23:39 UTC (History)
8 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.152.el6
Doc Type: Bug Fix
Doc Text:
Cause: Assigned devices may continue DMA operations across a reset of the VM. Consequence: Guest memory buffers are reset across reboot, resulting in continuing DMA operations possibly overwriting guest memory. Fix: Assigned devices are reset on VM reset. Result: Assigned device is quiesced and does not continue to DMA across a VM reset.
Clone Of:
: 689002 (view as bug list)
Environment:
Last Closed: 2011-05-19 11:24:01 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
boot with BCM5709 assigned (4.06 KB, text/plain)
2011-03-15 12:32 UTC, Chao Yang 		no flags Details
boot guest with 82576 assigned (21.44 KB, text/plain)
2011-03-15 12:34 UTC, Chao Yang 		no flags Details
lspci -vvv info of BCM5709 (2.95 KB, text/plain)
2011-03-16 10:41 UTC, Chao Yang 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2011:0534 0 normal SHIPPED_LIVE Important: qemu-kvm security, bug fix, and enhancement update 2011-05-19 11:20:36 UTC

Description Chao Yang 2011-03-15 12:32:02 UTC 
Created attachment 484450 [details]
boot with BCM5709 assigned

Description of problem:
Boot a rhel6.1 32 bit guest with a nic with MSI capability, guest got kernel panic when send system_reset in QEMU monitor.

Version-Release number of selected component (if applicable):
# uname -r
2.6.32-122.el6.x86_64
# rpm -qa|grep qemu
qemu-kvm-0.12.1.2-2.150.el6.x86_64
qemu-img-0.12.1.2-2.150.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.150.el6.x86_64
gpxe-roms-qemu-0.9.7-6.7.el6.noarch
qemu-kvm-debuginfo-0.12.1.2-2.150.el6.x86_64

CLI:
# /usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -m 4096 -smp 4 -cpu cpu64-rhel6 -name rhel6.1-32 -uuid `uuidgen` -rtc base=localtime,clock=vm,driftfix=slew -no-kvm-pit-reinjection -boot c -drive file=/root/image/rhel6.1-32-virtio.qcow2,if=none,id=drive-virtio-0-0,media=disk,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-virtio-0-0,id=virt0-0-0 -net none -usb -device usb-tablet,id=input1 -vnc :0 -monitor stdio -balloon none -device pci-assign,host=02:00.0,id=BC,addr=0x6 -serial unix:/root/moni,server,nowait

# lspci|grep Eth
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

How reproducible:
always

Steps to Reproduce:
1.Boot guest using cli above
2.Open a terminal, and ping remote host
3.send system_reset in monitor
  
Actual results:
Guest hangs, send system_reset again, then got kernel panic

Expected results:
guest should not get kernel panic.

Additional info:
Tried with BCM5709 and 82576, both hit this issue.
Comment 2 Chao Yang 2011-03-15 12:34:42 UTC 
Created attachment 484451 [details]
boot guest with 82576 assigned
Comment 3 Chao Yang 2011-03-16 10:37:15 UTC 
I mark this bug as a testblocker, cause it blocks device assignment/ MSI tests.
how reproducible: 4/4
I tested 4 times on a fresh installed rhel6.1 32bit os with following cli:

# /usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -m 4096 -smp 4 -cpu cpu64-rhel6 -name rhel6.1-32 -uuid `uuidgen` -rtc base=localtime,clock=vm,driftfix=slew -no-kvm-pit-reinjection -boot c -drive file=/root/image/rhel6.1-32.qcow2,if=none,id=drive-virtio-0-0,media=disk,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-virtio-0-0,id=virt0-0-0 -netdev tap,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:40:31:61:23 -usb -device usb-tablet,id=input1 -vnc :0 -monitor stdio -balloon none -device pci-assign,host=02:00.0,id=bcm5709 -serial unix:/root/seri,server,nowait

# nc -U /root/seri 
NMI watchdog disabled for cpu0: unable to create perf event: -2
NMI watchdog disabled for cpu1: unable to create perf event: -2
NMI watchdog disabled for cpu2: unable to create perf event: -2
NMI watchdog disabled for cpu3: unable to create perf event: -2
�		Welcome to Red Hat Enterprise Linux Server
Starting udev: bnx2 0000:00:05.0: pci_enable_pcie_error_reporting failed 0xfffffffb
BUG: unable to handle kernel NULL pointer dereference at 0000000000000074
IP: [<ffffffff811475ec>] valid_swaphandles+0x6c/0x160
PGD 115f25067 PUD 116654067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/virtual/block/dm-1/dev
CPU 0 
Modules linked in: bnx2(+) i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: scsi_wait_scan]

Modules linked in: bnx2(+) i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: scsi_wait_scan]
Pid: 681, comm: blkid Not tainted 2.6.32-122.el6.x86_64 #1 KVM
RIP: 0010:[<ffffffff811475ec>]  [<ffffffff811475ec>] valid_swaphandles+0x6c/0x160
RSP: 0000:ffff880119e51c28  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 00000000001da920 RCX: 0000000000000003
RDX: 0000000000000000 RSI: ffff880119e51c90 RDI: ffffffff81ee5130
RBP: ffff880119e51c68 R08: ffff880118588860 R09: 0000000000000008
R10: 0000000000000051 R11: 0000000000000246 R12: 00000000001da921
R13: 00000000001da920 R14: 0000000000000000 R15: ffff880119e51c90
FS:  00007fb4a1f3c740(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000074 CR3: 0000000114fb5000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process blkid (pid: 681, threadinfo ffff880119e50000, task ffff880116eb2040)
Stack:
 ffff880119e51c78 ffffffff00000003 ffff880119e51c88 ffff880115f2ca20
<0> 00000000000200da ffff880115f2ca20 00007fb4a190b668 00007fb4a190b668
<0> ffff880119e51cc8 ffffffff81146c5c ffff880119e51cd8 28000000001da921
Call Trace:
 [<ffffffff81146c5c>] swapin_readahead+0x2c/0xc0
 [<ffffffff8118a739>] ? touch_atime+0x69/0x170
 [<ffffffff811377db>] handle_pte_fault+0x70b/0xb40
 [<ffffffff8110ea70>] ? generic_file_aio_read+0x380/0x700
 [<ffffffff81137dfd>] handle_mm_fault+0x1ed/0x2b0
 [<ffffffff81170d1a>] ? do_sync_read+0xfa/0x140
 [<ffffffff810412d9>] __do_page_fault+0x139/0x480
 [<ffffffff8121060b>] ? selinux_file_permission+0xfb/0x150
 [<ffffffff814dedae>] do_page_fault+0x3e/0xa0
 [<ffffffff814dc155>] page_fault+0x25/0x30
Code: 3b 48 c7 c7 30 51 ee 81 4d 89 e5 4c 8b 34 c5 40 51 ee 81 89 4d c8 49 d3 ed 49 d3 e5 4d 85 ed 49 0f 45 dd e8 47 46 39 00 8b 4d c8 <41> 8b 56 74 b8 01 00 00 00 d3 e0 48 98 49 01 c5 49 39 d5 4c 0f 
RIP  [<ffffffff811475ec>] valid_swaphandles+0x6c/0x160
 RSP <ffff880119e51c28>
CR2: 0000000000000074
---[ end trace 0942b589781074d8 ]---
Kernel panic - not syncing: Fatal exception
Pid: 681, comm: blkid Tainted: G      D    ----------------   2.6.32-122.el6.x86_64 #1
Call Trace:
 [<ffffffff814d8d99>] ? panic+0x78/0x143
 [<ffffffff814dcde4>] ? oops_end+0xe4/0x100
 [<ffffffff81040a8b>] ? no_context+0xfb/0x260
 [<ffffffff81040d15>] ? __bad_area_nosemaphore+0x125/0x1e0
 [<ffffffff81247a5e>] ? generic_make_request+0x21e/0x5b0
 [<ffffffff81040e3e>] ? bad_area+0x4e/0x60
 [<ffffffff81041563>] ? __do_page_fault+0x3c3/0x480
 [<ffffffff8100987e>] ? __switch_to+0x26e/0x320
 [<ffffffff810370d8>] ? pvclock_clocksource_read+0x58/0xd0
 [<ffffffff814dedae>] ? do_page_fault+0x3e/0xa0
 [<ffffffff814dc155>] ? page_fault+0x25/0x30
 [<ffffffff811475ec>] ? valid_swaphandles+0x6c/0x160
 [<ffffffff811475e9>] ? valid_swaphandles+0x69/0x160
 [<ffffffff81146c5c>] ? swapin_readahead+0x2c/0xc0
 [<ffffffff8118a739>] ? touch_atime+0x69/0x170
 [<ffffffff811377db>] ? handle_pte_fault+0x70b/0xb40
 [<ffffffff8110ea70>] ? generic_file_aio_read+0x380/0x700
 [<ffffffff81137dfd>] ? handle_mm_fault+0x1ed/0x2b0
 [<ffffffff81170d1a>] ? do_sync_read+0xfa/0x140
 [<ffffffff810412d9>] ? __do_page_fault+0x139/0x480
 [<ffffffff8121060b>] ? selinux_file_permission+0xfb/0x150
 [<ffffffff814dedae>] ? do_page_fault+0x3e/0xa0
 [<ffffffff814dc155>] ? page_fault+0x25/0x30


  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                   
63297 root      20   0 4700m 4.0g 3044 S 107.2  0.9   1:07.13 qemu-kvm                                                                 
63365 root      20   0 15808 2120  956 R  2.0  0.0   0:00.26 top
Comment 5 Chao Yang 2011-03-16 10:41:38 UTC 
Created attachment 485703 [details]
lspci -vvv info of BCM5709

guest was assigned BCM5709 card for the four times tests.
Comment 7 Alex Williamson 2011-03-17 05:17:34 UTC 
I've reproduced this.  It looks like the card is continuing to DMA to memory after the system is reset.  Upstream has added a reset function to write to the command register of the card to stop DMA, but it doesn't seem to be sufficient.  I've sand boxed a fix that writes to the pci sysfs reset file from the reset function to do the same kind of hard device reset as when it's first assigned.  This appears to fix the problem.
Comment 14 Alex Jia 2011-03-25 10:49:10 UTC 
I can reproduce the bug with the above test environment(qemu-kvm-0.12.1.2-2.150.el6.x86_64, NICs is BCM5709 with MSI capability)

And the bug has been verified on rhel6.1(2.6.32-122.el6.x86_64) with qemu-kvm-0.12.1.2-2.152.el6.x86_64.

From libvirt point of view, I can't execute reset action for the device by virsh command:
# virsh nodedev-dettach pci_0000_01_00_1
Device pci_0000_01_00_1 dettached

# ls /sys/bus/pci/drivers/bnx2/
0000:02:00.0  0000:02:00.1  bind  module  new_id  remove_id  uevent  unbind

# ls /sys/bus/pci/drivers/pci-stub/
0000:01:00.0  0000:01:00.1  0000:09:00.0  bind  new_id  remove_id  uevent  unbind

# virsh nodedev-reset pci_0000_01_00_1
error: Failed to reset device pci_0000_01_00_1
error: this function is not supported by the connection driver: Unable to reset PCI device 0000:01:00.1: this function is not supported by the connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus reset

And it's also fail to directly hot-plug/cold-plug the device to guest by virt-manager, the same error information will be raise:

Error starting domain: this function is not supported by the connection driver: Unable to reset PCI device 0000:01:00.1: this function is not supported by the connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus reset

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 45, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/engine.py", line 956, in asyncfunc
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 1048, in startup
    self._backend.create()
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 325, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: this function is not supported by the connection driver: Unable to reset PCI device 0000:01:00.1: this function is not supported by the connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus reset

# uname -r
2.6.32-122.el6.x86_64

# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.152.el6.x86_64

# rpm -q libvirt
libvirt-0.8.7-14.el6.x86_64

# rpm -q virt-manager
virt-manager-0.8.6-3.el6.noarch

# rpm -q python-virtinst
python-virtinst-0.500.5-2.el6.noarch
Comment 15 Alex Williamson 2011-03-25 14:21:28 UTC 
(In reply to comment #14)
> I can reproduce the bug with the above test
> environment(qemu-kvm-0.12.1.2-2.150.el6.x86_64, NICs is BCM5709 with MSI
> capability)
> 
> And the bug has been verified on rhel6.1(2.6.32-122.el6.x86_64) with
> qemu-kvm-0.12.1.2-2.152.el6.x86_64.
> 
> From libvirt point of view, I can't execute reset action for the device by
> virsh command:
> # virsh nodedev-dettach pci_0000_01_00_1
> Device pci_0000_01_00_1 dettached
> 
> # ls /sys/bus/pci/drivers/bnx2/
> 0000:02:00.0  0000:02:00.1  bind  module  new_id  remove_id  uevent  unbind
> 
> # ls /sys/bus/pci/drivers/pci-stub/
> 0000:01:00.0  0000:01:00.1  0000:09:00.0  bind  new_id  remove_id  uevent 
> unbind
> 
> # virsh nodedev-reset pci_0000_01_00_1
> error: Failed to reset device pci_0000_01_00_1
> error: this function is not supported by the connection driver: Unable to reset
> PCI device 0000:01:00.1: this function is not supported by the connection
> driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus
> reset

I don't understand what you're trying to do here.  The original bug was shown by issuing a system_reset on the qemu monitor.  What you're trying to do is a completely different test.  If you want to verify using libvirt, you can do so using the qemu-monitor-command option of virsh with the --hmp option for system_reset.  Note that you'll need the libvirt fix for bz689002 if testing via libvirt.
 
> And it's also fail to directly hot-plug/cold-plug the device to guest by
> virt-manager, the same error information will be raise:

I don't understand what you're trying to do here.

> Error starting domain: this function is not supported by the connection driver:
> Unable to reset PCI device 0000:01:00.1: this function is not supported by the
> connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not
> doing bus reset
> 
> Traceback (most recent call last):
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 45, in
> cb_wrapper
>     callback(asyncjob, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/engine.py", line 956, in asyncfunc
>     vm.startup()
>   File "/usr/share/virt-manager/virtManager/domain.py", line 1048, in startup
>     self._backend.create()
>   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 325, in create
>     if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
> libvirtError: this function is not supported by the connection driver: Unable
> to reset PCI device 0000:01:00.1: this function is not supported by the
> connection driver: Active 0000:01:00.0 devices on bus with 0000:01:00.1, not
> doing bus reset
> 
> # uname -r
> 2.6.32-122.el6.x86_64
> 
> # rpm -q qemu-kvm
> qemu-kvm-0.12.1.2-2.152.el6.x86_64
> 
> # rpm -q libvirt
> libvirt-0.8.7-14.el6.x86_64
> 
> # rpm -q virt-manager
> virt-manager-0.8.6-3.el6.noarch
> 
> # rpm -q python-virtinst
> python-virtinst-0.500.5-2.el6.noarch
Comment 17 Miya Chen 2011-03-29 05:30:16 UTC 
move it to verified based on commnet#14
Comment 18 Alex Williamson 2011-05-05 15:43:42 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:
Assigned devices may continue DMA operations across a reset of the VM.
Consequence:
Guest memory buffers are reset across reboot, resulting in continuing DMA operations possibly overwriting guest memory.
Fix:
Assigned devices are reset on VM reset.
Result:
Assigned device is quiesced and does not continue to DMA across a VM reset.
Comment 19 errata-xmlrpc 2011-05-19 11:24:01 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html
Comment 20 errata-xmlrpc 2011-05-19 13:02:06 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html
Note You need to log in before you can comment on or make changes to this bug.