Bug 1058321

Summary: qemu-kvm-rhev: Qemu: Q35: hw: pci: use after free triggered via guest [rhel-7.2]
Product: Red Hat Enterprise Linux 7 Reporter: Markus Armbruster <armbru>
Component: qemu-kvm-rhevAssignee: Marcel Apfelbaum <marcel>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.2CC: juzhang, knoel, michen, pmatouse, ppandit, qzhang, rbalakri, shuang, virt-maint, yama
Target Milestone: rcKeywords: Security, SecurityTracking, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.3.0 Doc Type: Release Note
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-04 16:14:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 983344    
Bug Blocks: 1112271, 1227278    

Description Markus Armbruster 2014-01-27 14:38:37 UTC
Description of problem:
When I unplug a virtio-blk-pci device sitting in a PCIe slot of q35's
xio3130-downstream bridge, the guest kernel warns.

Version-Release number of selected component (if applicable):
At least qemu-kvm-1.5.3-43.el7, older versions crash (bug 983344)
Guest kernel-3.10.0-71.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Boot a RHEL-7 guest with an additional, unused virtio-blk-pci
device connected to PCIe.  This requires q35.  Relevant part of
command line
-M q35 -device ioh3420,bus=pcie.0,id=root.2,slot=3 -device x3130-upstream,bus=root.2,id=upstream2 -device xio3130-downstream,bus=upstream2,id=downstream2,chassis=3 -drive if=none,id=foo,file=tmp.qcow2 -device virtio-blk-pci,id=bar,bus=downstream2,drive=foo
2. When the guest is up, unplug with "device_del bar"

Actual results:
Unplug succeeds, but guest kernel warns (details below).

Expected results:
Unplug succeeds, guest kernel doesn't warn.

Additional info:
Also observed with current upstream QEMU.

Older guest kernels crash, details at
https://bugzilla.redhat.com/show_bug.cgi?id=983344#c14

Guest dmesg:
[   37.674257] ------------[ cut here ]------------
[   37.674296] WARNING: at drivers/virtio/virtio.c:158 virtio_dev_remove+0x74/0x80 [virtio]()
[   37.674301] Modules linked in: ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg kvm_amd kvm pcspkr serio_raw i2c_i801 lpc_ich mfd_core mperf shpchp xfs libcrc32c sr_mod cdrom cirrus virtio_net virtio_blk syscopyarea sysfillrect sysimgblt drm_kms_helper ahci ttm libahci drm libata virtio_pci virtio_ring virtio i2c_core dm_mirror dm_region_hash dm_log dm_mod
[   37.674374] CPU: 0 PID: 22 Comm: kworker/0:1 Not tainted 3.10.0-71.el7.x86_64 #1
[   37.674380] Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011
[   37.674393] Workqueue: pciehp-0 pciehp_power_thread
[   37.674397]  0000000000000009 ffff8802443dbb30 ffffffff815bd8c4 ffff8802443dbb68
[   37.674405]  ffffffff81059c61 ffff880241b0c400 ffff880241b0c408 ffffffffa0024000
[   37.674411]  ffff880244312098 0000000000000000 ffff8802443dbb78 ffffffff81059d3a
[   37.674418] Call Trace:
[   37.674431]  [<ffffffff815bd8c4>] dump_stack+0x19/0x1b
[   37.674443]  [<ffffffff81059c61>] warn_slowpath_common+0x61/0x80
[   37.674454]  [<ffffffff81059d3a>] warn_slowpath_null+0x1a/0x20
[   37.674465]  [<ffffffffa00220e4>] virtio_dev_remove+0x74/0x80 [virtio]
[   37.674476]  [<ffffffff8139535f>] __device_release_driver+0x7f/0xf0
[   37.674484]  [<ffffffff813953f3>] device_release_driver+0x23/0x30
[   37.674491]  [<ffffffff81394b88>] bus_remove_device+0x108/0x180
[   37.674498]  [<ffffffff81391485>] device_del+0x135/0x1d0
[   37.674505]  [<ffffffff8139153e>] device_unregister+0x1e/0x60
[   37.674516]  [<ffffffffa00224b6>] unregister_virtio_device+0x16/0x30 [virtio]
[   37.674527]  [<ffffffffa004c56b>] virtio_pci_remove+0x2b/0x70 [virtio_pci]
[   37.674537]  [<ffffffff812d252b>] pci_device_remove+0x3b/0xb0
[   37.674546]  [<ffffffff8139535f>] __device_release_driver+0x7f/0xf0
[   37.674553]  [<ffffffff813953f3>] device_release_driver+0x23/0x30
[   37.674560]  [<ffffffff81394b88>] bus_remove_device+0x108/0x180
[   37.674567]  [<ffffffff81391485>] device_del+0x135/0x1d0
[   37.674576]  [<ffffffff812cc064>] pci_stop_bus_device+0x94/0xa0
[   37.674583]  [<ffffffff812cc152>] pci_stop_and_remove_bus_device+0x12/0x20
[   37.674591]  [<ffffffff812e4bd8>] pciehp_unconfigure_device+0xa8/0x1b0
[   37.674599]  [<ffffffff812e4538>] pciehp_disable_slot+0x68/0x200
[   37.674607]  [<ffffffff812e4753>] pciehp_power_thread+0x83/0xf0
[   37.674616]  [<ffffffff8107862b>] process_one_work+0x17b/0x460
[   37.674623]  [<ffffffff810793db>] worker_thread+0x11b/0x400
[   37.674631]  [<ffffffff810792c0>] ? rescuer_thread+0x3e0/0x3e0
[   37.674638]  [<ffffffff8107fb90>] kthread+0xc0/0xd0
[   37.674646]  [<ffffffff8107fad0>] ? kthread_create_on_node+0x110/0x110
[   37.674653]  [<ffffffff815cd66c>] ret_from_fork+0x7c/0xb0
[   37.674659]  [<ffffffff8107fad0>] ? kthread_create_on_node+0x110/0x110
[   37.674665] ---[ end trace f5de3b0770382ce3 ]---

Comment 6 Yanhui Ma 2015-06-19 03:37:10 UTC
Reproduce:
Version of components:
qemu-kvm-1.5.3-43.el7
Guest kernel-3.10.0-71.el7.x86_64

steps:
1. Boot a RHEL-7 guest with an additional, unused virtio-blk-pci
device connected to PCIe.  

/usr/libexec/qemu-kvm -M q35 -m 4G -cpu Opteron_G3 -smp 4,sockets=4,cores=1,threads=1,maxcpus=4 -spice port=5931,disable-ticketing -monitor stdio -qmp tcp:0:6666,server,nowait -device ioh3420,bus=pcie.0,id=root.10,slot=1 -device x3130-upstream,bus=root.10,id=upstream10 -device xio3130-downstream,bus=upstream10,id=downstream10,chassis=0 -drive if=none,id=foo,file=/home/test.qcow2 -device virtio-blk-pci,id=bar,bus=downstream10,drive=foo -drive file=/home/rhel70-64-virtio-scsi.qcow2,if=none,id=drive-data-disk1,cache=writethrough,format=qcow2,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi1,addr=0x13 -device scsi-hd,drive=drive-data-disk1,bus=scsi1.0,id=data-disk1,bootindex=0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,id=virtio-net-pci0,mac=00:24:21:7f:b6:11,bus=pcie.0,addr=0x9

2. When the guest is up, unplug with "device_del bar"

Actual results:
Unplug succeeds, but guest kernel warns (details below).
[  203.171453] pciehp 0000:02:00.0:pcie24: Card not present on Slot(0)
[  203.174783] ------------[ cut here ]------------
[  203.175309] WARNING: at drivers/virtio/virtio.c:158 virtio_dev_remove+0x74/0x80 [virtio]()
[  203.176139] Modules linked in: fuse ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg i2c_i801 lpc_ich serio_raw shpchp kvm mfd_core mperf pcspkr uinput nfsd auth_rpcgss nfs_acl lockd xfs libcrc32c sd_mod crct10dif_generic crc_t10dif crct10dif_common cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper virtio_scsi virtio_blk ttm drm ahci libahci e1000 virtio_pci libata virtio_ring virtio i2c_core sunrpc dm_mirror dm_region_hash dm_log dm_mod
[  203.184857] CPU: 0 PID: 43 Comm: kworker/0:1 Not tainted 3.10.0-71.el7.x86_64 #1
[  203.185602] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[  203.186186] Workqueue: pciehp-0 pciehp_power_thread
[  203.186682]  0000000000000009 ffff88017966fb30 ffffffff815bd8c4 ffff88017966fb68
[  203.188574]  ffffffff81059c61 ffff8801723f1800 ffff8801723f1808 ffffffffa00ad000
[  203.189404]  ffff8801795b0098 0000000000000000 ffff88017966fb78 ffffffff81059d3a
[  203.190238] Call Trace:
[  203.190505]  [<ffffffff815bd8c4>] dump_stack+0x19/0x1b
[  203.191040]  [<ffffffff81059c61>] warn_slowpath_common+0x61/0x80
[  203.191664]  [<ffffffff81059d3a>] warn_slowpath_null+0x1a/0x20
[  203.192272]  [<ffffffffa00ab0e4>] virtio_dev_remove+0x74/0x80 [virtio]
[  203.192933]  [<ffffffff8139535f>] __device_release_driver+0x7f/0xf0
[  203.193574]  [<ffffffff813953f3>] device_release_driver+0x23/0x30
[  203.194195]  [<ffffffff81394b88>] bus_remove_device+0x108/0x180
[  203.194784]  [<ffffffff81391485>] device_del+0x135/0x1d0
[  203.195324]  [<ffffffff8139153e>] device_unregister+0x1e/0x60
[  203.195899]  [<ffffffffa00ab4b6>] unregister_virtio_device+0x16/0x30 [virtio]
[  203.196622]  [<ffffffffa008d56b>] virtio_pci_remove+0x2b/0x70 [virtio_pci]
[  203.197318]  [<ffffffff812d252b>] pci_device_remove+0x3b/0xb0
[  203.197891]  [<ffffffff8139535f>] __device_release_driver+0x7f/0xf0
[  203.198522]  [<ffffffff813953f3>] device_release_driver+0x23/0x30
[  203.199140]  [<ffffffff81394b88>] bus_remove_device+0x108/0x180
[  203.199723]  [<ffffffff81391485>] device_del+0x135/0x1d0
[  203.200264]  [<ffffffff812cc064>] pci_stop_bus_device+0x94/0xa0
[  203.200855]  [<ffffffff812cc152>] pci_stop_and_remove_bus_device+0x12/0x20
[  203.201559]  [<ffffffff812e4bd8>] pciehp_unconfigure_device+0xa8/0x1b0
[  203.202242]  [<ffffffff812e4538>] pciehp_disable_slot+0x68/0x200
[  203.202851]  [<ffffffff812e4753>] pciehp_power_thread+0x83/0xf0
[  203.203460]  [<ffffffff8107862b>] process_one_work+0x17b/0x460
[  203.204067]  [<ffffffff810793db>] worker_thread+0x11b/0x400
[  203.204626]  [<ffffffff810792c0>] ? rescuer_thread+0x3e0/0x3e0
[  203.205247]  [<ffffffff8107fb90>] kthread+0xc0/0xd0
[  203.205740]  [<ffffffff8107fad0>] ? kthread_create_on_node+0x110/0x110
[  203.206415]  [<ffffffff815cd66c>] ret_from_fork+0x7c/0xb0
[  203.206966]  [<ffffffff8107fad0>] ? kthread_create_on_node+0x110/0x110
[  203.207638] ---[ end trace 34b69ce9c6a3d31a ]---

As above show, this bz has been reproduce.


===================
Verify:
Version of components:
qemu-kvm-rhev-2.3.0-1.el7.x86_64
Guest kernel-3.10.0-71.el7.x86_64

Steps as above show, after step 2, unplug succeeds, no guest kernel warns
So this bz has been verified.

Comment 11 errata-xmlrpc 2015-12-04 16:14:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2546.html