Bug 1038375

Summary: qemu will core dumped when disable PCI option ROM with rombar=0 for vfio-pci (Broadcom BCM57810 card)
Product: Red Hat Enterprise Linux 7 Reporter: Sibiao Luo <sluo>
Component: qemu-kvmAssignee: Bandan Das <bdas>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 7.0CC: acathrow, alex.williamson, chayang, hhuang, juzhang, michen, qzhang, sluo, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-16 04:40:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Sibiao Luo 2013-12-05 02:19:52 UTC
Description of problem:
boot guest with a PF vfio-pci assgined and specified rombar=0, qemu will quit and core qumped.
BTW, also tried if specified rombar=1 that QEMU work well.

Version-Release number of selected component (if applicable):
host info:
3.10.0-57.el7.x86_64
qemu-kvm-1.5.3-20.el7.x86_64
seabios-1.7.2.2-4.el7.x86_64
guest info:
3.10.0-57.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Load vfio-pci, vfio, vfio_iommu_type1 modules.
# lsmod | grep vfio
vfio_pci               36474  0 
vfio_iommu_type1       17636  0 
vfio                   20777  2 vfio_iommu_type1,vfio_pci

2.Check what other devices are in the same group as PF, unbind all of them and bind to vfio-pci.
# lspci | grep -i BCM57810
08:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
08:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
# readlink /sys/bus/pci/devices/0000:08:00.0/iommu_group
../../../../kernel/iommu_groups/14
# readlink /sys/bus/pci/devices/0000:08:00.1/iommu_group
../../../../kernel/iommu_groups/14
# lspci -n -s 0000:08:00.0 | awk '{ print $3 }'
14e4:168e
# echo "14e4 168e" > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:00.0 > /sys/bus/pci/devices/0000\:08\:00.0/driver/unbind 
# echo 0000:08:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
# lspci -n -s 0000:08:00.1 | awk '{ print $3 }'
14e4:168e
# echo "14e4 168e" >> /sys/bus/pci/drivers/vfio-pci/new_id
# echo 0000:08:00.1 >> /sys/bus/pci/devices/0000\:08\:00.1/driver/unbind 
# echo 0000:08:00.1 >> /sys/bus/pci/drivers/vfio-pci/bind

3.Do vfio-pci with PF to guest and specified rombar=0.
e.g:...-device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0,rombar=0

Actual results:
after step 3, qemu quit and core dumped.
QEMU 1.5.3 monitor - type 'help' for more information
(qemu) c
(qemu) Segmentation fault (core dumped)

Core was generated by `/usr/libexec/qemu-kvm -M pc -S -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f8a8bfcbb7e in qemu_set_irq (irq=0x7f, level=0) at hw/core/irq.c:38
38	    irq->handler(irq->opaque, irq->n, level);
(gdb) bt
#0  0x00007f8a8bfcbb7e in qemu_set_irq (irq=0x7f, level=0) at hw/core/irq.c:38
#1  0x00007f8a8c10497b in vfio_disable_intx_kvm (vdev=vdev@entry=0x7f8a8e89f180)
    at /usr/src/debug/qemu-1.5.3/hw/misc/vfio.c:434
#2  0x00007f8a8c105b4b in vfio_update_irq (pdev=0x7f8a8e89f180) at /usr/src/debug/qemu-1.5.3/hw/misc/vfio.c:477
#3  0x00007f8a8c01643b in pci_bus_fire_intx_routing_notifier (bus=0x7f8a8e808ba0) at hw/pci/pci.c:1236
#4  0x00007f8a8c010115 in piix3_write_config (dev=0x7f8a8e80ae60, address=<optimized out>, val=<optimized out>, 
    len=<optimized out>) at hw/pci-host/piix.c:408
#5  0x00007f8a8c11d252 in access_with_adjusted_size (addr=addr@entry=2, value=value@entry=0x7f8a7e58bb58, size=1, 
    access_size_min=<optimized out>, access_size_max=<optimized out>, 
    access=access@entry=0x7f8a8c11d810 <memory_region_write_accessor>, opaque=opaque@entry=0x7f8a8e808a28)
    at /usr/src/debug/qemu-1.5.3/memory.c:364
#6  0x00007f8a8c11e727 in memory_region_iorange_write (iorange=<optimized out>, offset=2, width=1, data=139)
    at /usr/src/debug/qemu-1.5.3/memory.c:439
#7  0x00007f8a8c11bdf2 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=3326)
    at /usr/src/debug/qemu-1.5.3/kvm-all.c:1497
#8  kvm_cpu_exec (env=env@entry=0x7f8a8e7edea0) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1649
#9  0x00007f8a8c0c72c5 in qemu_kvm_cpu_thread_fn (arg=0x7f8a8e7edea0) at /usr/src/debug/qemu-1.5.3/cpus.c:793
#10 0x00007f8a89ed7de3 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f8a86fce26d in clone () from /lib64/libc.so.6
(gdb) bt full
#0  0x00007f8a8bfcbb7e in qemu_set_irq (irq=0x7f, level=0) at hw/core/irq.c:38
No locals.
#1  0x00007f8a8c10497b in vfio_disable_intx_kvm (vdev=vdev@entry=0x7f8a8e89f180)
    at /usr/src/debug/qemu-1.5.3/hw/misc/vfio.c:434
        irqfd = {fd = 25, gsi = 11, flags = 1, resamplefd = 0, pad = '\000' <repeats 15 times>}
#2  0x00007f8a8c105b4b in vfio_update_irq (pdev=0x7f8a8e89f180) at /usr/src/debug/qemu-1.5.3/hw/misc/vfio.c:477
        vdev = 0x7f8a8e89f180
        route = {mode = PCI_INTX_DISABLED, irq = -1}
#3  0x00007f8a8c01643b in pci_bus_fire_intx_routing_notifier (bus=0x7f8a8e808ba0) at hw/pci/pci.c:1236
        dev = <optimized out>
        sec = <optimized out>
#4  0x00007f8a8c010115 in piix3_write_config (dev=0x7f8a8e80ae60, address=<optimized out>, val=<optimized out>, 
    len=<optimized out>) at hw/pci-host/piix.c:408
        piix3 = 0x7f8a8e80ae60
        address = <optimized out>
        val = <optimized out>
        len = <optimized out>
        dev = 0x7f8a8e80ae60
#5  0x00007f8a8c11d252 in access_with_adjusted_size (addr=addr@entry=2, value=value@entry=0x7f8a7e58bb58, size=1, 
    access_size_min=<optimized out>, access_size_max=<optimized out>, 
    access=access@entry=0x7f8a8c11d810 <memory_region_write_accessor>, opaque=opaque@entry=0x7f8a8e808a28)
    at /usr/src/debug/qemu-1.5.3/memory.c:364
        access_mask = 255
        access_size = 1
        i = <optimized out>
#6  0x00007f8a8c11e727 in memory_region_iorange_write (iorange=<optimized out>, offset=2, width=1, data=139)
    at /usr/src/debug/qemu-1.5.3/memory.c:439
        mrio = <optimized out>
        mr = 0x7f8a8e808a28
        __PRETTY_FUNCTION__ = "memory_region_iorange_write"
#7  0x00007f8a8c11bdf2 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=3326)
    at /usr/src/debug/qemu-1.5.3/kvm-all.c:1497
        i = 0
        ptr = 0x7f8a8be99000 <Address 0x7f8a8be99000 out of bounds>
#8  kvm_cpu_exec (env=env@entry=0x7f8a8e7edea0) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1649
        cpu = 0x7f8a8e7edd90
        __func__ = "kvm_cpu_exec"
        run = 0x7f8a8be98000
        ret = <optimized out>
        run_ret = <optimized out>
#9  0x00007f8a8c0c72c5 in qemu_kvm_cpu_thread_fn (arg=0x7f8a8e7edea0) at /usr/src/debug/qemu-1.5.3/cpus.c:793
        cpu = 0x7f8a8e7edd90
        __func__ = "qemu_kvm_cpu_thread_fn"
        r = <optimized out>
#10 0x00007f8a89ed7de3 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00007f8a86fce26d in clone () from /lib64/libc.so.6
No symbol table info available.
(gdb)

Expected results:
it should no any qemu core dumped.

Additional info:
# /usr/libexec/qemu-kvm -M pc -S -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-7.0-20131127.1_Server_x86_64.qcow2,if=none,id=drive-disk,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,vectors=0,bus=pci.0,addr=0x4,scsi=off,drive=drive-disk,id=system-disk,bootindex=1 -net none -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice disable-ticketing,port=5931 -monitor stdio -device vfio-pci,host=08:00.0,id=sluo_guest_nic_pf0,rombar=0

Comment 1 Bandan Das 2013-12-11 23:30:52 UTC
(In reply to Sibiao Luo from comment #0)
> Description of problem:
> boot guest with a PF vfio-pci assgined and specified rombar=0, qemu will
> quit and core qumped.
> BTW, also tried if specified rombar=1 that QEMU work well.
> 
I am not able to reproduce this with qemu-kvm-1.5.3-21. I would actually expect it to not work with rombar=1 due to reasons mentioned in bug 1005629. What's the firmware version on the Broadcom card ?

Comment 2 Sibiao Luo 2013-12-12 02:15:13 UTC
(In reply to Bandan Das from comment #1)
> (In reply to Sibiao Luo from comment #0)
> > Description of problem:
> > boot guest with a PF vfio-pci assgined and specified rombar=0, qemu will
> > quit and core qumped.
> > BTW, also tried if specified rombar=1 that QEMU work well.
> > 
> I am not able to reproduce this with qemu-kvm-1.5.3-21. I would actually
> expect it to not work with rombar=1 due to reasons mentioned in bug 1005629.
> What's the firmware version on the Broadcom card ?
Boot guest with a PF vfio-pci assgined and specified rombar=1 that QEMU work well and guest can boot up successfully. Just specified rombar=0 will core dumped.

My firmware-version is bc 7.8.22.
# ethtool -i p2p2
driver: bnx2x
version: 1.78.17-0
firmware-version: bc 7.8.22
bus-info: 0000:08:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# modinfo be2net
filename:       /lib/modules/3.10.0-57.el7.x86_64/kernel/drivers/net/ethernet/emulex/benet/be2net.ko
license:        GPL
author:         Emulex Corporation
description:    Emulex OneConnect 10Gbps NIC Driver 4.9.224.0r
version:        4.9.224.0r
srcversion:     0D545EF0B52BCC867E4168E
alias:          pci:v000010DFd00000728sv*sd*bc*sc*i*
alias:          pci:v000010DFd00000720sv*sd*bc*sc*i*
alias:          pci:v000010DFd0000E228sv*sd*bc*sc*i*
alias:          pci:v000010DFd0000E220sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000710sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000700sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000221sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000211sv*sd*bc*sc*i*
depends:        
intree:         Y
vermagic:       3.10.0-57.el7.x86_64 SMP mod_unload modversions 
signer:         Red Hat Enterprise Linux kernel signing key
sig_key:        15:56:8C:5B:A2:B3:4B:15:2F:B4:AE:B1:B5:1B:F7:AA:25:05:11:78
sig_hashalgo:   sha256
parm:           num_vfs:Number of PCI VFs to initialize (uint)
parm:           rx_frag_size:Size of a fragment that holds rcvd data. (ushort)