Bug 1373154

(In reply to Pei Zhang from comment #0)
> Description of problem:
> Boot guest with ivshmem-plain, guest will fail start up with  'No bootable
> device'. 
> 
> Version-Release number of selected component (if applicable):
> Host:
> 3.10.0-503.el7.x86_64
> qemu-kvm-rhev-2.6.0-22.el7.x86_64
> 
> Guest:
> 3.10.0-503.el7.x86_64
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. Boot guest with ivshmem-plain
> # /usr/libexec/qemu-kvm -name rhel7.3 \
> -cpu IvyBridge,check -m 4G \
> -smp 4,sockets=2,cores=2,threads=1 \
> -netdev tap,id=hostnet0 \
> -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=22:54:00:5c:77:61,
> rx_queue_size=256 \
> -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16 \
> -spice
> port=5902,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-
> migration=on \
> -monitor stdio \
> -serial unix:/tmp/monitor,server,nowait \
> -qmp tcp:0:5551,server,nowait \
> -drive
> file=/home/pezhang/rhel7.3.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,
> werror=stop,rerror=stop \
> -device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0 \
> -usbdevice tablet \
> -object
> memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on \
> -device ivshmem-plain,memdev=mem \
> 
> 
> Actual results:
> Guest fails start.
> 
> Expected results:
> Guest should start.
> 
> Additional info:
> 1. This is a regression bug.
> qemu-kvm-rhev-2.6.0-18.el7.x86_64   work
> qemu-kvm-rhev-2.6.0-19.el7.x86_64   doesn't work
> qemu-kvm-rhev-2.6.0-20.el7.x86_64   doesn't work
> qemu-kvm-rhev-2.6.0-22.el7.x86_64   doesn't work
> 
> 2. The exact commit which causes this bug.
> # git bisect bad
> eca64aee65e32ed710af1185cefda038f599c86c is the first bad commit
> commit eca64aee65e32ed710af1185cefda038f599c86c
> Author: Marcel Apfelbaum <marcel>
> Date:   Wed Aug 3 08:36:43 2016 +0200
> 
>     hw/virtio-pci: fix virtio behaviour


The mentioned BZ only enables the virtio 1.0 .
In order to find the root cause we need to bisect with :
  -device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,disable-modern=off

I'll update when I'll know more.
Thanks,
Marcel

If there is no room to map all MMIO BARs into the 32-bit PCI window,
SeaBIOS will re-allocate all 64-bit MMIO BARs into over-4G space.

Virtio 1.0 block devices (virtio-blk/virtio-scsi) use a 64-bit BAR
unusable by SeaBIOS if mapped over 4G space, preventing the system
to boot.

The simplest solution is to follow the xhci model and simply
skip migrating the virtio 1.0 modern bar into over-4G space.

Patch posted upstream for SeaBIOS: 
[PATCH] pci: don't map virtio 1.0 storage devices above 4G
(No link yet)

If the solution is accepted I'll re-assign the BZ to SeaBIOS component.

Thanks,
Marcel

(In reply to Marcel Apfelbaum from comment #3)
[...]
> 
> The mentioned BZ only enables the virtio 1.0 .
> In order to find the root cause we need to bisect with :
>   -device
> virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,disable-modern=off
> 
> I'll update when I'll know more.
> Thanks,
> Marcel

Note: Sorry for missed this comment before. And put the re-test results here in case it's useful.

Re-test with adding 'disable-modern=off'.

1. As bug[1] is fixed in qemu-kvm-rhev-2.6.0-18.el7, so re-test this version.
[1]Bug 1333318 - ivshmem-plain support in RHEL 7.3

qemu-kvm-rhev-2.6.0-18.el7.x86_64   doesn't work('No bootable device')

2. Also tried the first version of 2.6. (ivshmem-plain has showed up)

qemu-kvm-rhev-2.6.0-1.el7.x86_64    doesn't work('No bootable device')

So seems disable-modern=off is incompatible with ivshmem-plain, even with it's first supported version.


Best Regards,
Pei

It seems there is already work in progress dealing with the same problem.

From an offline conversation with Gerd:

Look at these upstream commits:
  c579d2f85b4835e39490d862a18143cf130c4078
  0e21548b15e25e010c362ea13d170c61a3fcc899

They implement support for accessing virtio bars via pci config space,
which will make virtio devices work in seabios even if mapped over 4G.

Unfortunately they depend on some pci cleanups done on the master
branch.  They are not clean cherry-picks for 1.9-stable, and because of
that the 1.9.x releases do not have them.

I have an older version of those patches laying around in a local
branch, they do apply to 1.9.x, pushed it to my repo:
  https://www.kraxel.org/cgit/seabios/log/?h=virtio-old

Alternatively we could make seabios skip virtio devices when mapping
bars over 4G.  Have a look at pci_region_migrate_64bit_entries(), there
is already an exception for usb (so xhci isn't mapped high).

The upstream commit fixing it: 0e21548b15e25e010c362ea13d170c61a3fcc899).

(In reply to Pei Zhang from comment #5)
> (In reply to Marcel Apfelbaum from comment #3)
> [...]
> > 
> > The mentioned BZ only enables the virtio 1.0 .
> > In order to find the root cause we need to bisect with :
> >   -device
> > virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,disable-modern=off
> > 
> > I'll update when I'll know more.
> > Thanks,
> > Marcel
> 
> Note: Sorry for missed this comment before. And put the re-test results here
> in case it's useful.
> 
> Re-test with adding 'disable-modern=off'.
> 
> 1. As bug[1] is fixed in qemu-kvm-rhev-2.6.0-18.el7, so re-test this version.
> [1]Bug 1333318 - ivshmem-plain support in RHEL 7.3
> 
> qemu-kvm-rhev-2.6.0-18.el7.x86_64   doesn't work('No bootable device')
> 
> 2. Also tried the first version of 2.6. (ivshmem-plain has showed up)
> 
> qemu-kvm-rhev-2.6.0-1.el7.x86_64    doesn't work('No bootable device')
> 
> So seems disable-modern=off is incompatible with ivshmem-plain, even with
> it's first supported version.
> 

Thanks for checking it!

Please check (without the disable-modern=off) using the seabios package
from the brew build:
    https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=11741451 

> 
> Best Regards,
> Pei

(In reply to Marcel Apfelbaum from comment #10)
[...]
> Please check (without the disable-modern=off) using the seabios package
> from the brew build:
>     https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=11741451 
[...] 

Re-test:
Versions:
qemu-kvm-rhev-2.6.0-24.el7.x86_64

Steps:
1. Install above seabios build in host

2. Boot guest without disable-modern=off, guest can boot successfully.
[...]
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0 \

So with this seabios build, no error occurs.

Best Regards,
Pei

Fix included in seabios-1.9.1-5.el7

==Verification==

Versions:
Host:
3.10.0-506.rt56.411.el7.x86_64
qemu-kvm-rhev-2.6.0-25.el7.x86_64
seabios-1.9.1-5.el7.x86_64

Guest:
3.10.0-506.rt56.411.el7.x86_64


Steps:
1. Boot guest with ivshmem-plain, guest works well.
Same as Description.

So this bug has been fixed well. Thank you.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2207.html