Bug 1237250 - aarch64: libguestfs should now prefer virtio-pci
Summary: aarch64: libguestfs should now prefer virtio-pci
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.2
Hardware: aarch64
OS: Unspecified
high
medium
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 1359086
Blocks: 1212027 1221569 1288337 1301891
TreeView+ depends on / blocked
 
Reported: 2015-06-30 15:52 UTC by Andrew Jones
Modified: 2017-08-01 22:08 UTC (History)
8 users (show)

Fixed In Version: libguestfs-1.36.1-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 22:08:55 UTC


Attachments (Terms of Use)
log.txt (89.39 KB, text/plain)
2015-06-30 16:30 UTC, Richard W.M. Jones
no flags Details
0001-arm-Prefer-virtio-pci-instead-of-virtio-mmio-RHBZ-12.patch (4.19 KB, patch)
2015-06-30 18:06 UTC, Richard W.M. Jones
no flags Details | Diff
the output with "LIBGUESTFS_BACKEND=direct libguestfs-test-tool" (211.93 KB, text/plain)
2017-03-31 12:45 UTC, YongkuiGuo
no flags Details
the output with "libguestfs-test-tool" (217.20 KB, text/plain)
2017-03-31 12:47 UTC, YongkuiGuo
no flags Details
../libvirt/qemu/log/guestfs-xxx.log for libvirt case (2.95 KB, text/plain)
2017-03-31 12:55 UTC, YongkuiGuo
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2023 normal SHIPPED_LIVE libguestfs bug fix and enhancement update 2017-08-01 19:32:01 UTC

Description Andrew Jones 2015-06-30 15:52:49 UTC
RHELSA 7.2 qemu-kvm-rhev will very soon support ACPI generation (patches getting merged now). The RHELSA 7.2 guest kernel has support for the PCIe host bridge (which is also in the virt-rhelsa7.2 machine type) when it is boot with ACPI. We can now prefer using virtio-pci instead of virtio-mmio and should.

Comment 1 Richard W.M. Jones 2015-06-30 16:29:53 UTC
I couldn't get this to work upstream, unfortunately.  Any tips?
I'm using latest qemu from git and kernel 4.1.0-0.rc7.git0.1.fc23.aarch64

The qemu command line is:

/home/rjones/d/qemu/aarch64-softmmu/qemu-system-aarch64 \
    -global virtio-blk-pci.scsi=off \
    -nodefconfig \
    -enable-fips \
    -nodefaults \
    -display none \
    -M virt \
    -cpu host \
    -machine accel=kvm:tcg \
    -m 768 \
    -no-reboot \
    -rtc driftfix=slew \
    -global kvm-pit.lost_tick_policy=discard \
    -drive if=pflash,format=raw,file=/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw,readonly \
    -drive if=pflash,format=raw,file=/home/rjones/d/libguestfs/tmp/libguestfsUATPeC/vars.fd.2 \
    -kernel /home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel \
    -initrd /home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/initrd \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/home/rjones/d/libguestfs/tmp/libguestfsUATPeC/scratch.1,cache=unsafe,format=raw,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -chardev socket,path=/home/rjones/d/libguestfs/tmp/libguestfsUATPeC/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -append 'panic=1 console=ttyAMA0 earlyprintk=pl011,0x9000000 ignore_loglevel efi-rtc=noprobe udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color'

UEFI loads and runs, apparently normally.

The kernel loads and starts running, apparently normally.

However the kernel cannot see the SCSI drives.

There are virtio_scsi.ko and virtio_pci.ko modules and they are loaded by
the guest kernel.

I will attach the full log shortly.

Comment 2 Richard W.M. Jones 2015-06-30 16:30:41 UTC
Created attachment 1044732 [details]
log.txt

Comment 3 Richard W.M. Jones 2015-06-30 16:35:44 UTC
No difference after removing acpi=off from the command line.

Comment 4 Richard W.M. Jones 2015-06-30 18:06:30 UTC
Created attachment 1044770 [details]
0001-arm-Prefer-virtio-pci-instead-of-virtio-mmio-RHBZ-12.patch

The patch would be something like this one, but as mentioned
above I cannot get it to work on Fedora.

Comment 5 Richard W.M. Jones 2015-06-30 18:16:54 UTC
There's an interesting thing: The patch in comment 3 *works* on
armv7hl.  Now it's not an exact comparison because I'm using quite
a different (actually, somewhat older) kernel on 32 bit ARM:
4.0.6-300.fc22.armv7hl+lpae

So I don't exactly know what to make of that.

Comment 6 Andrew Jones 2015-07-01 10:11:51 UTC
(In reply to Richard W.M. Jones from comment #1)
> I couldn't get this to work upstream, unfortunately.  Any tips?
> I'm using latest qemu from git and kernel 4.1.0-0.rc7.git0.1.fc23.aarch64
> 

I'm not sure the Fedora kernel has everything it needs for ACPI+PCI. I was able to boot a RHELSA guest with a nearly identical commandline as is in comment 1. I only changed the kernel and initrd to 4.1.0-0.rc7.10.el7.aarch64 (the initrd also needed virtio-pci added), changed the root disk image path to my guest image, and removed acpi=off from the command line. I also used latest qemu from git.

I'm not sure what to make of comment 5 either (I haven't looked at the status of the arm kernel and pci support), but I guess it's possible that the PCIe host bridge support actually works with that guest kernel and devicetree.

Comment 8 Richard W.M. Jones 2016-06-27 15:37:18 UTC
In RHEL 7.3 virtio-mmio will be preferred.  In 7.4 we can look
at defaulting to virtio-pci.

Comment 9 Jon Masters 2016-08-30 07:17:22 UTC
Thanks - looking forward to 7.4 :)

Comment 10 Richard W.M. Jones 2017-02-16 14:20:33 UTC
Upstream commit is 4a9af91e3632047579c3a7e011c1484c97bd959a, so this
is expected to be fixed in the rebase.

Comment 12 YongkuiGuo 2017-03-31 11:27:57 UTC
packages info:
libguestfs-1.36.3-1.el7.aarch64
qemu-kvm-rhev-2.6.0-28.el7_3.9.aarch64
kernel-4.5.0-15.el7.aarch64

I didn't understand this bug clearly. I just executed the command of libguestfs-test-tool on a rhel7.3 machine with aarch64 cpu. I don't kown how to verify whether virtio-pci has been used instead of virtio-mmio in new libguestfs version. Any other tips that you can give me about the reproduce steps? thanks.

Comment 13 Richard W.M. Jones 2017-03-31 11:37:18 UTC
(In reply to YongkuiGuo from comment #12)
> packages info:
> libguestfs-1.36.3-1.el7.aarch64
> qemu-kvm-rhev-2.6.0-28.el7_3.9.aarch64
> kernel-4.5.0-15.el7.aarch64
> 
> I didn't understand this bug clearly. I just executed the command of
> libguestfs-test-tool on a rhel7.3 machine with aarch64 cpu. I don't kown how
> to verify whether virtio-pci has been used instead of virtio-mmio in new
> libguestfs version. Any other tips that you can give me about the reproduce
> steps? thanks.

libguestfs-test-tool is the correct approach.  Can you attach the
full log from that to this bug and I will show you what to look
for.

Comment 14 Richard W.M. Jones 2017-03-31 11:38:48 UTC
(In reply to Richard W.M. Jones from comment #13)
> (In reply to YongkuiGuo from comment #12)
> > packages info:
> > libguestfs-1.36.3-1.el7.aarch64
> > qemu-kvm-rhev-2.6.0-28.el7_3.9.aarch64
> > kernel-4.5.0-15.el7.aarch64
> > 
> > I didn't understand this bug clearly. I just executed the command of
> > libguestfs-test-tool on a rhel7.3 machine with aarch64 cpu. I don't kown how
> > to verify whether virtio-pci has been used instead of virtio-mmio in new
> > libguestfs version. Any other tips that you can give me about the reproduce
> > steps? thanks.
> 
> libguestfs-test-tool is the correct approach.  Can you attach the
> full log from that to this bug and I will show you what to look
> for.

And I should say that you actually need to run libguestfs-test-tool
both ways:

LIBGUESTFS_BACKEND=direct libguestfs-test-tool > test.direct 2>&1
libguestfs-test-tool > test.libvirt 2>&1

and for the libvirt case you'll also need to find the right file
~/.cache/libvirt/qemu/log/guestfs-XXXX.log and attach that.

Comment 15 YongkuiGuo 2017-03-31 12:45:56 UTC
Created attachment 1267845 [details]
the output with "LIBGUESTFS_BACKEND=direct libguestfs-test-tool"

Comment 16 YongkuiGuo 2017-03-31 12:47:07 UTC
Created attachment 1267846 [details]
the output with "libguestfs-test-tool"

Comment 17 Richard W.M. Jones 2017-03-31 12:54:39 UTC
(In reply to YongkuiGuo from comment #15)
> Created attachment 1267845 [details]
> the output with "LIBGUESTFS_BACKEND=direct libguestfs-test-tool"

In this one (running qemu directly), notice the use of virtio-scsi-pci:

    -device virtio-scsi-pci,id=scsi \
    -drive file=/tmp/libguestfs9QSxvc/scratch.1,cache=unsafe,format=raw,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
    -device scsi-hd,drive=appliance \

Previously this would have been using 'virtio-scsi-device' which
is qemu's name for virtio-mmio.

(In reply to YongkuiGuo from comment #16)
> Created attachment 1267846 [details]
> the output with "libguestfs-test-tool"

You'll need to find /var/log/libvirt/qemu/guestfs-mn4kkegshfkaihjr.log
to see the differences, since there is no visible difference in
the libvirt XML that libguestfs generates, but there should be
a difference in the qemu command line which libvirt generates.

Comment 18 YongkuiGuo 2017-03-31 12:55:49 UTC
Created attachment 1267852 [details]
../libvirt/qemu/log/guestfs-xxx.log for libvirt case

Comment 19 Richard W.M. Jones 2017-03-31 13:08:45 UTC
(In reply to YongkuiGuo from comment #18)
> Created attachment 1267852 [details]
> ../libvirt/qemu/log/guestfs-xxx.log for libvirt case

In this one it's the presence again of:

-device virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0

which shows that this is using virtio-pci.  If it said
"virtio-scsi-device", it would still be using virtio-mmio.

This bug can be marked as VERIFIED based on the evidence provided.

Comment 20 YongkuiGuo 2017-03-31 13:26:22 UTC
In the file of guestfs-mn4kkegshfkaihjr.log, I find out  virtio-scsi-pci, virtio-serial-pci and virtio-rng-pci, which were modified in 0001-arm-Prefer-virtio-pci-instead-of-virtio-mmio-RHBZ-12.patch. Now I understand. Thanks again.

Comment 21 YongkuiGuo 2017-03-31 13:40:54 UTC
Verified with packages info:
libguestfs-1.36.3-1.el7.aarch64


Step:
1. Prepare a rhel7.3 machine with aarch64 cpu
2. LIBGUESTFS_BACKEND=direct libguestfs-test-tool
---------------------------------------
... ...
device virtio-scsi-pci,id=scsi \
    -drive file=/tmp/libguestfs9QSxvc/scratch.1,cache=unsafe,format=raw,id=hd0,if=none \
----------------------------------------
3. libguestfs-test-tool
4. cat ./libvirt/qemu/log/guestfs-xxx.log
----------------------------------------
... ...
-device virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.3,addr=0x0
----------------------------------------

It is using virtio-pci clearly,rather than virtio-mmio. 
So verified.

Comment 22 errata-xmlrpc 2017-08-01 22:08:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2023


Note You need to log in before you can comment on or make changes to this bug.