Bug 1403785

Summary: libvirt picks qemu-system-arm to run ppc64le VM on ppc64le host
Product: Red Hat Enterprise Linux 7 Reporter: Daniel Berrangé <berrange>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: agedosier, berrange, clalancette, dzheng, extras-qa, itamar, laine, libvirt-maint, rbalakri, rjones, veillard, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: ppc64le   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1403745 Environment:
Last Closed: 2016-12-13 15:39:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Berrangé 2016-12-12 11:25:23 UTC 
Cloned from Fedora, since rhel7.4 rebase will have pulled in this bug

+++ This bug was initially created as a clone of Bug #1403745 +++

Description of problem:

Running libguestfs-test-tool on ppc64le host gives the error:

Original error from libvirt: internal error: process exited while connecting to monitor: 2016-12-12T10:06:56.792870Z qemu-system-arm: -machine pseries,accel=tcg,usb=off,dump-guest-core=off: unsupported machine type
Use -machine help to list supported machines [code=1 int1=-1]

The problem is that libvirt is using qemu-system-arm to
run the VM, even though we never specified ARM anywhere
and obviously it makes no sense.

Version-Release number of selected component (if applicable):

libvirt-2.5.0-2.fc26.ppc64le
qemu-2:2.8.0-0.2.rc2.fc26.ppc64le

How reproducible:

Unknown.

Steps to Reproduce:
1. Run libguestfs-test-tool on ppc64le host.

Additional info:

build.log & root.log attached.

--- Additional comment from Richard W.M. Jones on 2016-12-12 10:20 GMT ---



--- Additional comment from Richard W.M. Jones on 2016-12-12 10:22 GMT ---



--- Additional comment from Daniel Berrange on 2016-12-12 11:16:22 GMT ---

Capabilities from build.log:

<capabilities>
  <host>
    <uuid>98d4d4ab-0bc3-4f8d-b7f6-18394a59acd0</uuid>
    <cpu>
      <arch>ppc64le</arch>
      <model>POWER8</model>
      <vendor>IBM</vendor>
      <topology sockets='1' cores='4' threads='1'/>
      <pages unit='KiB' size='64'/>
    </cpu>
    <power_management/>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>10452928</memory>
          <pages unit='KiB' size='64'>163327</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='4'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0'/>
            <cpu id='1' socket_id='0' core_id='8' siblings='1'/>
            <cpu id='2' socket_id='0' core_id='16' siblings='2'/>
            <cpu id='3' socket_id='0' core_id='24' siblings='3'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>none</model>
      <doi>0</doi>
    </secmodel>
  </host>
  <guest>
    <os_type>hvm</os_type>
    <arch name='armv7l'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-arm</emulator>
      <machine maxCpus='1'>integratorcp</machine>
      <machine maxCpus='2'>nuri</machine>
      <machine maxCpus='1'>verdex</machine>
      <machine maxCpus='1'>ast2500-evb</machine>
      <machine maxCpus='2'>smdkc210</machine>
      <machine maxCpus='1'>collie</machine>
      <machine maxCpus='1'>imx25-pdk</machine>
      <machine maxCpus='1'>spitz</machine>
      <machine maxCpus='4'>realview-pbx-a9</machine>
      <machine maxCpus='1'>realview-eb</machine>
      <machine maxCpus='1'>versatilepb</machine>
      <machine maxCpus='1'>realview-pb-a8</machine>
      <machine maxCpus='1'>musicpal</machine>
      <machine maxCpus='1'>z2</machine>
      <machine maxCpus='1'>akita</machine>
      <machine maxCpus='255'>virt-2.7</machine>
      <machine maxCpus='1'>kzm</machine>
      <machine maxCpus='255'>virt-2.8</machine>
      <machine canonical='virt-2.8' maxCpus='255'>virt</machine>
      <machine maxCpus='4'>realview-eb-mpcore</machine>
      <machine maxCpus='1'>sx1</machine>
      <machine maxCpus='1'>sx1-v1</machine>
      <machine maxCpus='255'>virt-2.6</machine>
      <machine maxCpus='1'>cubieboard</machine>
      <machine maxCpus='4'>highbank</machine>
      <machine maxCpus='4'>raspi2</machine>
      <machine maxCpus='1'>netduino2</machine>
      <machine maxCpus='1'>terrier</machine>
      <machine maxCpus='1'>n810</machine>
      <machine maxCpus='1'>mainstone</machine>
      <machine maxCpus='1'>palmetto-bmc</machine>
      <machine maxCpus='4'>sabrelite</machine>
      <machine maxCpus='4'>midway</machine>
      <machine maxCpus='1'>cheetah</machine>
      <machine maxCpus='1'>tosa</machine>
      <machine maxCpus='1'>borzoi</machine>
      <machine maxCpus='1'>versatileab</machine>
      <machine maxCpus='1'>lm3s6965evb</machine>
      <machine maxCpus='1'>n800</machine>
      <machine maxCpus='1'>connex</machine>
      <machine maxCpus='1'>xilinx-zynq-a9</machine>
      <machine maxCpus='4'>vexpress-a9</machine>
      <machine maxCpus='4'>vexpress-a15</machine>
      <machine maxCpus='1'>canon-a1100</machine>
      <machine maxCpus='1'>lm3s811evb</machine>
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
    </features>
  </guest>
  <guest>
    <os_type>hvm</os_type>
    <arch name='aarch64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-aarch64</emulator>
      <machine maxCpus='1'>integratorcp</machine>
      <machine maxCpus='2'>nuri</machine>
      <machine maxCpus='1'>verdex</machine>
      <machine maxCpus='1'>ast2500-evb</machine>
      <machine maxCpus='2'>smdkc210</machine>
      <machine maxCpus='1'>collie</machine>
      <machine maxCpus='1'>imx25-pdk</machine>
      <machine maxCpus='1'>spitz</machine>
      <machine maxCpus='4'>realview-pbx-a9</machine>
      <machine maxCpus='1'>realview-eb</machine>
      <machine maxCpus='1'>versatilepb</machine>
      <machine maxCpus='1'>realview-pb-a8</machine>
      <machine maxCpus='1'>musicpal</machine>
      <machine maxCpus='1'>z2</machine>
      <machine maxCpus='1'>akita</machine>
      <machine maxCpus='255'>virt-2.7</machine>
      <machine maxCpus='1'>kzm</machine>
      <machine maxCpus='255'>virt-2.8</machine>
      <machine canonical='virt-2.8' maxCpus='255'>virt</machine>
      <machine maxCpus='4'>realview-eb-mpcore</machine>
      <machine maxCpus='1'>sx1</machine>
      <machine maxCpus='1'>sx1-v1</machine>
      <machine maxCpus='255'>virt-2.6</machine>
      <machine maxCpus='1'>cubieboard</machine>
      <machine maxCpus='4'>highbank</machine>
      <machine maxCpus='4'>raspi2</machine>
      <machine maxCpus='1'>netduino2</machine>
      <machine maxCpus='1'>terrier</machine>
      <machine maxCpus='1'>n810</machine>
      <machine maxCpus='1'>mainstone</machine>
      <machine maxCpus='1'>palmetto-bmc</machine>
      <machine maxCpus='4'>sabrelite</machine>
      <machine maxCpus='4'>midway</machine>
      <machine maxCpus='1'>cheetah</machine>
      <machine maxCpus='1'>tosa</machine>
      <machine maxCpus='1'>borzoi</machine>
      <machine maxCpus='1'>versatileab</machine>
      <machine maxCpus='1'>lm3s6965evb</machine>
      <machine maxCpus='1'>n800</machine>
      <machine maxCpus='1'>connex</machine>
      <machine maxCpus='1'>xilinx-zynq-a9</machine>
      <machine maxCpus='1'>xlnx-ep108</machine>
      <machine maxCpus='4'>vexpress-a9</machine>
      <machine maxCpus='4'>vexpress-a15</machine>
      <machine maxCpus='1'>xlnx-zcu102</machine>
      <machine maxCpus='1'>canon-a1100</machine>
      <machine maxCpus='1'>lm3s811evb</machine>
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
    </features>
  </guest>
  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-i386</emulator>
      <machine maxCpus='255'>pc-i440fx-2.8</machine>
      <machine canonical='pc-i440fx-2.8' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-i440fx-2.4</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-q35-2.7</machine>
      <machine maxCpus='255'>pc-q35-2.6</machine>
      <machine maxCpus='255'>pc-i440fx-1.7</machine>
      <machine maxCpus='255'>pc-i440fx-1.6</machine>
      <machine maxCpus='255'>pc-i440fx-2.7</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-i440fx-2.3</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='255'>pc-i440fx-2.2</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-q35-2.5</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-i440fx-1.5</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-i440fx-2.6</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='255'>pc-i440fx-2.5</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-i440fx-2.1</machine>
      <machine maxCpus='288'>pc-q35-2.8</machine>
      <machine canonical='pc-q35-2.8' maxCpus='288'>q35</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-i440fx-2.0</machine>
      <machine maxCpus='255'>pc-q35-2.4</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>
  <guest>
    <os_type>hvm</os_type>
    <arch name='ppc'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-ppc</emulator>
      <machine maxCpus='1'>g3beige</machine>
      <machine maxCpus='1'>ref405ep</machine>
      <machine maxCpus='1'>virtex-ml507</machine>
      <machine maxCpus='32'>ppce500</machine>
      <machine maxCpus='15'>mpc8544ds</machine>
      <machine maxCpus='1'>bamboo</machine>
      <machine maxCpus='1'>prep</machine>
      <machine maxCpus='1'>mac99</machine>
      <machine maxCpus='1'>taihu</machine>
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
    </features>
  </guest>
  <guest>
    <os_type>hvm</os_type>
    <arch name='ppc64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-ppc64</emulator>
      <machine maxCpus='255'>pseries-2.8</machine>
      <machine canonical='pseries-2.8' maxCpus='255'>pseries</machine>
      <machine maxCpus='1'>ref405ep</machine>
      <machine maxCpus='1'>virtex-ml507</machine>
      <machine maxCpus='2048'>powernv</machine>
      <machine maxCpus='32'>ppce500</machine>
      <machine maxCpus='15'>mpc8544ds</machine>
      <machine maxCpus='1'>bamboo</machine>
      <machine maxCpus='1'>g3beige</machine>
      <machine maxCpus='1'>prep</machine>
      <machine maxCpus='1'>mac99</machine>
      <machine maxCpus='255'>pseries-2.6</machine>
      <machine maxCpus='255'>pseries-2.7</machine>
      <machine maxCpus='255'>pseries-2.4</machine>
      <machine maxCpus='255'>pseries-2.5</machine>
      <machine maxCpus='255'>pseries-2.2</machine>
      <machine maxCpus='1'>taihu</machine>
      <machine maxCpus='255'>pseries-2.3</machine>
      <machine maxCpus='255'>pseries-2.1</machine>
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
    </features>
  </guest>
  <guest>
    <os_type>hvm</os_type>
    <arch name='ppcemb'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-ppcemb</emulator>
      <machine maxCpus='1'>ref405ep</machine>
      <machine maxCpus='1'>virtex-ml507</machine>
      <machine maxCpus='1'>bamboo</machine>
      <machine maxCpus='1'>taihu</machine>
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
    </features>
  </guest>
  <guest>
    <os_type>hvm</os_type>
    <arch name='s390x'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-s390x</emulator>
      <machine maxCpus='248'>s390-ccw-virtio-2.8</machine>
      <machine canonical='s390-ccw-virtio-2.8' maxCpus='248'>s390-ccw-virtio</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.7</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.6</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.5</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.4</machine>
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
    </features>
  </guest>
  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <machine maxCpus='255'>pc-i440fx-2.8</machine>
      <machine canonical='pc-i440fx-2.8' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-i440fx-2.4</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-q35-2.7</machine>
      <machine maxCpus='255'>pc-q35-2.6</machine>
      <machine maxCpus='255'>pc-i440fx-1.7</machine>
      <machine maxCpus='255'>pc-i440fx-1.6</machine>
      <machine maxCpus='255'>pc-i440fx-2.7</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-i440fx-2.3</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='255'>pc-i440fx-2.2</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-q35-2.5</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-i440fx-1.5</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-i440fx-2.6</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='255'>pc-i440fx-2.5</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-i440fx-2.1</machine>
      <machine maxCpus='288'>pc-q35-2.8</machine>
      <machine canonical='pc-q35-2.8' maxCpus='288'>q35</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-i440fx-2.0</machine>
      <machine maxCpus='255'>pc-q35-2.4</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <domain type='qemu'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>
</capabilities>


It correctly detected 'ppc64le' as the host arch.

There is no <guest> reported with an arch of ppc64le though - only ppc and ppc64 (be).

There domain XML does not request any architecture. So libvirt tries to find an emulator for the host architecture (ppc64le) and fails, so tries to find any emulator and thus finds arm7.

So the key question is why we've not found ppc64le emulators

--- Additional comment from Daniel Berrange on 2016-12-12 11:22:32 GMT ---

When probing caps, libvirt iterates over every arch and tries to find qemu-system-$ARCH.  Now ppc64le is a special case, so we have code to look for qemu-system-ppc64 for the ppc64le arch. This is in virQEMUCapsFindTarget().

Unfortunately looking at the code, it seems we have a regression introduced there by

commit f2e71550d62f08dad93697ec56a6bd1c960b5f03
Author: Jiri Denemark <jdenemar>
Date:   Tue Jun 14 17:25:58 2016 +0200

    qemu: Introduce virQEMUCapsGuestIsNative
    
THis commit replaced

 
-    /* armv7l guests on aarch64 hosts can use the aarch64 target
-     * i686 guests on x86_64 hosts can use the x86_64 target */
-    if ((guestarch == VIR_ARCH_ARMV7L && hostarch == VIR_ARCH_AARCH64) ||
-        (guestarch == VIR_ARCH_I686 && hostarch == VIR_ARCH_X86_64)) {
-        return hostarch;
-    }
+    if (virQEMUCapsGuestIsNative(hostarch, guestarch))
+        guestarch = hostarch;


which causes virQEMUCapsFindTarget to return VIR_ARCH_PPC64LE again, instead of VIR_ARCH_PPC64. So we try to probe a qemu binary that doesn't exist.
Comment 2 Jiri Denemark 2016-12-13 15:39:26 UTC 
This bug does not affect RHEL since the QEMU binary is always called qemu-kvm no matter what the host architecture is. Anyway, I will post a fix for upstream soon.