Description of problem ---------------------- Invoking `libguestfs-test-tool` with 'direct' backend on Fedora-21 running on a Cubietruck fails with: kvm_init_vcpu failed: Invalid argument Version ------- cubietruck]$ uname -r; rpm -q qemu-system-arm libguestfs 3.17.4-302.fc21.armv7hl+lpae qemu-system-arm-2.1.2-7.fc21.armv7hl libguestfs-1.28.4-2.fc21.armv7hl How reproducible: Consistently. Steps to Reproduce ------------------ (1) Install Fedora-21 ARM on a Cubietruck. I obtained the image from here: http://be.mirror.eurid.eu/fedora/linux/releases/test/21-Beta/Images/armhfp/Fedora-Minimal-armhfp-21_Beta-4-sda.raw.xz (2) Install libguestfs tools: $ yum install libguestfs-tools-c -y (3) Enable debugging and libguestfs 'direct' backend: $ export LIBGUESTFS_DEBUG=1 $ export LIBGUESTFS_TRACE=1 $ export LIBGUESTFS_BACKEND=direct (3) Invoke the 'libguestfs-test-tool': $ libguestfs-test-tool Actual results -------------- ======================================================================= $ libguestfs-test-tool libguestfs: trace: set_verbose true libguestfs: trace: set_verbose = 0 libguestfs: trace: set_backend "direct" libguestfs: trace: set_backend = 0 libguestfs: trace: set_verbose true libguestfs: trace: set_verbose = 0 ************************************************************ * IMPORTANT NOTICE * * When reporting bugs, include the COMPLETE, UNEDITED * output below in your bug report. * ************************************************************ LIBGUESTFS_DEBUG=1 LIBGUESTFS_BACKEND=direct LIBGUESTFS_TRACE=1 PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin SELinux: Permissive libguestfs: trace: add_drive_scratch 104857600 libguestfs: trace: get_tmpdir libguestfs: trace: get_tmpdir = "/tmp" libguestfs: trace: disk_create "/tmp/libguestfsDKHlqd/scratch.1" "raw" 104857600 libguestfs: trace: disk_create = 0 libguestfs: trace: add_drive "/tmp/libguestfsDKHlqd/scratch.1" "format:raw" "cachemode:unsafe" libguestfs: trace: add_drive = 0 libguestfs: trace: add_drive_scratch = 0 libguestfs: trace: get_append libguestfs: trace: get_append = "NULL" libguestfs: trace: get_autosync libguestfs: trace: get_autosync = 1 libguestfs: trace: get_backend libguestfs: trace: get_backend = "direct" libguestfs: trace: get_backend_settings libguestfs: trace: get_backend_settings = [] libguestfs: trace: get_cachedir libguestfs: trace: get_cachedir = "/var/tmp" libguestfs: trace: get_direct libguestfs: trace: get_direct = 0 libguestfs: trace: get_hv libguestfs: trace: get_hv = "/usr/bin/qemu-system-arm" libguestfs: trace: get_memsize libguestfs: trace: get_memsize = 500 libguestfs: trace: get_network libguestfs: trace: get_network = 0 libguestfs: trace: get_path libguestfs: trace: get_path = "/usr/lib/guestfs" libguestfs: trace: get_pgroup libguestfs: trace: get_pgroup = 0 libguestfs: trace: get_program libguestfs: trace: get_program = "libguestfs-test-tool" libguestfs: trace: get_recovery_proc libguestfs: trace: get_recovery_proc = 1 libguestfs: trace: get_selinux libguestfs: trace: get_selinux = 0 libguestfs: trace: get_smp libguestfs: trace: get_smp = 1 libguestfs: trace: get_tmpdir libguestfs: trace: get_tmpdir = "/tmp" libguestfs: trace: get_trace libguestfs: trace: get_trace = 1 libguestfs: trace: get_verbose libguestfs: trace: get_verbose = 1 guestfs_get_append: (null) guestfs_get_autosync: 1 guestfs_get_backend: direct guestfs_get_backend_settings: [] guestfs_get_cachedir: /var/tmp guestfs_get_direct: 0 guestfs_get_hv: /usr/bin/qemu-system-arm guestfs_get_memsize: 500 guestfs_get_network: 0 guestfs_get_path: /usr/lib/guestfs guestfs_get_pgroup: 0 guestfs_get_program: libguestfs-test-tool guestfs_get_recovery_proc: 1 guestfs_get_selinux: 0 guestfs_get_smp: 1 guestfs_get_tmpdir: /tmp guestfs_get_trace: 1 guestfs_get_verbose: 1 host_cpu: armv7hl Launching appliance, timeout set to 600 seconds. libguestfs: trace: launch libguestfs: trace: version libguestfs: trace: version = <struct guestfs_version *> libguestfs: trace: get_backend libguestfs: trace: get_backend = "direct" libguestfs: launch: program=libguestfs-test-tool libguestfs: launch: version=1.28.4fedora=21,release=2.fc21,libvirt libguestfs: launch: backend registered: unix libguestfs: launch: backend registered: uml libguestfs: launch: backend registered: libvirt libguestfs: launch: backend registered: direct libguestfs: launch: backend=direct libguestfs: launch: tmpdir=/tmp/libguestfsDKHlqd libguestfs: launch: umask=0022 libguestfs: launch: euid=0 libguestfs: trace: get_backend_setting "force_tcg" libguestfs: trace: get_backend_setting = NULL (error) libguestfs: trace: get_cachedir libguestfs: trace: get_cachedir = "/var/tmp" libguestfs: [00005ms] begin building supermin appliance libguestfs: [00005ms] run supermin libguestfs: command: run: /usr/bin/supermin libguestfs: command: run: \ --build libguestfs: command: run: \ --verbose libguestfs: command: run: \ --if-newer libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock libguestfs: command: run: \ --copy-kernel libguestfs: command: run: \ -f ext2 libguestfs: command: run: \ --host-cpu armv7hl libguestfs: command: run: \ --dtb vexpress*a15-tc1.dtb libguestfs: command: run: \ /usr/lib/guestfs/supermin.d libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d supermin: version: 5.1.10 supermin: rpm: detected RPM version 4.12 supermin: package handler: fedora/rpm supermin: acquiring lock on /var/tmp/.guestfs-0/lock supermin: build: /usr/lib/guestfs/supermin.d supermin: build: visiting /usr/lib/guestfs/supermin.d/base.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/guestfs/supermin.d/excludefiles type uncompressed excludefiles supermin: build: visiting /usr/lib/guestfs/supermin.d/hostfiles type uncompressed hostfiles supermin: build: visiting /usr/lib/guestfs/supermin.d/init.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib/guestfs/supermin.d/packages type uncompressed packages supermin: build: visiting /usr/lib/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar) supermin: build: 190 packages, including dependencies supermin: build: 27373 files supermin: build: 13117 files, after removing unreadable files supermin: build: 7198 files, after matching excludefiles supermin: build: 7203 files, after adding hostfiles supermin: build: 7227 files, after munging supermin: kernel: picked kernel vmlinuz-3.17.4-302.fc21.armv7hl supermin: kernel: picked dtb /boot/dtb-3.17.4-302.fc21.armv7hl/vexpress-v2p-ca15-tc1.dtb supermin: kernel: picked modules path /lib/modules/3.17.4-302.fc21.armv7hl supermin: kernel: kernel_version 3.17.4-302.fc21.armv7hl supermin: kernel: modules /lib/modules/3.17.4-302.fc21.armv7hl supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.9t7x3qry/root' warning: Unable to get device geometry for /var/tmp/.guestfs-0/appliance.d.9t7x3qry/root supermin: ext2: populating from base image supermin: ext2: copying files from host filesystem supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.9t7x3qry/initrd' supermin: ext2: wrote 23 modules to minimal initrd supermin: renaming /var/tmp/.guestfs-0/appliance.d.9t7x3qry to /var/tmp/.guestfs-0/appliance.d libguestfs: [189597ms] finished building supermin appliance libguestfs: [189597ms] begin testing qemu features libguestfs: command: run: /usr/bin/qemu-system-arm libguestfs: command: run: \ -display none libguestfs: command: run: \ -help libguestfs: command: run: /usr/bin/qemu-system-arm libguestfs: command: run: \ -display none libguestfs: command: run: \ -version libguestfs: qemu version 2.1 libguestfs: command: run: /usr/bin/qemu-system-arm libguestfs: command: run: \ -display none libguestfs: command: run: \ -M vexpress-a15 libguestfs: command: run: \ -machine accel=kvm:tcg libguestfs: command: run: \ -device ? libguestfs: [192330ms] finished testing qemu features libguestfs: trace: get_backend_setting "gdb" libguestfs: trace: get_backend_setting = NULL (error) [192340ms] /usr/bin/qemu-system-arm \ -global virtio-blk-device.scsi=off \ -nodefconfig \ -enable-fips \ -nodefaults \ -display none \ -M vexpress-a15 \ -machine accel=kvm:tcg \ -m 500 \ -no-reboot \ -rtc driftfix=slew \ -global kvm-pit.lost_tick_policy=discard \ -kernel /var/tmp/.guestfs-0/appliance.d/kernel \ -dtb /var/tmp/.guestfs-0/appliance.d/dtb \ -initrd /var/tmp/.guestfs-0/appliance.d/initrd \ -device virtio-scsi-device,id=scsi \ -drive file=/tmp/libguestfsDKHlqd/scratch.1,cache=unsafe,format=raw,id=hd0,if=none \ -device scsi-hd,drive=hd0 \ -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \ -device scsi-hd,drive=appliance \ -device virtio-serial-device \ -serial stdio \ -chardev socket,path=/tmp/libguestfsDKHlqd/guestfsd.sock,id=channel0 \ -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \ -append 'panic=1 mem=500M console=ttyAMA0 udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=vt102' kvm_init_vcpu failed: Invalid argument libguestfs: error: appliance closed the connection unexpectedly, see earlier error messages libguestfs: child_cleanup: 0xb8afa768: child process died libguestfs: sending SIGTERM to process 8430 libguestfs: error: /usr/bin/qemu-system-arm exited with error status 1, see debug messages above libguestfs: error: guestfs_launch failed, see earlier error messages libguestfs: trace: launch = -1 (error) libguestfs-test-tool: failed to launch appliance libguestfs: trace: close libguestfs: closing guestfs handle 0xb8afa768 (state 0) libguestfs: command: run: rm libguestfs: command: run: \ -rf /tmp/libguestfsDKHlqd ======================================================================= Expected results ---------------- `libguestfs-test-tool` should invoke successfully Additional info --------------- Before running the `libguestfs-test-tool`, I flushed the buffers, to ensure I had enough memory: $ free && sync && echo 3 > /proc/sys/vm/drop_caches && free total used free shared buffers cached Mem: 2064920 1559580 505340 232 57248 1361652 -/+ buffers/cache: 140680 1924240 Swap: 0 0 0 total used free shared buffers cached Mem: 2064920 129520 1935400 232 928 39180 -/+ buffers/cache: 89412 1975508 Swap: 0 0 0
From the inter-webs, seems like this is a reasonably popular error for ARM and KVM. I stumbled on a thread (Tue Dec 10 09:39:09 UTC 2013) on fedora-arm list: https://lists.fedoraproject.org/pipermail/arm/2013-December/007155.html uboot with hyp mode support now available for the cubietruck Quoting Richard Jones from the above email: "I can confirm this works at least as far as running libguestfs appliances using KVM. I had to use the attached hack to make qemu create Cortex-A7 guests, otherwise you get the following error": kvm_init_vcpu failed: Invalid argument ------------------------------------------------- diff --git a/target-arm/kvm.c b/target-arm/kvm.c index 6e5cd36..48b2773 100644 --- a/target-arm/kvm.c +++ b/target-arm/kvm.c @@ -86,7 +86,7 @@ int kvm_arch_init_vcpu(CPUState *cs) struct kvm_reg_list *rlp; ARMCPU *cpu = ARM_CPU(cs); - init.target = KVM_ARM_TARGET_CORTEX_A15; + init.target = KVM_ARM_TARGET_CORTEX_A7; /* XXX HACK */ memset(init.features, 0, sizeof(init.features)); ret = kvm_vcpu_ioctl(cs, KVM_ARM_VCPU_INIT, &init); if (ret) {
For some reason I thought this had been fixed upstream, but now that I've finally got my CT working again, I see that I am still carrying that patch in my custom qemu. diff --git a/target-arm/cpu.c b/target-arm/cpu.c index 5ce7350..04d69d1 100644 --- a/target-arm/cpu.c +++ b/target-arm/cpu.c @@ -858,7 +858,7 @@ static void cortex_a15_initfn(Object *obj) set_feature(&cpu->env, ARM_FEATURE_DUMMY_C15_REGS); set_feature(&cpu->env, ARM_FEATURE_CBAR_RO); set_feature(&cpu->env, ARM_FEATURE_LPAE); - cpu->kvm_target = QEMU_KVM_ARM_TARGET_CORTEX_A15; + cpu->kvm_target = QEMU_KVM_ARM_TARGET_CORTEX_A7; cpu->midr = 0x412fc0f1; cpu->reset_fpsid = 0x410430f0; cpu->mvfr0 = 0x10110222; So that's the answer really, it's a qemu bug. Actually it looks as if qemu contains some code to try to get the host CPU type, but it doesn't work, or maybe we need to pass a -cpu option ...
This is fixed in libguestfs with this commit (committed in upstream git): https://github.com/libguestfs/libguestfs/commit/6468b9d60172bcef76b36c14b72f9a1913d94954 arm: Use -cpu host on KVM. With the above commit applied, KVM guest (a libguestfs appliance) successfully boots on Cubietruck: . . . [01433ms] /bin/qemu-system-arm \ -global virtio-blk-device.scsi=off \ -nodefconfig \ -enable-fips \ -nodefaults \ -display none \ -M vexpress-a15 \ -cpu host \ -machine accel=kvm:tcg \ -m 500 \ -no-reboot \ -rtc driftfix=slew \ -global kvm-pit.lost_tick_policy=discard \ -kernel /home/kashyapc/src/libguestfs/tmp/.guestfs-1000/appliance.d/kernel \ -dtb /home/kashyapc/src/libguestfs/tmp/.guestfs-1000/appliance.d/dtb \ -initrd /home/kashyapc/src/libguestfs/tmp/.guestfs-1000/appliance.d/initrd \ -device virtio-scsi-device,id=scsi \ -drive file=/home/kashyapc/src/libguestfs/tmp/libguestfsRMfHzO/scratch.1,cache=unsafe,format=raw,id=hd0,if=none \ -device scsi-hd,drive=hd0 \ -drive file=/home/kashyapc/src/libguestfs/tmp/.guestfs-1000/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \ -device scsi-hd,drive=appliance \ -device virtio-serial-device \ -serial stdio \ -chardev socket,path=/home/kashyapc/src/libguestfs/tmp/libguestfsRMfHzO/guestfsd.sock,id=channel0 \ -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \ -append 'panic=1 mem=500M console=ttyAMA0 udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=vt102' . . . Some more context on this upstream QEMU mailing list thread: http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg01093.html
Yup, all good now so let's close this bug.