Bug 1535973

Summary: kernel 3.10.0-693.11.6.el7.x86_64 fails to boot inside nested KVM virtual machines on some Intel hardware
Product: Red Hat Enterprise Linux 7 Reporter: Alfredo Moralejo <amoralej>
Component: kernelAssignee: Paolo Bonzini <pbonzini>
kernel sub component: KVM QA Contact: FuXiangChun <xfu>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: high    
Priority: high CC: bdas, bloch, cblake, chayang, dhoward, juriarte, juzhang, knoel, mtessun, pbonzini, ptoscano, rjones, virt-maint, xfu, ykarel
Version: 7.4Keywords: Regression, TestOnly, ZStream
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-3.10.0-859.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1561089 (view as bug list) Environment:
Last Closed: 2019-12-13 13:39:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1537379    
Bug Blocks: 1561089    
Attachments:
Description Flags
output of libguesfs-test-tool none

Description Alfredo Moralejo 2018-01-18 11:13:06 UTC 
Created attachment 1382857 [details]
output of libguesfs-test-tool

Description of problem:

libguesfs-test-tool (or virt-customize) fails in a virtual machine with kernel 3.10.0-693.11.6.el7.x86_64, following error appears:

[    2.573801] general protection fault: 0000 [#1] SMP 
[    2.574091] Modules linked in:
[    2.574091] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-693.11.6.el7.x86_64 #1
[    2.574091] Hardware name: Red Hat KVM, BIOS 1.10.2-3.el7_4.1 04/01/2014
[    2.574091] task: ffff88001eac8000 ti: ffff88001ead0000 task.ti: ffff88001ead0000
[    2.574091] RIP: 0010:[<ffffffff8120a3f0>]  [<ffffffff8120a3f0>] flush_old_exec+0x3b0/0x930
[    2.574091] RSP: 0000:ffff88001ead3d10  EFLAGS: 00010246
[    2.574091] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000049
[    2.574091] RDX: 0000000000000000 RSI: ffff88001d558000 RDI: ffffffff81a7c100
[    2.574091] RBP: ffff88001ead3d68 R08: 000000000001bfc0 R09: ffffffff8120a1e2
[    2.574091] R10: ffff88001ead3b30 R11: 0000000000000007 R12: ffff88001d558000
[    2.574091] R13: ffffffff81a7c100 R14: ffff88001eac8000 R15: ffff88001d53e400
[    2.574091] FS:  0000000000000000(0000) GS:ffff88001ee00000(0000) knlGS:0000000000000000
[    2.574091] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.574091] CR2: 0000000000000000 CR3: 00000000019fa000 CR4: 00000000003606f0
[    2.574091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    2.574091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    2.574091] Call Trace:
[    2.574091]  [<ffffffff8126199c>] load_elf_binary+0x33c/0xe00
[    2.574091]  [<ffffffff812d6743>] ? ima_get_action+0x23/0x30
[    2.574091]  [<ffffffff812d5d7e>] ? process_measurement+0x8e/0x250
[    2.574091]  [<ffffffff812d6239>] ? ima_bprm_check+0x49/0x50
[    2.574091]  [<ffffffff81261660>] ? load_elf_library+0x220/0x220
[    2.574091]  [<ffffffff81209b9d>] search_binary_handler+0xed/0x300
[    2.574091]  [<ffffffff8120b1d6>] do_execve_common.isra.24+0x5b6/0x6c0
[    2.574091]  [<ffffffff81694d50>] ? rest_init+0x80/0x80
[    2.574091]  [<ffffffff8120b2f8>] do_execve+0x18/0x20
[    2.574091]  [<ffffffff8100202b>] run_init_process+0x2b/0x30
[    2.574091]  [<ffffffff81694d8d>] kernel_init+0x3d/0xf0
[    2.574091]  [<ffffffff816b8798>] ret_from_fork+0x58/0x90
[    2.574091]  [<ffffffff81694d50>] ? rest_init+0x80/0x80
[    2.574091] Code: 00 3e 0f ab 08 48 8b 05 db e5 90 00 48 c1 e8 34 a8 01 74 19 65 8b 05 28 2c e0 7e a8 02 74 0e 31 d2 b8 01 00 00 00 b9 49 00 00 00 <0f> 30 bf 00 00 00 80 49 03 7c 24 58 48 8b 05 0d 7c 7f 00 72 0e 
[    2.574091] RIP  [<ffffffff8120a3f0>] flush_old_exec+0x3b0/0x930
[    2.574091]  RSP <ffff88001ead3d10>
[    2.966949] ---[ end trace 9f4f0bee8dbd524c ]---
[    2.975115] Kernel panic - not syncing: Fatal exception


Version-Release number of selected component (if applicable):

kernel-3.10.0-693.11.6.el7.x86_64
supermin5-5.1.16-4.el7.x86_64
libguestfs-1.36.3-6.el7_4.3.x86_64

How reproducible:

We are hitting this running libguestfs-test-tool in a virtual machine which reports cpu:

model name : Intel Core Processor (Skylake, IBRS)

It seems that other CPU models are not showing this issue.

Steps to Reproduce:

In a virtual machine  reporting cpu model Intel Core Processor (Skylake, IBRS)

1. Update your rhel-7 to latest kernel and packages and reboot.
2. Install libguestfs-tools
3. run libguestfs-test-tool

Actual results:

Command fails with error shown before. Full command output is attached.

Expected results:

Command should succeed

Additional info:

The bug still happens updating packages from rhel-7.5 preview repo.
The but does NOT appear with kernel .10.0-693.11.1.el7.x86_64
Setting following environment variables, command works fine:

export SUPERMIN_KERNEL_VERSION=3.10.0-693.el7.x86_64
export SUPERMIN_KERNEL=/boot/vmlinuz-$SUPERMIN_KERNEL_VERSION
export SUPERMIN_MODULES=/lib/modules/$SUPERMIN_KERNEL_VERSION
Comment 2 Richard W.M. Jones 2018-01-18 12:07:21 UTC 
Additionally the Intel host is using nested KVM.
Comment 3 Richard W.M. Jones 2018-01-18 12:29:27 UTC 
I wasn't able to reproduce this locally, but it's clearly a kernel
bug, probably in the IMA code, so reassigning.
Comment 4 Richard W.M. Jones 2018-01-18 17:27:24 UTC 
Git bisection script for this.  Note you have to examine the output
of the final command to see if it's hitting this particular bug.

-----
set -e
set -x

make olddefconfig
make -j`nproc` all
rm -rf /tmp/lib/modules/*
make modules_install INSTALL_MOD_PATH=/tmp
find /tmp/lib/modules/ -name '*.ko' | xargs strip --strip-debug

rm -rf /var/tmp/.guestfs-`id -u`
export SUPERMIN_KERNEL=arch/x86/boot/bzImage
export SUPERMIN_MODULES=/tmp/lib/modules/`ls -1 /tmp/lib/modules/`
libguestfs-test-tool
Comment 5 Richard W.M. Jones 2018-01-18 19:01:48 UTC 
git bisect result:

a79683ef92366ae4ee17c0ee07011907bce1d25a is the first bad commit
commit a79683ef92366ae4ee17c0ee07011907bce1d25a
Author: Andrea Arcangeli <aarcange>
Date:   Fri Dec 15 20:41:49 2017 +0100

    [x86] spec_ctrl: issue a __spec_ctrl_ibpb if a credential check isn't possible
    
    Message-id: <20171215204149.1194-2-aarcange>
    Patchwork-id: 6239
    O-Subject: [RHEL7.4.z for z-stream PATCH CONFIDENTIAL v6.2 1/1] x86/spec_ctrl: issue a __spec_ctrl_ibpb if a credential check isn't possible
    Bugzilla: 1519798
    Z-Bugzilla: 1519795
    
    If we can't do a credential check on the next task (for example
    because it's an execve) for safety issue a ibpb, just in case the
    execve'd task is suid/sgid root.
    
    Acked-by: Josh Poimboeuf <jpoimboe>
    Acked-by: Waiman Long <longman>
    Signed-off-by: Andrea Arcangeli <aarcange>
    Signed-off-by: Denys Vlasenko <dvlasenk>

:040000 040000 0f7c274ecab1d2d15f9520421a4828777c650665 f561b7a80813648ecb929c39d26462f03f8a5d0b M	arch
Comment 6 Richard W.M. Jones 2018-01-18 20:46:15 UTC 
I reverted this patch on top of kernel-3.10.0-693.17.1.el7
and things change.  This bug goes away, but there's a later
GPF during boot in do_group_exit -> __do_page_fault -> do_exit ->
schedule which kills the VM.  Not sure if this is related or
if I'm exposing a different bug by fixing this one.
Comment 7 Richard W.M. Jones 2018-01-18 20:58:49 UTC 
I'm wondering if this is really a qemu problem.  I notice that what's
being run is

native_wrmsrl(MSR_IA32_PRED_CMD, FEATURE_SET_IBPB);

That's likely the new feature added by recent Intel microcode updates.
However I wonder if nested KVM shows the feature is available but
qemu isn't capable of emulating it because it's too new?
Comment 8 Paolo Bonzini 2018-01-18 22:25:15 UTC 
QEMU is not involved in emulating MSRs. However, indeed nested virtualization does not know about those new MSRs, which are special because they are not trapped by the hypervisor.

This will probably be fixed in a couple weeks upstream.
Comment 9 FuXiangChun 2018-01-24 09:32:33 UTC 
Reproduced bug with 3.10.0-837.el7.x86_64 & qemu-kvm-rhev-2.10.0-18.el7.x86_64 & Skylake host.


steps:

1. Boot L1 guest
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox off  \
-machine pc  \
-nodefaults  \
-vga cirrus  \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=rhel7.5.l0.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1,bootindex=2 \
-device virtio-net-pci,mac=9a:5c:5d:5e:5f:60,bootindex=4,id=idNic1,vectors=4,netdev=idUwnwAo,bus=pci.0,addr=0x4  \
-netdev tap,id=idUwnwAo,vhost=on \
-device e1000,mac=9a:61:62:63:64:65,bootindex=3,id=idNic2,netdev=idAFcMIH,bus=pci.0,addr=0x5  \
-netdev tap,id=idAFcMIH,vhost=on \
-m 8192  \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
-cpu 'host',+kvm_pv_unhalt \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on \
-device usb-tablet,id=usb-tablet1 \
-vnc :0  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=on,strict=off,order=cdn,once=c \
-enable-kvm  \
-device sga \
-monitor stdio \

2.Boot L2 guest

/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox off  \
-machine pc  \
-nodefaults  \
-vga cirrus  \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/rhel7.5.l2.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1,bootindex=2 \
-device virtio-net-pci,mac=9a:5c:5d:5e:5f:62,bootindex=4,id=idNic1,vectors=4,netdev=idUwnwAo,bus=pci.0,addr=0x4  \
-netdev tap,id=idUwnwAo,vhost=on \
-device e1000,mac=9a:61:62:63:64:66,bootindex=3,id=idNic2,netdev=idAFcMIH,bus=pci.0,addr=0x5  \
-netdev tap,id=idAFcMIH,vhost=on \
-m 8192  \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
-cpu 'IvyBridge-IBRS',+kvm_pv_unhalt \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on \
-device usb-tablet,id=usb-tablet1 \
-vnc :1  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=on,strict=off,order=cdn,once=c \
-enable-kvm  \
-device sga \
-monitor stdio \
-serial unix:/tmp/console,server,nowait \

result:

[    1.904426] Modules linked in:
[    1.904426] CPU: 3 PID: 64 Comm: kworker/u8:0 Not tainted 3.10.0-837.el7.x86_64 #1
[    1.904426] Hardware name: Red Hat KVM, BIOS 1.11.0-1.el7 04/01/2014
[    1.904426] task: ffff9ada76ff9fa0 ti: ffff9ada73254000 task.ti: ffff9ada73254000
[    1.904426] RIP: 0010:[<ffffffff8741fb90>]  [<ffffffff8741fb90>] flush_old_exec+0x3b0/0x940
[    1.904426] RSP: 0000:ffff9ada73257d00  EFLAGS: 00010246
[    1.904426] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000049
[    1.904426] RDX: 0000000000000000 RSI: ffff9ada73260000 RDI: ffffffff87e914e0
[    1.904426] RBP: ffff9ada73257d58 R08: 0000000000000046 R09: 0000000000000000
[    1.904426] R10: 000000001804b67f R11: ffffe40548ff4a00 R12: ffff9ada73260000
[    1.904426] R13: ffffffff87e914e0 R14: ffff9ada76ff9fa0 R15: ffff9ad9bc80b000
[    1.904426] FS:  0000000000000000(0000) GS:ffff9ada7fd80000(0000) knlGS:0000000000000000
[    1.904426] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.904426] CR2: 00000000ffffffff CR3: 00000000a5a0e000 CR4: 00000000001406e0
[    1.904426] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.904426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    1.904426] Call Trace:
[    1.904426]  [<ffffffff8747b0dc>] load_elf_binary+0x33c/0xe50
[    1.904426]  [<ffffffff874ef9f9>] ? ima_bprm_check+0x49/0x50
[    1.904426]  [<ffffffff8747ada0>] ? load_elf_library+0x220/0x220
[    1.904426]  [<ffffffff8741f32d>] search_binary_handler+0xed/0x300
[    1.904426]  [<ffffffff874209c0>] do_execve_common.isra.24+0x5b0/0x6b0
[    1.904426]  [<ffffffff87420ad8>] do_execve+0x18/0x20
[    1.904426]  [<ffffffff872ae23c>] ____call_usermodehelper+0xfc/0x130
[    1.904426]  [<ffffffff872ae140>] ? call_usermodehelper+0x60/0x60
[    1.904426]  [<ffffffff87916898>] ret_from_fork+0x58/0x90
[    1.904426]  [<ffffffff872ae140>] ? call_usermodehelper+0x60/0x60
[    1.904426] Code: 00 f0 0f ab 08 48 8b 05 fb 4c b1 00 48 c1 e8 34 a8 01 74 19 65 8b 05 88 d4 be 78 a8 02 74 0e 31 d2 b8 01 00 00 00 b9 49 00 00 00 <0f> 30 bf 00 00 00 80 49 03 7c 24 58 48 8b 05 6d 64 9f 00 72 0e 
[    1.904426] RIP  [<ffffffff8741fb90>] flush_old_exec+0x3b0/0x940
[    1.904426]  RSP <ffff9ada73257d00>
[    2.354220] ---[ end trace 102806a0f7891ef4 ]---
[    2.367786] Kernel panic - not syncing: Fatal exception
[    2.368751] Kernel Offset: 0x6200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Comment 10 Paolo Bonzini 2018-02-23 13:46:03 UTC 
Fixed by the patches in bug 1537379.
Comment 11 Paolo Bonzini 2018-03-16 13:11:22 UTC 
Miya, can you please qa_ack this one?
Comment 12 FuXiangChun 2018-03-19 05:55:33 UTC 
Verified bug with 3.10.0-861.el7.x86_64.

Re-tested it with comment9 and according to https://bugzilla.redhat.com/show_bug.cgi?id=1538033#c7.

Result:
L2 RHEL7.5 guest works well. 

So,set this bug as verified.
Comment 19 Yatin Karel 2018-05-10 14:21:55 UTC 
Tested with centos 7.5 and seeing the issue on a vm on RDO Cloud, but not sure it's the same issue


[centos@ykarel-test-temp ~]$ cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 94
model name	: Intel Core Processor (Skylake, IBRS)
stepping	: 3
microcode	: 0x1
cpu MHz		: 2599.996
cache size	: 16384 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt ibpb ibrs arat spec_ctrl
bogomips	: 5199.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:


[centos@ykarel-test-temp ~]$ uname -a
Linux ykarel-test-temp.rdocloud 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[centos@ykarel-test-temp ~]$ rpm -q kernel
kernel-3.10.0-693.el7.x86_64
kernel-3.10.0-862.2.3.el7.x86_64
[centos@ykarel-test-temp ~]$ rpm -q qemu-kvm
qemu-kvm-1.5.3-156.el7.x86_64



[centos@ykarel-test-temp ~]$ libguestfs-test-tool 
     ************************************************************
     *                    IMPORTANT NOTICE
     *
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
SUPERMIN_KERNEL_VERSION=3.10.0-862.2.3.el7.x86_64
SUPERMIN_MODULES=/lib/modules/3.10.0-862.2.3.el7.x86_64
SUPERMIN_KERNEL=/boot/vmlinuz-3.10.0-862.2.3.el7.x86_64
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/centos/.local/bin:/home/centos/bin
XDG_RUNTIME_DIR=/run/user/1000
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 500
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /run/user/1000
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.36.10rhel=7,release=6.el7.centos,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfsd1jhpX
libguestfs: launch: umask=0002
libguestfs: launch: euid=1000
libguestfs: libvirt version = 3009000 (3.9.0)
libguestfs: guest random name = guestfs-40cfe85rhfkrofny
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///session, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x558f1f50e670
libguestfs: qemu version (reported by libvirt) = 1005003 (1.5.3)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: build appliance
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin5
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-1000/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-1000/appliance.d
supermin: version: 5.1.19
supermin: rpm: detected RPM version 4.11
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-1000/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 190 packages, including dependencies
supermin: build: 31545 files
supermin: build: 7572 files, after matching excludefiles
supermin: build: 7578 files, after adding hostfiles
supermin: build: 7571 files, after removing unreadable files
supermin: build: 7595 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: SUPERMIN_KERNEL=/boot/vmlinuz-3.10.0-862.2.3.el7.x86_64
supermin: kernel: SUPERMIN_KERNEL_VERSION=3.10.0-862.2.3.el7.x86_64
supermin: kernel: SUPERMIN_MODULES=/lib/modules/3.10.0-862.2.3.el7.x86_64
supermin: kernel: picked vmlinuz /boot/vmlinuz-3.10.0-862.2.3.el7.x86_64
supermin: kernel: kernel_version 3.10.0-862.2.3.el7.x86_64
supermin: kernel: modpath /lib/modules/3.10.0-862.2.3.el7.x86_64
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-1000/appliance.d.lxblh261/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: warning: /usr/bin/chfn: Permission denied (ignored)
Some distro files are not public readable, so supermin cannot copy them
into the appliance.  This is a problem with your Linux distro.  Please ask
your distro to stop doing pointless security by obscurity.
You can ignore these warnings.  You *do not* need to use sudo.
supermin: warning: /usr/bin/chsh: Permission denied (ignored)
supermin: warning: /usr/libexec/dbus-1/dbus-daemon-launch-helper: Permission denied (ignored)
supermin: warning: /usr/libexec/utempter/utempter: Permission denied (ignored)
supermin: warning: /usr/sbin/build-locale-archive: Permission denied (ignored)
supermin: warning: /usr/sbin/glibc_post_upgrade.x86_64: Permission denied (ignored)
supermin: warning: /usr/sbin/groupadd: Permission denied (ignored)
supermin: warning: /usr/sbin/groupdel: Permission denied (ignored)
supermin: warning: /usr/sbin/groupmems: Permission denied (ignored)
supermin: warning: /usr/sbin/groupmod: Permission denied (ignored)
supermin: warning: /usr/sbin/unix_update: Permission denied (ignored)
supermin: warning: /usr/sbin/useradd: Permission denied (ignored)
supermin: warning: /usr/sbin/userdel: Permission denied (ignored)
supermin: warning: /usr/sbin/usermod: Permission denied (ignored)
supermin: warning: /var/lib/systemd/random-seed: Permission denied (ignored)
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-1000/appliance.d.lxblh261/initrd'
supermin: ext2: wrote 31 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-1000/appliance.d.lxblh261 to /var/tmp/.guestfs-1000/appliance.d
libguestfs: finished building supermin appliance
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-1000/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfsd1jhpX/overlay2.qcow2
Formatting '/tmp/libguestfsd1jhpX/overlay2.qcow2', fmt=qcow2 size=4294967296 backing_file='/var/tmp/.guestfs-1000/appliance.d/root' backing_fmt='raw' encryption=off cluster_size=65536 lazy_refcounts=off 
libguestfs: create libvirt XML
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-40cfe85rhfkrofny</name>\n  <memory unit="MiB">500</memory>\n  <currentMemory unit="MiB">500</currentMemory>\n  <cpu mode="host-passthrough">\n    <model fallback="allow"/>\n  </cpu>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n    <timer name="hpet" present="no"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-1000/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-1000/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>\n    <bios useserial="yes"/>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfsd1jhpX/scratch1.img"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="raw" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfsd1jhpX/overlay2.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <serial type="unix">\n      <source mode="connect" path="/run/user/1000/libguestfs5RsFkU/console.sock"/>\n      <target port="0"/>\n    </serial>\n    <channel type="unix">\n      <source mode="connect" path="/run/user/1000/libguestfs5RsFkU/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /var/tmp/.guestfs-1000
libguestfs: /var/tmp/.guestfs-1000:
libguestfs: drwxr-xr-x. centos centos unconfined_u:object_r:user_tmp_t:s0 .
libguestfs: drwxrwxrwt. root   root   system_u:object_r:tmp_t:s0       ..
libguestfs: drwxr-xr-x. centos centos unconfined_u:object_r:user_tmp_t:s0 appliance.d
libguestfs: -rw-r--r--. centos centos unconfined_u:object_r:user_tmp_t:s0 lock
libguestfs: 
libguestfs: /var/tmp/.guestfs-1000/appliance.d:
libguestfs: drwxr-xr-x. centos centos unconfined_u:object_r:user_tmp_t:s0 .
libguestfs: drwxr-xr-x. centos centos unconfined_u:object_r:user_tmp_t:s0 ..
libguestfs: -rw-r--r--. centos centos unconfined_u:object_r:user_tmp_t:s0 initrd
libguestfs: -rwxr-xr-x. centos centos unconfined_u:object_r:user_tmp_t:s0 kernel
libguestfs: -rw-r--r--. centos centos unconfined_u:object_r:user_tmp_t:s0 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /run/user/1000/libguestfs5RsFkU
libguestfs: drwx------. centos centos unconfined_u:object_r:user_tmp_t:s0 .
libguestfs: drwx------. centos centos system_u:object_r:user_tmp_t:s0  ..
libguestfs: srwxrwxr-x. centos centos unconfined_u:object_r:user_tmp_t:s0 console.sock
libguestfs: srwxrwxr-x. centos centos unconfined_u:object_r:user_tmp_t:s0 guestfsd.sock
libguestfs: launch libvirt guest
libguestfs: responding to serial console Device Status Report
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 06/09/14
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@) Mon Jun  9 21:33:48 UTC 2014
Term: 80x24
4 0
SeaBIOS (version 1.11.0-2.el7)
Machine UUID 5efd5ea0-c811-43fd-ae56-90c897d04879
Booting from ROM...
\x1b[2J[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-862.2.3.el7.x86_64 (builder.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Wed May 9 18:05:47 UTC 2018
[    0.000000] Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001f3fdfff] usable
[    0.000000] BIOS-e820: [mem 0x000000001f3fe000-0x000000001f3fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] Hypervisor detected: KVM
[    0.000000] e820: last_pfn = 0x1f3fe max_arch_pfn = 0x400000000
[    0.000000] PAT configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- UC  
[    0.000000] found SMP MP-table at [mem 0x000f63b0-0x000f63bf] mapped at [ffffffffff2003b0]
[    0.000000] RAMDISK: [mem 0x1f0bd000-0x1f3effff]
[    0.000000] Early table checksum verification disabled
[    0.000000] ACPI BIOS Error (bug): A valid RSDP was not found (20130517/tbxfroot-243)
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000001f3fdfff]
[    0.000000] NODE_DATA(0) allocated [mem 0x1f096000-0x1f0bcfff]
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 0:1f046001, primary cpu clock
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009efff]
[    0.000000]   node   0: [mem 0x00100000-0x1f3fdfff]
[    0.000000] Initmem setup node 0 [mem 0x00001000-0x1f3fdfff]
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] Intel MultiProcessor Specification v1.4
[    0.000000] MPTABLE: OEM ID: BOCHSCPU
[    0.000000] MPTABLE: Product ID: 0.1         
[    0.000000] MPTABLE: APIC at: 0xFEE00000
[    0.000000] Processor #0 (Bootup-CPU)
[    0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.000000] Processors: 1
[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.000000] e820: [mem 0x1f400000-0xfeffbfff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on KVM
[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff95b29ee00000 s104856 r8192 d30312 u2097152
[    0.000000] KVM setup async PF for cpu 0
[    0.000000] kvm-stealtime: cpu 0, msr 1ee134c0
[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 125879
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color
[    0.000000] Disabling memory control group subsystem
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form
[    0.000000] Memory: 480528k/511992k available (7324k kernel code, 392k absent, 31072k reserved, 6305k data, 1832k init)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] x86/pti: Unmapping kernel while in userspace
[    0.000000] Hierarchical RCU implementation.
[    0.000000] \tRCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.
[    0.000000] NR_IRQS:327936 nr_irqs:256 0
[    0.000000] Console: colour *CGA 80x25
[    0.000000] console [ttyS0] enabled
[    0.000000] tsc: Detected 2599.998 MHz processor
[    0.003000] Calibrating delay loop (skipped) preset value.. 5199.99 BogoMIPS (lpj=2599998)
[    0.003000] pid_max: default: 32768 minimum: 301
[    0.003049] Security Framework initialized
[    0.004000] SELinux:  Disabled at boot.
[    0.004028] Yama: becoming mindful.
[    0.005000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.005000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.005075] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.006030] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.007000] Initializing cgroup subsys memory
[    0.007041] Initializing cgroup subsys devices
[    0.008000] Initializing cgroup subsys freezer
[    0.008029] Initializing cgroup subsys net_cls
[    0.009000] Initializing cgroup subsys blkio
[    0.009000] Initializing cgroup subsys perf_event
[    0.009030] Initializing cgroup subsys hugetlb
[    0.010000] Initializing cgroup subsys pids
[    0.010028] Initializing cgroup subsys net_prio
[    0.011000] mce: CPU supports 10 MCE banks
[    0.011912] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[    0.012000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0
[    0.012027] tlb_flushall_shift: 6
[    0.013000] FEATURE SPEC_CTRL Present
[    0.013027] FEATURE IBPB_SUPPORT Present
[    0.014000] Spectre V2 : Mitigation: IBRS (kernel)
[    0.032000] Freeing SMP alternatives: 24k freed
[    0.048000] ftrace: allocating 28047 entries in 110 pages
[    0.050000] Enabling x2apic
[    0.050000] Enabled x2apic
[    0.050030] Switched APIC routing to physical x2apic.
[    0.053000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.053003] smpboot: CPU0: Intel Core Processor (Skylake, IBRS) (fam: 06, model: 5e, stepping: 03)
[    0.054242] Performance Events: unsupported p6 CPU model 94 no PMU driver, software events only.
[    0.057077] KVM setup paravirtual spinlock
[    0.059183] Brought up 1 CPUs
[    0.060032] smpboot: Max logical packages: 1
[    0.061034] smpboot: Total of 1 processors activated (5199.99 BogoMIPS)
[    0.065242] devtmpfs: initialized
[    0.067311] EVM: security.selinux
[    0.068034] EVM: security.ima
[    0.069034] EVM: security.capability
[    0.070722] atomic64 test passed for x86-64 platform with CX8 and with SSE
[    0.071041] pinctrl core: initialized pinctrl subsystem
[    0.072704] RTC time: 14:05:26, date: 05/10/18
[    0.073174] NET: Registered protocol family 16
[    0.075265] PCI: Using configuration type 1 for base access
[    0.077342] ACPI: Interpreter disabled.
[    0.078093] vgaarb: loaded
[    0.079104] SCSI subsystem initialized
[    0.080074] usbcore: registered new interface driver usbfs
[    0.081047] usbcore: registered new interface driver hub
[    0.082062] usbcore: registered new device driver usb
[    0.083107] EDAC MC: Ver: 3.0.0
[    0.084190] PCI: Probing PCI hardware
[    0.085057] PCI host bridge to bus 0000:00
[    0.086035] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.087040] pci_bus 0000:00: root bus resource [mem 0x00000000-0x3fffffffffff]
[    0.088069] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
[    0.105268] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    0.106033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    0.107033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    0.108033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    0.145531] pci 0000:00:01.0: PIIX/ICH IRQ router [8086:7000]
[    0.147195] NetLabel: Initializing
[    0.148032] NetLabel:  domain hash size = 128
[    0.149032] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.150071] NetLabel:  unlabeled traffic allowed by default
[    0.151093] amd_nb: Cannot enumerate AMD northbridges
[    0.152048] Switched to clocksource kvm-clock
[    0.163278] pnp: PnP ACPI: disabled
[    0.168432] NET: Registered protocol family 2
[    0.173165] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[    0.180300] TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
[    0.187087] TCP: Hash tables configured (established 4096 bind 4096)
[    0.193535] TCP: reno registered
[    0.196881] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.202752] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.209280] NET: Registered protocol family 1
[    0.213764] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    0.219790] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    0.225709] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    0.232520] Unpacking initramfs...
[    0.241523] Freeing initrd memory: 3276k freed
[    0.246633] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.254766] sha1_ssse3: Using AVX2 optimized SHA-1 implementation
[    0.260925] sha256_ssse3: Using AVX2 optimized SHA-256 implementation
[    0.267660] futex hash table entries: 256 (order: 2, 16384 bytes)
[    0.274075] Initialise system trusted keyring
[    0.278577] audit: initializing netlink socket (disabled)
[    0.284056] type=2000 audit(1525961125.504:1): initialized
[    0.310581] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.318354] zpool: loaded
[    0.321156] zbud: loaded
[    0.324205] VFS: Disk quotas dquot_6.5.2
[    0.328278] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.334998] msgmni has been set to 944
[    0.338891] Key type big_key registered
[    0.343383] NET: Registered protocol family 38
[    0.347961] Key type asymmetric registered
[    0.352119] Asymmetric key parser 'x509' registered
[    0.357154] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[    0.364542] io scheduler noop registered
[    0.368834] io scheduler deadline registered (default)
[    0.374087] io scheduler cfq registered
[    0.378077] io scheduler mq-deadline registered
[    0.382696] io scheduler kyber registered
[    0.387228] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.392885] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.399750] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[    0.442141] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    0.448481] Non-volatile memory driver v1.3
[    0.452869] Linux agpgart interface v0.103
[    0.457279] crash memory driver: version 1.1
[    0.461712] rdac: device handler registered
[    0.466238] hp_sw: device handler registered
[    0.470620] emc: device handler registered
[    0.474875] alua: device handler registered
[    0.479181] libphy: Fixed MDIO Bus: probed
[    0.483490] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.490287] ehci-pci: EHCI PCI platform driver
[    0.494842] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.501039] ohci-pci: OHCI PCI platform driver
[    0.505613] uhci_hcd: USB Universal Host Controller Interface driver
[    0.512255] usbcore: registered new interface driver usbserial
[    0.518151] usbcore: registered new interface driver usbserial_generic
[    0.524741] usbserial: USB Serial support registered for generic
[    0.531004] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    0.543580] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.548614] serio: i8042 AUX port at 0x60,0x64 irq 12
[    0.553989] mousedev: PS/2 mouse device common for all mice
[    0.561245] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[    0.572484] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.582837] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input1
[    0.591674] rtc_cmos rtc_cmos: alarms up to one day, 114 bytes nvram
[    0.598318] cpuidle: using governor menu
[    0.603851] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2
[    0.612350] hidraw: raw HID events driver (C) Jiri Kosina
[    0.618195] usbcore: registered new interface driver usbhid
[    0.623861] usbhid: USB HID core driver
[    0.628270] drop_monitor: Initializing network drop monitor service
[    0.634660] TCP: cubic registered
[    0.638076] Initializing XFRM netlink socket
[    0.642857] NET: Registered protocol family 10
[    0.647592] NET: Registered protocol family 17
[    0.652185] mpls_gso: MPLS GSO support
[    0.656305] microcode: CPU0 sig=0x506e3, pf=0x1, revision=0x1
[    0.663102] microcode: Microcode Update Driver: v2.01 <tigran.co.uk>, Peter Oruba
[    0.672077] Loading compiled-in X.509 certificates
[    0.676990] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'
[    0.686708] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'
[    0.697303] Loaded X.509 cert 'CentOS Linux kernel signing key: 666ef031933f5127062372832ce9ba8a49005c8f'
[    0.707096] registered taskstats version 1
[    0.711666] Key type trusted registered
[    0.715777] Key type encrypted registered
[    0.720110] IMA: No TPM chip found, activating TPM-bypass! (rc=-19)
[    0.728244]   Magic number: 2:741:80
[    0.732712] rtc_cmos rtc_cmos: setting system clock to 2018-05-10 14:05:27 UTC (1525961127)
[    0.743755] Freeing unused kernel memory: 1832k freed
[    0.749381] Write protecting the kernel read-only data: 12288k
[    0.757266] Freeing unused kernel memory: 856k freed
[    0.763765] Freeing unused kernel memory: 684k freed
supermin: mounting /proc^H^H^H
supermin: ext2 mini initrd starting up: 5.1.19 glibc
supermin: cmdline: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color
supermin: uptime: 0.79 0.04
supermin: mounting /sys
supermin: internal insmod crc32-pclmul.ko
supermin: internal insmod crc32c-intel.ko
supermin: internal insmod crct10dif_common.ko
supermin: internal insmod crct10dif-pclmul.ko
supermin: internal insmod crc32_generic.ko
supermin: internal insmod crct10dif_generic.ko
supermin: internal insmod libnvdimm.ko
supermin: internal insmod nfit.ko
insmod: init_module: nfit.ko: No such device
supermin: internal insmod libata.ko
supermin: internal insmod ata_piix.ko
[    0.926785] scsi host0: ata_piix
[    0.930366] scsi host1: ata_piix
[    0.933906] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc080 irq 14
[    0.941070] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc088 irq 15
supermin: internal insmod virtio.ko
supermin: internal insmod virtio_ring.ko
supermin: internal insmod virtio_blk.ko
supermin: internal insmod virtio-rng.ko
supermin: internal insmod virtio_console.ko
supermin: internal insmod virtio_net.ko
supermin: internal insmod nd_btt.ko
supermin: internal insmod nd_pmem.ko
supermin: internal insmod crc-t10dif.ko
supermin: internal insmod sd_mod.ko
supermin: internal insmod virtio_scsi.ko
supermin: internal insmod virtio_balloon.ko
supermin: internal insmod virtio_input.ko
supermin: internal insmod virtio_pci.ko
[    1.189956] virtio-pci 0000:00:03.0: PCI->APIC IRQ transform: INT A -> IRQ 25
[    1.198563] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    1.214094] scsi host2: Virtio SCSI HBA
[    1.220200] virtio-pci 0000:00:04.0: PCI->APIC IRQ transform: INT A -> IRQ 25
[    1.228860] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
[    1.236700] scsi 2:0:0:0: Direct-Access     QEMU     QEMU HARDDISK    1.5. PQ: 0 ANSI: 5
[    1.313327] virtio-pci 0000:00:05.0: PCI->APIC IRQ transform: INT A -> IRQ 24
[    1.321134] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
[    1.331860] scsi 2:0:1:0: Direct-Access     QEMU     QEMU HARDDISK    1.5. PQ: 0 ANSI: 5
[    1.340914] random: fast init done
[    1.345546] random: crng init done
[    1.409432] sd 2:0:0:0: [sda] 204800 512-byte logical blocks: (104 MB/100 MiB)
[    1.417565] sd 2:0:1:0: [sdb] 8388608 512-byte logical blocks: (4.29 GB/4.00 GiB)
[    1.428172] sd 2:0:0:0: [sda] Write Protect is off
[    1.433369] sd 2:0:1:0: [sdb] Write Protect is off
[    1.439382] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.449275] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.479612] sd 2:0:0:0: [sda] Attached SCSI disk
[    1.484644] sd 2:0:1:0: [sdb] Attached SCSI disk
supermin: internal insmod jbd2.ko
supermin: internal insmod mbcache.ko
supermin: internal insmod ext4.ko
supermin: internal insmod crc-ccitt.ko
supermin: internal insmod crc-itu-t.ko
supermin: internal insmod crc8.ko
supermin: internal insmod libcrc32c.ko
supermin: picked /sys/block/sdb/dev (8:16) as root device
supermin: creating /dev/root as block special 8:16
supermin: mounting new root on /root
[    1.630312] EXT4-fs (sdb): mounting ext2 file system using the ext4 subsystem
[    1.642496] EXT4-fs (sdb): mounted filesystem without journal. Opts: 
supermin: deleting initramfs files
supermin: chroot
[    1.657392] general protection fault: 0000 [#1] SMP 
[    1.658366] Modules linked in: libcrc32c crc8 crc_itu_t crc_ccitt ext4 mbcache jbd2 virtio_pci virtio_input virtio_balloon virtio_scsi sd_mod crc_t10dif nd_pmem nd_btt virtio_net virtio_console virtio_rng virtio_blk virtio_ring virtio ata_piix libata libnvdimm crct10dif_generic crc32_generic crct10dif_pclmul crct10dif_common crc32c_intel crc32_pclmul
[    1.658366] CPU: 0 PID: 1 Comm: init Not tainted 3.10.0-862.2.3.el7.x86_64 #1
[    1.658366] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[    1.658366] task: ffff95b29ead0000 ti: ffff95b29eacc000 task.ti: ffff95b29eacc000
[    1.658366] RIP: 0010:[<ffffffffa8c22c85>]  [<ffffffffa8c22c85>] flush_old_exec+0x725/0x980
[    1.658366] RSP: 0018:ffff95b29eacfc30  EFLAGS: 00010246
[    1.658366] RAX: 0000000000000001 RBX: ffff95b29d17c000 RCX: 0000000000000049
[    1.658366] RDX: 0000000000000000 RSI: ffff95b29d17c640 RDI: ffff95b29d17c000
[    1.658366] RBP: ffff95b29eacfc88 R08: 0000000000000000 R09: 0000000000000000
[    1.658366] R10: 6801e4782a61d580 R11: 0000006410d91040 R12: ffff95b29d17c640
[    1.658366] R13: ffff95b29d17c000 R14: ffff95b29ead0000 R15: ffff95b29ce77400
[    1.658366] FS:  0000000000000000(0000) GS:ffff95b29ee00000(0000) knlGS:0000000000000000
[    1.658366] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.658366] CR2: 000000000041fc40 CR3: 000000001d180000 CR4: 00000000003606f0
[    1.658366] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.658366] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    1.658366] Call Trace:
[    1.658366]  [<ffffffffa8c7e3ec>] load_elf_binary+0x33c/0xe50
[    1.658366]  [<ffffffffa8cf3aa3>] ? ima_get_action+0x23/0x30
[    1.658366]  [<ffffffffa8cf2fbe>] ? process_measurement+0x8e/0x250
[    1.658366]  [<ffffffffa8c7c3f0>] ? compat_sys_ioctl+0x360/0x360
[    1.658366]  [<ffffffffa8cf3479>] ? ima_bprm_check+0x49/0x50
[    1.658366]  [<ffffffffa8c7e0b0>] ? load_elf_library+0x220/0x220
[    1.658366]  [<ffffffffa8c21fcf>] search_binary_handler+0xef/0x310
[    1.658366]  [<ffffffffa8c7c3f0>] ? compat_sys_ioctl+0x360/0x360
[    1.658366]  [<ffffffffa8c7c655>] load_script+0x265/0x2a0
[    1.658366]  [<ffffffffa8cf3479>] ? ima_bprm_check+0x49/0x50
[    1.658366]  [<ffffffffa8c21fcf>] search_binary_handler+0xef/0x310
[    1.658366]  [<ffffffffa8c237ab>] do_execve_common.isra.24+0x5db/0x6e0
[    1.658366]  [<ffffffffa8c23b49>] SyS_execve+0x29/0x30
[    1.658366]  [<ffffffffa911fd18>] stub_execve+0x48/0x80
[    1.658366] Code: d0 e9 ff 4c 89 f7 e8 5b 33 e7 ff e9 4d fa ff ff 65 8b 05 97 a3 3e 57 a8 01 0f 84 85 fc ff ff 31 d2 b8 01 00 00 00 b9 49 00 00 00 <0f> 30 0f 1f 44 00 00 48 c7 c0 10 00 00 00 e8 07 00 00 00 f3 90 
[    1.658366] RIP  [<ffffffffa8c22c85>] flush_old_exec+0x725/0x980
[    1.658366]  RSP <ffff95b29eacfc30>
[    1.959422] ---[ end trace 224d670c68d22f56 ]---
[    1.964984] Kernel panic - not syncing: Fatal exception
[    1.965948] Kernel Offset: 0x27a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    1.965948] Rebooting in 1 seconds..libguestfs: error: appliance closed the connection unexpectedly, see earlier error messages
libguestfs: child_cleanup: 0x558f1f50cc60: child process died
libguestfs: error: guestfs_launch failed, see earlier error messages
libguestfs: closing guestfs handle 0x558f1f50cc60 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsd1jhpX
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /run/user/1000/libguestfs5RsFkU


Also not seeing the issue with: export LIBGUESTFS_BACKEND_SETTINGS=force_tcg
Comment 20 Bandan Das 2018-05-10 16:54:58 UTC 
(In reply to Yatin Karel from comment #19)
> Tested with centos 7.5 and seeing the issue on a vm on RDO Cloud, but not
> sure it's the same issue
> 

It looks the same to me from the trace. Can you try a RHEL kernel > 3.10.0-861.el7.x86_64 ?
Comment 21 Yatin Karel 2018-05-11 10:44:38 UTC 
(In reply to Bandan Das from comment #20)
> (In reply to Yatin Karel from comment #19)
> > Tested with centos 7.5 and seeing the issue on a vm on RDO Cloud, but not
> > sure it's the same issue
> > 
> 
> It looks the same to me from the trace. Can you try a RHEL kernel >
> 3.10.0-861.el7.x86_64 ?


I tried following RHEL kernels on Centos VM and still see the issue:-
kernel-3.10.0-862.el7.x86_64.rpm and kernel-3.10.0-862.2.3.el7.x86_64.rpm
Comment 22 Adam Huffman 2018-06-18 13:48:37 UTC 
I too am still seeing this with 3.10.0-862.2.3 on Skylake nodes from various vendors.
Comment 25 Richard W.M. Jones 2018-07-20 19:46:36 UTC 
*** Bug 1605034 has been marked as a duplicate of this bug. ***