Bug 2024662
| Summary: | Cirros guest appears to come up, but fails liveliness SSH check and console.log is not created. | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Artom Lifshitz <alifshit> |
| Component: | qemu-kvm | Assignee: | Virtualization Maintenance <virt-maint> |
| qemu-kvm sub component: | General | QA Contact: | Lei Yang <leiyang> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | coli, jinzhao, juzhang, lyarwood, virt-maint |
| Version: | 8.2 | ||
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-22 13:27:49 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I've perhaps been a bit trigger-happy with this, the following two other BZ may be related, or even outright duplicates: https://bugzilla.redhat.com/show_bug.cgi?id=2024605 https://bugzilla.redhat.com/show_bug.cgi?id=2007129 Hi Artom Could you please help me check the instance you mentioned above, whether it is really boot successful? I used the command line you provided, and the guest did not boot successfully. Guest show as: "Guest has not initialized the display (yet)" . I will add the guest info to the attachment, the name is: guest_status. From the perspective of this phenomenon, this is the same issue as the Bug 2001525 that I reported before. Please correct me if I'm wrong. Best Regards Lei Test Version: qemu-kvm-6.1.0-4.module+el8.6.0+13039+4b81a1dc.x86_64 kernel-4.18.0-350.el8.x86_64 Guest: rhel8.6 qemu CLI: /usr/libexec/qemu-kvm \ -name guest=instance-00000001,debug-threads=on \ -S \ -machine pc-q35-rhel8.5.0,accel=tcg,usb=off,dump-guest-core=off \ -cpu EPYC,acpi=on,ss=on,monitor=on,hypervisor=on,erms=on,mpx=on,pcommit=on,clwb=on,pku=on,la57=on,3dnowext=on,3dnow=on,npt=on,vme=off,fma=off,avx=off,f16c=off,avx2=off,rdseed=off,sha-ni=off,xsavec=off,fxsr-opt=off,misalignsse=off,3dnowprefetch=off,osvw=off,topoext=off,nrip-save=off \ -m 128 \ -overcommit mem-lock=off \ -smp 1,sockets=1,dies=1,cores=1,threads=1 \ -uuid 5f1da129-6595-41a3-b709-d21afcc1fb1e \ -no-user-config \ -nodefaults \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \ -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \ -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \ -device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \ -device pcie-root-port,port=0x18,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 \ -device pcie-root-port,port=0x19,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \ -device pcie-root-port,port=0x1a,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \ -device pcie-root-port,port=0x1b,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \ -device pcie-root-port,port=0x1c,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \ -device pcie-root-port,port=0x1d,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \ -device pcie-root-port,port=0x1e,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 \ -device pcie-root-port,port=0x1f,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 \ -device pcie-root-port,port=0x20,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 \ -device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 \ -device piix3-usb-uhci,id=usb,bus=pci.18,addr=0x1 \ -blockdev driver=file,filename=/home/kvm_autotest_root/images/rhel860-64-virtio-scsi.qcow2,node-name=libvirt-3-storage,cache.direct=on,cache.no-flush=off,auto-read-only=on,discard=unmap \ -blockdev node-name=libvirt-3-format,read-only=on,cache.direct=on,cache.no-flush=off,driver=qcow2,file=libvirt-3-storage \ -netdev tap,id=hostnet0 \ -device virtio-net-pci,rx_queue_size=512,host_mtu=1442,netdev=hostnet0,id=net0,mac=fa:16:3e:18:7a:aa,bus=pci.2,addr=0x0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -audiodev id=audio1,driver=none \ -vnc :0 \ -device VGA,id=video0,bus=pcie.0,addr=0x1 \ -device vmcoreinfo \ -msg timestamp=on \ -monitor stdio \ Hi Lei, If I understand your question correctly, then the answer is no, the boot is not successful. Libvirt and Nova seem to think that the guest has booted, but the guest OS itself hasn't actually started. Cheers! (In reply to Artom Lifshitz from comment #4) > Hi Lei, > > If I understand your question correctly, then the answer is no, the boot is > not successful. Libvirt and Nova seem to think that the guest has booted, > but the guest OS itself hasn't actually started. > > Cheers! Hi Artom Your understanding is correct. I think we encountered the same issue.that is: q35 cannot support more than 14 pcie-root-port devices. Best Regards Lei (In reply to Artom Lifshitz from comment #1) > I've perhaps been a bit trigger-happy with this, the following two other BZ > may be related, or even outright duplicates: > > https://bugzilla.redhat.com/show_bug.cgi?id=2024605 > https://bugzilla.redhat.com/show_bug.cgi?id=2007129 Yeah lets close this out as a duplicate of 2007129 *** This bug has been marked as a duplicate of bug 2007129 *** |
Description of problem: Cirros guest OS appears to boot just fine, but no console.log file is created, and the guest is not SSH'able. This is being observed in an OpenStack CI environment, but I've done my best to leave any OpenStack bits out of this, and provide only the relevant XML and qemu command lines. I've also made a point of taking the exact same CI test, and compare the passing result with qemu 6.0 vs the failing result with qemu 6.1. Version-Release number of selected component (if applicable): qemu 6.1, see further down for exact version information. How reproducible: Seems to be 100% when attempting to check guest liveliness. Some of the tests in the CI run succeed, but I was not able to find an example of a test that attempts to SSH into the guest succeeding. Steps to Reproduce: 1. Boot a cirros guest 2. Wait for the guest to become active by attempting to SSH to it. Actual results: SSH times out. In addition, console.log file is not created. Expected results: SSH succeeds, console.log file present and has contents. Additional info: XML (as given to libvirt by Nova) and qemu log (including command line) for failing instance on qemu 6.1: <domain type="qemu"> <uuid>23a551e2-d25b-4c41-b4b7-ba7f50cd2989</uuid> <name>instance-00000008</name> <memory>131072</memory> <vcpu>1</vcpu> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.1"> <nova:package version="24.1.0-0.20211109144301.78d398a.el8"/> <nova:name>tempest-TestNetworkBasicOps-server-1998301252</nova:name> <nova:creationTime>2021-11-16 00:13:21</nova:creationTime> <nova:flavor name="m1.nano"> <nova:memory>128</nova:memory> <nova:disk>1</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="4c626b8e560e48a49a68f45ffa0ca54f">tempest-TestNetworkBasicOps-372267726-project</nova:user> <nova:project uuid="16ec1dc2e1f2413d9cc977efba0047c9">tempest-TestNetworkBasicOps-372267726</nova:project> </nova:owner> <nova:root type="image" uuid="865242ae-097c-495d-8ad9-f94425f1935b"/> <nova:ports> <nova:port uuid="2306687c-b98b-462b-b9a8-099c9f275720"> <nova:ip type="fixed" address="10.100.0.5" ipVersion="4"/> </nova:port> </nova:ports> </nova:instance> </metadata> <sysinfo type="smbios"> <system> <entry name="manufacturer">RDO</entry> <entry name="product">OpenStack Compute</entry> <entry name="version">24.1.0-0.20211109144301.78d398a.el8</entry> <entry name="serial">23a551e2-d25b-4c41-b4b7-ba7f50cd2989</entry> <entry name="uuid">23a551e2-d25b-4c41-b4b7-ba7f50cd2989</entry> <entry name="family">Virtual Machine</entry> </system> </sysinfo> <os> <type machine="q35">hvm</type> <boot dev="hd"/> <smbios mode="sysinfo"/> </os> <features> <acpi/> <apic/> <vmcoreinfo/> </features> <cputune> <shares>1024</shares> </cputune> <clock offset="utc"/> <cpu mode="custom" match="exact"> <model>Nehalem</model> <topology sockets="1" cores="1" threads="1"/> </cpu> <devices> <disk type="file" device="disk"> <driver name="qemu" type="qcow2" cache="none"/> <source file="/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/disk"/> <target bus="virtio" dev="vda"/> </disk> <disk type="file" device="cdrom"> <driver name="qemu" type="raw" cache="none"/> <source file="/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/disk.config"/> <target bus="sata" dev="sda"/> </disk> <interface type="ethernet"> <mac address="fa:16:3e:1a:c2:fd"/> <model type="virtio"/> <driver name="qemu" rx_queue_size="512"/> <mtu size="1442"/> <target dev="tap2306687c-b9"/> </interface> <serial type="pty"> <log file="/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/console.log" append="off"/> </serial> <graphics type="vnc" autoport="yes" listen="192.168.24.1"/> <video> <model type="cirrus"/> </video> <input type="tablet" bus="usb"/> <rng model="virtio"> <backend model="random">/dev/urandom</backend> </rng> <controller type="pci" model="pcie-root"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="usb" index="0"/> <memballoon model="virtio"> <stats period="10"/> </memballoon> </devices> </domain> [artom@zoe not-working-qemu-6.1]$ cat ./logs/undercloud/var/log/containers/libvirt/qemu/instance-00000008.log 2021-11-16 00:13:22.224+0000: starting up libvirt version: 7.9.0, package: 1.module_el8.6.0+983+a7505f3f (CentOS Buildsys <bugs>, 2021-11-09-20:38:08, ), qemu version: 6.1.0qemu-kvm-6.1.0-4.module_el8.6.0+983+a7505f3f, kernel: 4.18.0-348.el8.x86_64, hostname: standalone.localdomain LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-8-instance-00000008 \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-8-instance-00000008/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-8-instance-00000008/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-8-instance-00000008/.config \ /usr/libexec/qemu-kvm \ -name guest=instance-00000008,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-8-instance-00000008/master-key.aes"}' \ -machine pc-q35-rhel8.5.0,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram \ -cpu Nehalem \ -m 128 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":134217728}' \ -overcommit mem-lock=off \ -smp 1,sockets=1,dies=1,cores=1,threads=1 \ -uuid 23a551e2-d25b-4c41-b4b7-ba7f50cd2989 \ -smbios 'type=1,manufacturer=RDO,product=OpenStack Compute,version=24.1.0-0.20211109144301.78d398a.el8,serial=23a551e2-d25b-4c41-b4b7-ba7f50cd2989,uuid=23a551e2-d25b-4c41-b4b7-ba7f50cd2989,family=Virtual Machine' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=29,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \ -device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \ -device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \ -device pcie-root-port,port=23,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \ -device pcie-root-port,port=24,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 \ -device pcie-root-port,port=25,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \ -device pcie-root-port,port=26,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \ -device pcie-root-port,port=27,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \ -device pcie-root-port,port=28,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \ -device pcie-root-port,port=29,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \ -device pcie-root-port,port=30,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 \ -device pcie-root-port,port=31,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 \ -device pcie-root-port,port=32,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 \ -device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 \ -device piix3-usb-uhci,id=usb,bus=pci.18,addr=0x1 \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/d2db1cfc799c17bfd288a14cb9a44a306c466c62","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/disk","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \ -device virtio-blk-pci,bus=pci.3,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/disk.config","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device ide-cd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0,write-cache=on \ -netdev tap,fd=31,id=hostnet0 \ -device virtio-net-pci,rx_queue_size=512,host_mtu=1442,netdev=hostnet0,id=net0,mac=fa:16:3e:1a:c2:fd,bus=pci.2,addr=0x0 \ -add-fd set=2,fd=39 \ -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \ -device isa-serial,chardev=charserial0,id=serial0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -audiodev id=audio1,driver=none \ -vnc 192.168.24.1:0,audiodev=audio1 \ -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \ -device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \ -object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \ -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.5,addr=0x0 \ -device vmcoreinfo \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/0 (label charserial0) 2021-11-16T00:13:22.331486Z qemu-kvm: -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead 2021-11-16T00:15:30.005083Z qemu-kvm: terminating on signal 15 from pid 113833 (/usr/sbin/virtqemud) 2021-11-16 00:15:30.406+0000: shutting down, reason=destroyed XML (as given to libvirt by Nova) and qemu log (including command line) for failing instance on qemu 6.0: <domain type="qemu"> <uuid>5f1da129-6595-41a3-b709-d21afcc1fb1e</uuid> <name>instance-00000001</name> <memory>131072</memory> <vcpu>1</vcpu> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.1"> <nova:package version="24.1.0-0.20211109144301.78d398a.el8"/> <nova:name>tempest-TestNetworkBasicOps-server-1523392928</nova:name> <nova:creationTime>2021-11-16 08:43:06</nova:creationTime> <nova:flavor name="m1.nano"> <nova:memory>128</nova:memory> <nova:disk>1</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>1</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="9a2f0e469c414133a222635dd2a8bac3">tempest-TestNetworkBasicOps-547174707-project</nova:user> <nova:project uuid="1434734d75bc4c37a87d84d506d75eca">tempest-TestNetworkBasicOps-547174707</nova:project> </nova:owner> <nova:root type="image" uuid="6e28a1c4-4301-4441-bf71-7e21549ab1ac"/> <nova:ports> <nova:port uuid="e753e7e3-f03a-4a87-ae11-cc668df35890"> <nova:ip type="fixed" address="10.100.0.3" ipVersion="4"/> </nova:port> </nova:ports> </nova:instance> </metadata> <sysinfo type="smbios"> <system> <entry name="manufacturer">RDO</entry> <entry name="product">OpenStack Compute</entry> <entry name="version">24.1.0-0.20211109144301.78d398a.el8</entry> <entry name="serial">5f1da129-6595-41a3-b709-d21afcc1fb1e</entry> <entry name="uuid">5f1da129-6595-41a3-b709-d21afcc1fb1e</entry> <entry name="family">Virtual Machine</entry> </system> </sysinfo> <os> <type machine="q35">hvm</type> <boot dev="hd"/> <smbios mode="sysinfo"/> </os> <features> <acpi/> <apic/> <vmcoreinfo/> </features> <cputune> <shares>1024</shares> </cputune> <clock offset="utc"/> <cpu mode="host-model" match="exact"> <topology sockets="1" cores="1" threads="1"/> </cpu> <devices> <disk type="file" device="disk"> <driver name="qemu" type="qcow2" cache="none"/> <source file="/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/disk"/> <target bus="virtio" dev="vda"/> </disk> <disk type="file" device="cdrom"> <driver name="qemu" type="raw" cache="none"/> <source file="/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/disk.config"/> <target bus="sata" dev="sda"/> </disk> <interface type="ethernet"> <mac address="fa:16:3e:18:7a:aa"/> <model type="virtio"/> <driver name="qemu" rx_queue_size="512"/> <mtu size="1442"/> <target dev="tape753e7e3-f0"/> </interface> <serial type="pty"> <log file="/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/console.log" append="off"/> </serial> <graphics type="vnc" autoport="yes" listen="192.168.24.1"/> <video> <model type="cirrus"/> </video> <input type="tablet" bus="usb"/> <rng model="virtio"> <backend model="random">/dev/urandom</backend> </rng> <controller type="pci" model="pcie-root"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="pci" model="pcie-root-port"/> <controller type="usb" index="0"/> <memballoon model="virtio"> <stats period="10"/> </memballoon> </devices> </domain> [artom@zoe working-qemu-6.0]$ cat ./logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log 2021-11-16 08:43:09.318+0000: starting up libvirt version: 7.6.0, package: 6.el8s (CBS <cbs>, 2021-10-29-15:04:36, ), qemu version: 6.0.0qemu-kvm-6.0.0-33.el8s, kernel: 4.18.0-348.el8.x86_64, hostname: standalone.localdomain LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001 \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.config \ /usr/libexec/qemu-kvm \ -name guest=instance-00000001,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-instance-00000001/master-key.aes"}' \ -machine pc-q35-rhel8.5.0,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram \ -cpu EPYC,acpi=on,ss=on,monitor=on,hypervisor=on,erms=on,mpx=on,pcommit=on,clwb=on,pku=on,la57=on,3dnowext=on,3dnow=on,npt=on,vme=off,fma=off,avx=off,f16c=off,avx2=off,rdseed=off,sha-ni=off,xsavec=off,fxsr-opt=off,misalignsse=off,3dnowprefetch=off,osvw=off,topoext=off,nrip-save=off \ -m 128 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":134217728}' \ -overcommit mem-lock=off \ -smp 1,sockets=1,dies=1,cores=1,threads=1 \ -uuid 5f1da129-6595-41a3-b709-d21afcc1fb1e \ -smbios 'type=1,manufacturer=RDO,product=OpenStack Compute,version=24.1.0-0.20211109144301.78d398a.el8,serial=5f1da129-6595-41a3-b709-d21afcc1fb1e,uuid=5f1da129-6595-41a3-b709-d21afcc1fb1e,family=Virtual Machine' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=25,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \ -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \ -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \ -device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \ -device pcie-root-port,port=0x18,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 \ -device pcie-root-port,port=0x19,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \ -device pcie-root-port,port=0x1a,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \ -device pcie-root-port,port=0x1b,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \ -device pcie-root-port,port=0x1c,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \ -device pcie-root-port,port=0x1d,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \ -device pcie-root-port,port=0x1e,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 \ -device pcie-root-port,port=0x1f,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 \ -device pcie-root-port,port=0x20,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 \ -device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 \ -device piix3-usb-uhci,id=usb,bus=pci.18,addr=0x1 \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/4008edaf667b4650b8e7e03e58f2147fb52de3dd","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/disk","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \ -device virtio-blk-pci,bus=pci.3,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/disk.config","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device ide-cd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0,write-cache=on \ -netdev tap,fd=27,id=hostnet0 \ -device virtio-net-pci,rx_queue_size=512,host_mtu=1442,netdev=hostnet0,id=net0,mac=fa:16:3e:18:7a:aa,bus=pci.2,addr=0x0 \ -add-fd set=2,fd=29 \ -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \ -device isa-serial,chardev=charserial0,id=serial0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -audiodev id=audio1,driver=none \ -vnc 192.168.24.1:0,audiodev=audio1 \ -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \ -device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \ -object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \ -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.5,addr=0x0 \ -device vmcoreinfo \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/0 (label charserial0) 2021-11-16T08:43:09.446180Z qemu-kvm: -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead 2021-11-16T08:44:36.410097Z qemu-kvm: terminating on signal 15 from pid 112474 (/usr/sbin/virtqemud) 2021-11-16 08:44:36.611+0000: shutting down, reason=destroyed We are able to hold a CI node that's experiencing the failures if any further debugging/information is needed.