RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2024662 - Cirros guest appears to come up, but fails liveliness SSH check and console.log is not created.
Summary: Cirros guest appears to come up, but fails liveliness SSH check and console.l...
Keywords:
Status: CLOSED DUPLICATE of bug 2007129
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: 8.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Virtualization Maintenance
QA Contact: Lei Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-18 15:58 UTC by Artom Lifshitz
Modified: 2021-11-24 01:14 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-22 13:27:49 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-103185 0 None None None 2021-11-18 16:00:26 UTC

Description Artom Lifshitz 2021-11-18 15:58:53 UTC 
Description of problem:

Cirros guest OS appears to boot just fine, but no console.log file is created, and the guest is not SSH'able.

This is being observed in an OpenStack CI environment, but I've done my best to leave any OpenStack bits out of this, and provide only the relevant XML and qemu command lines.

I've also made a point of taking the exact same CI test, and compare the passing result with qemu 6.0 vs the failing result with qemu 6.1.


Version-Release number of selected component (if applicable):

qemu 6.1, see further down for exact version information.


How reproducible:

Seems to be 100% when attempting to check guest liveliness. Some of the tests in the CI run succeed, but I was not able to find an example of a test that attempts to SSH into the guest succeeding.


Steps to Reproduce:
1. Boot a cirros guest
2. Wait for the guest to become active by attempting to SSH to it.

Actual results:
SSH times out. In addition, console.log file is not created.

Expected results:
SSH succeeds, console.log file present and has contents.


Additional info:


XML (as given to libvirt by Nova) and qemu log (including command line) for failing instance on qemu 6.1:

<domain type="qemu">
  <uuid>23a551e2-d25b-4c41-b4b7-ba7f50cd2989</uuid>
  <name>instance-00000008</name>
  <memory>131072</memory>
  <vcpu>1</vcpu>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.1">
      <nova:package version="24.1.0-0.20211109144301.78d398a.el8"/>
      <nova:name>tempest-TestNetworkBasicOps-server-1998301252</nova:name>
      <nova:creationTime>2021-11-16 00:13:21</nova:creationTime>
      <nova:flavor name="m1.nano">
        <nova:memory>128</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="4c626b8e560e48a49a68f45ffa0ca54f">tempest-TestNetworkBasicOps-372267726-project</nova:user>
        <nova:project uuid="16ec1dc2e1f2413d9cc977efba0047c9">tempest-TestNetworkBasicOps-372267726</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="865242ae-097c-495d-8ad9-f94425f1935b"/>
      <nova:ports>
        <nova:port uuid="2306687c-b98b-462b-b9a8-099c9f275720">
          <nova:ip type="fixed" address="10.100.0.5" ipVersion="4"/>
        </nova:port>
      </nova:ports>
    </nova:instance>
  </metadata>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">RDO</entry>
      <entry name="product">OpenStack Compute</entry>
      <entry name="version">24.1.0-0.20211109144301.78d398a.el8</entry>
      <entry name="serial">23a551e2-d25b-4c41-b4b7-ba7f50cd2989</entry>
      <entry name="uuid">23a551e2-d25b-4c41-b4b7-ba7f50cd2989</entry>
      <entry name="family">Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type machine="q35">hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmcoreinfo/>
  </features>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <clock offset="utc"/>
  <cpu mode="custom" match="exact">
    <model>Nehalem</model>
    <topology sockets="1" cores="1" threads="1"/>
  </cpu>
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" cache="none"/>
      <source file="/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/disk.config"/>
      <target bus="sata" dev="sda"/>
    </disk>
    <interface type="ethernet">
      <mac address="fa:16:3e:1a:c2:fd"/>
      <model type="virtio"/>
      <driver name="qemu" rx_queue_size="512"/>
      <mtu size="1442"/>
      <target dev="tap2306687c-b9"/>
    </interface>
    <serial type="pty">
      <log file="/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/console.log" append="off"/>
    </serial>
    <graphics type="vnc" autoport="yes" listen="192.168.24.1"/>
    <video>
      <model type="cirrus"/>
    </video>
    <input type="tablet" bus="usb"/>
    <rng model="virtio">
      <backend model="random">/dev/urandom</backend>
    </rng>
    <controller type="pci" model="pcie-root"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="usb" index="0"/>
    <memballoon model="virtio">
      <stats period="10"/>
    </memballoon>
  </devices>
</domain>

[artom@zoe not-working-qemu-6.1]$ cat ./logs/undercloud/var/log/containers/libvirt/qemu/instance-00000008.log
2021-11-16 00:13:22.224+0000: starting up libvirt version: 7.9.0, package: 1.module_el8.6.0+983+a7505f3f (CentOS Buildsys <bugs>, 2021-11-09-20:38:08, ), qemu version: 6.1.0qemu-kvm-6.1.0-4.module_el8.6.0+983+a7505f3f, kernel: 4.18.0-348.el8.x86_64, hostname: standalone.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-8-instance-00000008 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-8-instance-00000008/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-8-instance-00000008/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-8-instance-00000008/.config \
/usr/libexec/qemu-kvm \
-name guest=instance-00000008,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-8-instance-00000008/master-key.aes"}' \
-machine pc-q35-rhel8.5.0,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-cpu Nehalem \
-m 128 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":134217728}' \
-overcommit mem-lock=off \
-smp 1,sockets=1,dies=1,cores=1,threads=1 \
-uuid 23a551e2-d25b-4c41-b4b7-ba7f50cd2989 \
-smbios 'type=1,manufacturer=RDO,product=OpenStack Compute,version=24.1.0-0.20211109144301.78d398a.el8,serial=23a551e2-d25b-4c41-b4b7-ba7f50cd2989,uuid=23a551e2-d25b-4c41-b4b7-ba7f50cd2989,family=Virtual Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=29,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
-device pcie-root-port,port=23,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \
-device pcie-root-port,port=24,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 \
-device pcie-root-port,port=25,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \
-device pcie-root-port,port=26,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \
-device pcie-root-port,port=27,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \
-device pcie-root-port,port=28,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \
-device pcie-root-port,port=29,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \
-device pcie-root-port,port=30,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 \
-device pcie-root-port,port=31,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 \
-device pcie-root-port,port=32,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 \
-device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 \
-device piix3-usb-uhci,id=usb,bus=pci.18,addr=0x1 \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/d2db1cfc799c17bfd288a14cb9a44a306c466c62","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/disk","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device virtio-blk-pci,bus=pci.3,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/23a551e2-d25b-4c41-b4b7-ba7f50cd2989/disk.config","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0,write-cache=on \
-netdev tap,fd=31,id=hostnet0 \
-device virtio-net-pci,rx_queue_size=512,host_mtu=1442,netdev=hostnet0,id=net0,mac=fa:16:3e:1a:c2:fd,bus=pci.2,addr=0x0 \
-add-fd set=2,fd=39 \
-chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev id=audio1,driver=none \
-vnc 192.168.24.1:0,audiodev=audio1 \
-device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.5,addr=0x0 \
-device vmcoreinfo \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
2021-11-16T00:13:22.331486Z qemu-kvm: -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2021-11-16T00:15:30.005083Z qemu-kvm: terminating on signal 15 from pid 113833 (/usr/sbin/virtqemud)
2021-11-16 00:15:30.406+0000: shutting down, reason=destroyed


XML (as given to libvirt by Nova) and qemu log (including command line) for failing instance on qemu 6.0:

<domain type="qemu">
  <uuid>5f1da129-6595-41a3-b709-d21afcc1fb1e</uuid>
  <name>instance-00000001</name>
  <memory>131072</memory>
  <vcpu>1</vcpu>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.1">
      <nova:package version="24.1.0-0.20211109144301.78d398a.el8"/>
      <nova:name>tempest-TestNetworkBasicOps-server-1523392928</nova:name>
      <nova:creationTime>2021-11-16 08:43:06</nova:creationTime>
      <nova:flavor name="m1.nano">
        <nova:memory>128</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="9a2f0e469c414133a222635dd2a8bac3">tempest-TestNetworkBasicOps-547174707-project</nova:user>
        <nova:project uuid="1434734d75bc4c37a87d84d506d75eca">tempest-TestNetworkBasicOps-547174707</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="6e28a1c4-4301-4441-bf71-7e21549ab1ac"/>
      <nova:ports>
        <nova:port uuid="e753e7e3-f03a-4a87-ae11-cc668df35890">
          <nova:ip type="fixed" address="10.100.0.3" ipVersion="4"/>
        </nova:port>
      </nova:ports>
    </nova:instance>
  </metadata>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">RDO</entry>
      <entry name="product">OpenStack Compute</entry>
      <entry name="version">24.1.0-0.20211109144301.78d398a.el8</entry>
      <entry name="serial">5f1da129-6595-41a3-b709-d21afcc1fb1e</entry>
      <entry name="uuid">5f1da129-6595-41a3-b709-d21afcc1fb1e</entry>
      <entry name="family">Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type machine="q35">hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmcoreinfo/>
  </features>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <clock offset="utc"/>
  <cpu mode="host-model" match="exact">
    <topology sockets="1" cores="1" threads="1"/>
  </cpu>
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw" cache="none"/>
      <source file="/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/disk.config"/>
      <target bus="sata" dev="sda"/>
    </disk>
    <interface type="ethernet">
      <mac address="fa:16:3e:18:7a:aa"/>
      <model type="virtio"/>
      <driver name="qemu" rx_queue_size="512"/>
      <mtu size="1442"/>
      <target dev="tape753e7e3-f0"/>
    </interface>
    <serial type="pty">
      <log file="/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/console.log" append="off"/>
    </serial>
    <graphics type="vnc" autoport="yes" listen="192.168.24.1"/>
    <video>
      <model type="cirrus"/>
    </video>
    <input type="tablet" bus="usb"/>
    <rng model="virtio">
      <backend model="random">/dev/urandom</backend>
    </rng>
    <controller type="pci" model="pcie-root"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="pci" model="pcie-root-port"/>
    <controller type="usb" index="0"/>
    <memballoon model="virtio">
      <stats period="10"/>
    </memballoon>
  </devices>
</domain>

[artom@zoe working-qemu-6.0]$ cat ./logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log
2021-11-16 08:43:09.318+0000: starting up libvirt version: 7.6.0, package: 6.el8s (CBS <cbs>, 2021-10-29-15:04:36, ), qemu version: 6.0.0qemu-kvm-6.0.0-33.el8s, kernel: 4.18.0-348.el8.x86_64, hostname: standalone.localdomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.config \
/usr/libexec/qemu-kvm \
-name guest=instance-00000001,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-instance-00000001/master-key.aes"}' \
-machine pc-q35-rhel8.5.0,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-cpu EPYC,acpi=on,ss=on,monitor=on,hypervisor=on,erms=on,mpx=on,pcommit=on,clwb=on,pku=on,la57=on,3dnowext=on,3dnow=on,npt=on,vme=off,fma=off,avx=off,f16c=off,avx2=off,rdseed=off,sha-ni=off,xsavec=off,fxsr-opt=off,misalignsse=off,3dnowprefetch=off,osvw=off,topoext=off,nrip-save=off \
-m 128 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":134217728}' \
-overcommit mem-lock=off \
-smp 1,sockets=1,dies=1,cores=1,threads=1 \
-uuid 5f1da129-6595-41a3-b709-d21afcc1fb1e \
-smbios 'type=1,manufacturer=RDO,product=OpenStack Compute,version=24.1.0-0.20211109144301.78d398a.el8,serial=5f1da129-6595-41a3-b709-d21afcc1fb1e,uuid=5f1da129-6595-41a3-b709-d21afcc1fb1e,family=Virtual Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=25,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
-device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \
-device pcie-root-port,port=0x18,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 \
-device pcie-root-port,port=0x19,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \
-device pcie-root-port,port=0x1a,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \
-device pcie-root-port,port=0x1b,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \
-device pcie-root-port,port=0x1c,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \
-device pcie-root-port,port=0x1d,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \
-device pcie-root-port,port=0x1e,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 \
-device pcie-root-port,port=0x1f,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 \
-device pcie-root-port,port=0x20,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 \
-device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 \
-device piix3-usb-uhci,id=usb,bus=pci.18,addr=0x1 \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/4008edaf667b4650b8e7e03e58f2147fb52de3dd","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/disk","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device virtio-blk-pci,bus=pci.3,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/5f1da129-6595-41a3-b709-d21afcc1fb1e/disk.config","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0,write-cache=on \
-netdev tap,fd=27,id=hostnet0 \
-device virtio-net-pci,rx_queue_size=512,host_mtu=1442,netdev=hostnet0,id=net0,mac=fa:16:3e:18:7a:aa,bus=pci.2,addr=0x0 \
-add-fd set=2,fd=29 \
-chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev id=audio1,driver=none \
-vnc 192.168.24.1:0,audiodev=audio1 \
-device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.5,addr=0x0 \
-device vmcoreinfo \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
2021-11-16T08:43:09.446180Z qemu-kvm: -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2021-11-16T08:44:36.410097Z qemu-kvm: terminating on signal 15 from pid 112474 (/usr/sbin/virtqemud)
2021-11-16 08:44:36.611+0000: shutting down, reason=destroyed


We are able to hold a CI node that's experiencing the failures if any further debugging/information is needed.
Comment 1 Artom Lifshitz 2021-11-18 16:51:32 UTC 
I've perhaps been a bit trigger-happy with this, the following two other BZ may be related, or even outright duplicates:

https://bugzilla.redhat.com/show_bug.cgi?id=2024605
https://bugzilla.redhat.com/show_bug.cgi?id=2007129
Comment 3 Lei Yang 2021-11-19 10:08:10 UTC 
Hi Artom

Could you please help me check the instance you mentioned above, whether it is really boot successful? I used the command line you provided, and the guest did not boot successfully. Guest show as: "Guest has not initialized the display (yet)" . I will add the guest info to the attachment, the name is: guest_status. From the perspective of this phenomenon, this is the same issue as the Bug 2001525 that I reported before. Please correct me if I'm wrong.

Best Regards
Lei

Test Version:
qemu-kvm-6.1.0-4.module+el8.6.0+13039+4b81a1dc.x86_64
kernel-4.18.0-350.el8.x86_64

Guest: rhel8.6

qemu CLI:
/usr/libexec/qemu-kvm \
-name guest=instance-00000001,debug-threads=on \
-S \
-machine pc-q35-rhel8.5.0,accel=tcg,usb=off,dump-guest-core=off \
-cpu EPYC,acpi=on,ss=on,monitor=on,hypervisor=on,erms=on,mpx=on,pcommit=on,clwb=on,pku=on,la57=on,3dnowext=on,3dnow=on,npt=on,vme=off,fma=off,avx=off,f16c=off,avx2=off,rdseed=off,sha-ni=off,xsavec=off,fxsr-opt=off,misalignsse=off,3dnowprefetch=off,osvw=off,topoext=off,nrip-save=off \
-m 128 \
-overcommit mem-lock=off \
-smp 1,sockets=1,dies=1,cores=1,threads=1 \
-uuid 5f1da129-6595-41a3-b709-d21afcc1fb1e \
-no-user-config \
-nodefaults \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
-device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \
-device pcie-root-port,port=0x18,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 \
-device pcie-root-port,port=0x19,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \
-device pcie-root-port,port=0x1a,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \
-device pcie-root-port,port=0x1b,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \
-device pcie-root-port,port=0x1c,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \
-device pcie-root-port,port=0x1d,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \
-device pcie-root-port,port=0x1e,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 \
-device pcie-root-port,port=0x1f,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 \
-device pcie-root-port,port=0x20,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 \
-device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 \
-device piix3-usb-uhci,id=usb,bus=pci.18,addr=0x1 \
-blockdev driver=file,filename=/home/kvm_autotest_root/images/rhel860-64-virtio-scsi.qcow2,node-name=libvirt-3-storage,cache.direct=on,cache.no-flush=off,auto-read-only=on,discard=unmap \
-blockdev node-name=libvirt-3-format,read-only=on,cache.direct=on,cache.no-flush=off,driver=qcow2,file=libvirt-3-storage \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,rx_queue_size=512,host_mtu=1442,netdev=hostnet0,id=net0,mac=fa:16:3e:18:7a:aa,bus=pci.2,addr=0x0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev id=audio1,driver=none \
-vnc :0 \
-device VGA,id=video0,bus=pcie.0,addr=0x1 \
-device vmcoreinfo \
-msg timestamp=on \
-monitor stdio \
Comment 4 Artom Lifshitz 2021-11-19 14:34:02 UTC 
Hi Lei,

If I understand your question correctly, then the answer is no, the boot is not successful. Libvirt and Nova seem to think that the guest has booted, but the guest OS itself hasn't actually started.

Cheers!
Comment 5 Lei Yang 2021-11-22 01:28:27 UTC 
(In reply to Artom Lifshitz from comment #4)
> Hi Lei,
> 
> If I understand your question correctly, then the answer is no, the boot is
> not successful. Libvirt and Nova seem to think that the guest has booted,
> but the guest OS itself hasn't actually started.
> 
> Cheers!

Hi Artom

Your understanding is correct. I think we encountered the same issue.that is: q35 cannot support more than 14 pcie-root-port devices.

Best Regards
Lei
Comment 6 Lee Yarwood 2021-11-22 13:27:49 UTC 
(In reply to Artom Lifshitz from comment #1)
> I've perhaps been a bit trigger-happy with this, the following two other BZ
> may be related, or even outright duplicates:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=2024605
> https://bugzilla.redhat.com/show_bug.cgi?id=2007129

Yeah lets close this out as a duplicate of 2007129

*** This bug has been marked as a duplicate of bug 2007129 ***
Note You need to log in before you can comment on or make changes to this bug.