Description of problem: Newly created VM, with the following vfio devices: - GPU (nVidia 1080). - Sound card (Sound Blaster XF-I + nVidia HDMI audio). - USB 2 (Intel) + USB 3 (Renesas). When the machine is configured as Q35 BIOS / Q35 UEFI machine, qemu fails when trying to start the VM. When configured as i440FX, qemu work just fine. The host is part of a 3 machine gluster CentOS 8 streams HE setup, with the specific host (dual E5-2658v2 w/ 128GB RAM) is lightly loaded (~80GB free). Reducing the VM memory requirements has no effect. NOTE.1: Possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=2023313 NOTE.2: qemu currently downgraded to 6.0 to avoid https://bugzilla.redhat.com/show_bug.cgi?id=2007129 Version-Release number of selected component (if applicable): $ $ rpm -qa | /bin/egrep -e 'qemu|vdsm' | sort ipxe-roms-qemu-20181214-8.git133f4c47.el8.noarch libvirt-daemon-driver-qemu-7.9.0-1.module_el8.6.0+983+a7505f3f.x86_64 qemu-img-6.0.0-33.el8s.x86_64 qemu-kvm-6.0.0-33.el8s.x86_64 qemu-kvm-block-curl-6.0.0-33.el8s.x86_64 qemu-kvm-block-gluster-6.0.0-33.el8s.x86_64 qemu-kvm-block-iscsi-6.0.0-33.el8s.x86_64 qemu-kvm-block-rbd-6.0.0-33.el8s.x86_64 qemu-kvm-block-ssh-6.0.0-33.el8s.x86_64 qemu-kvm-common-6.0.0-33.el8s.x86_64 qemu-kvm-core-6.0.0-33.el8s.x86_64 qemu-kvm-docs-6.0.0-33.el8s.x86_64 qemu-kvm-hw-usbredir-6.0.0-33.el8s.x86_64 qemu-kvm-ui-opengl-6.0.0-33.el8s.x86_64 qemu-kvm-ui-spice-6.0.0-33.el8s.x86_64 vdsm-4.40.90.4-1.el8.x86_64 vdsm-api-4.40.90.4-1.el8.noarch vdsm-client-4.40.90.4-1.el8.noarch vdsm-common-4.40.90.4-1.el8.noarch vdsm-gluster-4.40.90.4-1.el8.x86_64 vdsm-hook-cpuflags-4.40.90.4-1.el8.noarch vdsm-hook-nestedvt-4.40.90.4-1.el8.noarch vdsm-hook-qemucmdline-4.40.90.4-1.el8.noarch vdsm-hook-scratchpad-4.40.90.4-1.el8.noarch vdsm-hook-spiceoptions-4.40.90.4-1.el8.noarch vdsm-http-4.40.90.4-1.el8.noarch vdsm-jsonrpc-4.40.90.4-1.el8.noarch vdsm-network-4.40.90.4-1.el8.x86_64 vdsm-python-4.40.90.4-1.el8.noarch vdsm-yajsonrpc-4.40.90.4-1.el8.noarch $ rpm -qa | /bin/egrep -e '^ovirt-engine' | sort ovirt-engine-4.4.9.5-1.el8.noarch ovirt-engine-backend-4.4.9.5-1.el8.noarch ovirt-engine-dbscripts-4.4.9.5-1.el8.noarch ovirt-engine-dwh-4.4.9.1-1.el8.noarch ovirt-engine-dwh-grafana-integration-setup-4.4.9.1-1.el8.noarch ovirt-engine-dwh-setup-4.4.9.1-1.el8.noarch ovirt-engine-extension-aaa-jdbc-1.2.0-1.el8.noarch ovirt-engine-extension-aaa-ldap-1.4.5-1.el8.noarch ovirt-engine-extension-aaa-ldap-setup-1.4.5-1.el8.noarch ovirt-engine-extension-aaa-misc-1.1.0-1.el8.noarch ovirt-engine-extension-logger-log4j-1.1.1-1.el8.noarch ovirt-engine-extensions-api-1.0.1-1.el8.noarch ovirt-engine-metrics-1.4.4-1.el8.noarch ovirt-engine-restapi-4.4.9.5-1.el8.noarch ovirt-engine-setup-4.4.9.5-1.el8.noarch ovirt-engine-setup-base-4.4.9.5-1.el8.noarch ovirt-engine-setup-plugin-cinderlib-4.4.9.5-1.el8.noarch ovirt-engine-setup-plugin-imageio-4.4.9.5-1.el8.noarch ovirt-engine-setup-plugin-ovirt-engine-4.4.9.5-1.el8.noarch ovirt-engine-setup-plugin-ovirt-engine-common-4.4.9.5-1.el8.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.4.9.5-1.el8.noarch ovirt-engine-setup-plugin-websocket-proxy-4.4.9.5-1.el8.noarch ovirt-engine-tools-4.4.9.5-1.el8.noarch ovirt-engine-tools-backup-4.4.9.5-1.el8.noarch ovirt-engine-ui-extensions-1.2.7-1.el8.noarch ovirt-engine-vmconsole-proxy-helper-4.4.9.5-1.el8.noarch ovirt-engine-webadmin-portal-4.4.9.5-1.el8.noarch ovirt-engine-websocket-proxy-4.4.9.5-1.el8.noarch ovirt-engine-wildfly-23.0.2-1.el8.x86_64 ovirt-engine-wildfly-overlay-23.0.2-1.el8.noarch How reproducible: Always. Additional info: qemu error: 2021-11-26T11:03:15.550768Z qemu-kvm: -device vfio-pci,host=0000:07:00.0,id=ua-4b800d46-2c67-4d7f-a9b8-9282c2c9de4c,bus=pci.9,addr=0x0: VFIO_MAP_DMA failed: Cannot allocate memory 2021-11-26T11:03:15.619430Z qemu-kvm: -device vfio-pci,host=0000:07:00.0,id=ua-4b800d46-2c67-4d7f-a9b8-9282c2c9de4c,bus=pci.9,addr=0x0: VFIO_MAP_DMA failed: Cannot allocate memory 2021-11-26T11:03:15.619762Z qemu-kvm: -device vfio-pci,host=0000:07:00.0,id=ua-4b800d46-2c67-4d7f-a9b8-9282c2c9de4c,bus=pci.9,addr=0x0: vfio 0000:07:00.0: failed to setup container for group 33: memory listener initialization failed: Region ram-node0: vfio_dma_map(0x5607e41ffe40, 0xc0000, 0x7ff40000, 0x7fc02bec0000) = -12 (Cannot allocate memory) 2021-11-26 11:03:15.810+0000: shutting down, reason=failed qemu command line (q35-bios): 2021-11-26 11:03:04.711+0000: starting up libvirt version: 7.9.0, package: 1.module_el8.6.0+983+a7505f3f (CentOS Buildsys <bugs>, 2021-11-09-20:38:08, ), qemu version: 6.0.0qemu-kvm-6.0.0-33.el8s, kernel: 4.18.0-348.2.1.el8_5.x86_64, hostname: gilboa-home-hv1-dev.localdomain LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ HOME=/var/lib/libvirt/qemu/domain-12-gilboa-home-dev \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-12-gilboa-home-dev/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-12-gilboa-home-dev/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-12-gilboa-home-dev/.config \ /usr/libexec/qemu-kvm \ -name guest=gilboa-home-dev,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-12-gilboa-home-dev/master-key.aes"}' \ -machine pc-q35-rhel8.4.0,accel=kvm,usb=off,dump-guest-core=off,kernel_irqchip=split \ -cpu SandyBridge,pcid=on,spec-ctrl=on,ssbd=on,md-clear=on \ -m size=33554432k,slots=16,maxmem=134217728k \ -overcommit mem-lock=off \ -smp 20,maxcpus=320,sockets=16,dies=1,cores=20,threads=1 \ -object '{"qom-type":"iothread","id":"iothread1"}' \ -object '{"qom-type":"iothread","id":"iothread2"}' \ -object '{"qom-type":"memory-backend-ram","id":"ram-node0","size":34359738368}' \ -numa node,nodeid=0,cpus=0-319,memdev=ram-node0 \ -uuid 8d0d1e95-0088-4a5c-a63c-26302a89e4ae \ -smbios type=1,manufacturer=oVirt,product=RHEL,version=8.6-1.el8,serial=61644261-a4f0-e211-a6de-001e67824e05,uuid=8d0d1e95-0088-4a5c-a63c-26302a89e4ae,family=oVirt \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=59,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=2021-11-26T11:03:03,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device intel-iommu,intremap=on,eim=on \ -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \ -device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \ -device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \ -device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \ -device pcie-root-port,port=23,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 \ -device pcie-root-port,port=24,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 \ -device pcie-root-port,port=25,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \ -device pcie-root-port,port=26,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \ -device pcie-root-port,port=27,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \ -device pcie-root-port,port=28,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \ -device pcie-root-port,port=29,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \ -device pcie-root-port,port=30,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 \ -device pcie-root-port,port=31,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 \ -device pcie-root-port,port=32,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 \ -device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 \ -device qemu-xhci,p2=8,p3=8,id=ua-c852836d-45df-4929-a9f3-6eb5f7dbf1ee,bus=pci.7,addr=0x0 \ -device virtio-scsi-pci,iothread=iothread1,id=ua-39753cf7-5906-45c8-b50d-b3dba0eee42a,num_queues=1,bus=pci.5,addr=0x0 \ -device virtio-scsi-pci,iothread=iothread2,id=ua-c7f4258a-5f20-48cf-8402-7cfe611fd091,num_queues=1,bus=pci.6,addr=0x0 \ -device virtio-serial-pci,id=ua-0851a3fc-f8ff-4930-90be-29bf4bbf790f,max_ports=16,bus=pci.4,addr=0x0 \ -device ide-cd,bus=ide.2,id=ua-f7029ec6-a593-4ccd-8141-5501b110dc51,bootindex=2,werror=report,rerror=report \ -blockdev '{"driver":"file","filename":"/rhev/data-center/mnt/glusterSD/gilboa-home-hv1-dev.localdomain:_GFS__2__Data__SSD/e3775152-81d7-487f-813f-d944ec819b65/images/07001b1c-52eb-41e4-bcb1-481f8c4f108b/134afcd9-121f-4d71-9287-b3225a12f4d5","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device scsi-hd,bus=ua-39753cf7-5906-45c8-b50d-b3dba0eee42a.0,channel=0,scsi-id=0,lun=0,device_id=07001b1c-52eb-41e4-bcb1-481f8c4f108b,drive=libvirt-2-format,id=ua-07001b1c-52eb-41e4-bcb1-481f8c4f108b,bootindex=1,write-cache=on,serial=07001b1c-52eb-41e4-bcb1-481f8c4f108b,werror=stop,rerror=stop \ -blockdev '{"driver":"file","filename":"/rhev/data-center/mnt/glusterSD/gilboa-home-hv1-dev.localdomain:_GFS__2__Data__HDD/e6b83b37-216c-41da-a96c-b8598688b4b7/images/7aeb871d-b83d-474a-a5fc-b3c731743f6f/590f87f1-d88f-49ad-a5b1-6beb8879aeb2","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device scsi-hd,bus=ua-c7f4258a-5f20-48cf-8402-7cfe611fd091.0,channel=0,scsi-id=0,lun=0,device_id=7aeb871d-b83d-474a-a5fc-b3c731743f6f,drive=libvirt-1-format,id=ua-7aeb871d-b83d-474a-a5fc-b3c731743f6f,write-cache=on,serial=7aeb871d-b83d-474a-a5fc-b3c731743f6f,werror=stop,rerror=stop \ -netdev tap,fds=64:65:66:67,id=hostua-cac7a538-93fa-477d-8855-d6b8a9a03b65,vhost=on,vhostfds=68:69:70:71 \ -device virtio-net-pci,mq=on,vectors=10,host_mtu=1500,netdev=hostua-cac7a538-93fa-477d-8855-d6b8a9a03b65,id=ua-cac7a538-93fa-477d-8855-d6b8a9a03b65,mac=56:6f:e6:0a:00:03,bus=pci.2,addr=0x0 \ -netdev tap,fds=72:73:74:75,id=hostua-c19e228b-8568-4bbd-8eb5-bc7075a79564,vhost=on,vhostfds=76:77:78:79 \ -device virtio-net-pci,mq=on,vectors=10,host_mtu=1500,netdev=hostua-c19e228b-8568-4bbd-8eb5-bc7075a79564,id=ua-c19e228b-8568-4bbd-8eb5-bc7075a79564,mac=56:6f:e6:0a:00:05,bus=pci.3,addr=0x0 \ -chardev socket,id=charchannel0,fd=80,server=on,wait=off \ -device virtserialport,bus=ua-0851a3fc-f8ff-4930-90be-29bf4bbf790f.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -chardev spicevmc,id=charchannel1,name=vdagent \ -device virtserialport,bus=ua-0851a3fc-f8ff-4930-90be-29bf4bbf790f.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 \ -device usb-tablet,id=input0,bus=ua-c852836d-45df-4929-a9f3-6eb5f7dbf1ee.0,port=1 \ -audiodev id=audio1,driver=none \ -spice port=5921,tls-port=5922,addr=192.168.9.101,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on \ -vnc 192.168.9.101:23,password=on,audiodev=audio1 \ -k en-us \ -device qxl-vga,id=ua-938d7dde-aaaa-4c0b-a5dc-abc4c522de75,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \ -device intel-hda,id=ua-8f5a44b5-98ac-4052-ab14-4552cbcb852c,bus=pci.18,addr=0x1 \ -device hda-duplex,id=ua-8f5a44b5-98ac-4052-ab14-4552cbcb852c-codec0,bus=ua-8f5a44b5-98ac-4052-ab14-4552cbcb852c.0,cad=0,audiodev=audio1 \ -device vfio-pci,host=0000:83:00.0,id=ua-43af8860-beff-4eb6-923c-5249d26c322a,bus=pci.18,addr=0x2 \ -device vfio-pci,host=0000:07:00.0,id=ua-4b800d46-2c67-4d7f-a9b8-9282c2c9de4c,bus=pci.9,addr=0x0 \ -device vfio-pci,host=0000:00:1d.0,id=ua-c7478da1-3499-4c6b-8f4a-aa8d9f45a785,bus=pci.18,addr=0x3 \ -device vfio-pci,host=0000:84:00.1,id=ua-f23ebcb4-c554-4e48-bbb2-164d198026ce,bus=pci.10,addr=0x0 \ -device vfio-pci,host=0000:84:00.0,id=ua-f7021817-39ef-47d7-b162-2af96367cb6a,bus=pci.11,addr=0x0 \ -device virtio-balloon-pci,id=ua-5fb5246e-dfbf-4888-8765-5aa26b90cb99,bus=pci.8,addr=0x0 \ -object '{"qom-type":"rng-random","id":"objua-c9b45e5b-4e53-4ab1-8bf2-4b9b09aa6ff6","filename":"/dev/urandom"}' \ -device virtio-rng-pci,rng=objua-c9b45e5b-4e53-4ab1-8bf2-4b9b09aa6ff6,id=ua-c9b45e5b-4e53-4ab1-8bf2-4b9b09aa6ff6,bus=pci.12,addr=0x0 \ -device vmcoreinfo \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.
your maxcpu is 320, so most likely duplicate of bug 2023313
I assumed as much. That said, if I remove a couple of PCI-E pass-through devices (leaving only the USB or HDMI audio) I can trigger https://bugzilla.redhat.com/show_bug.cgi?id=2023313, getting the caching-mode=on error. However, if I try to pass all the PCI-E devices, I get memory allocation error instead of the caching-mode=on error. Can I somehow reduce the maxcpu value to test if this bug is indeed related to 2023313?
try to reconfigure the vm to make it smaller, just for test perhaps just set to few sockets, 1 core/socket.
Alternatively, you can disable CPU hot-plug with (assuming you're using x86): engine-config -s HotPlugCpuSupported='{"x86":"false","ppc":"true","s390x":"true"}'
(In reply to Arik from comment #5) > Alternatively, you can disable CPU hot-plug with (assuming you're using x86): > engine-config -s > HotPlugCpuSupported='{"x86":"false","ppc":"true","s390x":"true"}' And then restart ovirt-engine
Thanks all. Bug can be closed as duplicate. Reducing the number of vcpus and/or disabling hotplug solves the problem.
BTW, any reason why maxcpus is hard-wired to be 8 times to vcpus count? (max 320 vs 40 vcpus in my case)? Can I change the ratio somehow?
It is not hard-wired to be 8 times, setting the limit is more complex (you can look into https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/common/src/main/java/org/ovirt/engine/core/common/utils/VmCpuCountHelper.java if interested). Somewhat simplified, with UEFI, it's usually the highest power of 2 that fits in the cluster level limit on the maximum number of vCPUs and doesn't cause exceeding the maximum number of vCPU sockets.
(In reply to Milan Zamazal from comment #9) > it's usually the highest power of 2 Sorry, not power of 2, but multiplication of the total number of threads in a single socket. And you can influence it by changing the limits in Engine configuration or by using different vCPU cores/threads configurations.
We can continue with the discussion on devel in case further clarifications are needed. *** This bug has been marked as a duplicate of bug 2023313 ***
After reading at the supplied code, I managed to find some odd combination (more sockets, less core, no threads) that got the maxvcpu well under 256 (only 4*smp_count). Many thanks, again, for the help.
I hit this again on 4.4.10 qemu-kvm-6.1.0-5.module_el8.6.0+1040+0ae94936.x86_64 libvirt-7.10.0-1.module_el8.6.0+1046+bd8eec5e.x86_64 vdsm-4.40.100.2-1.el8.x86_64 Simillar HW (2 NUMAs), doing passtrough a GPU with multiple other devices (GPU is causing this issue) Exactly the same reproduction steps. Same behavior with number of CPUs or switching Q35 - I440FX (lowering CPUs or switching to I440FX will solve this issue) Tried qemu 6.0 without any luck 2022-01-28T09:03:08.866199Z qemu-kvm: -device vfio-pci,host=0000:af:00.1,id=ua-582527a5-a6ea-475a-9c86-816f973ea027,bus=pci.6,addr=0x0: VFIO_MAP_DMA failed: Cannot allocate memory 2022-01-28T09:03:08.912870Z qemu-kvm: -device vfio-pci,host=0000:af:00.1,id=ua-582527a5-a6ea-475a-9c86-816f973ea027,bus=pci.6,addr=0x0: VFIO_MAP_DMA failed: Cannot allocate memory 2022-01-28T09:03:08.912990Z qemu-kvm: -device vfio-pci,host=0000:af:00.1,id=ua-582527a5-a6ea-475a-9c86-816f973ea027,bus=pci.6,addr=0x0: vfio 0000:af:00.1: failed to setup container for group 150: memory listener initialization failed: Region ram-node0: vfio_dma_map(0x5630dd4b9af0, 0x0, 0x80000000, 0x7f5a73e00000) = -12 (Cannot allocate memory) 2022-01-28 09:03:09.005+0000: shutting down, reason=failed LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ HOME=/var/lib/libvirt/qemu/domain-1-Windows-01-GPU0 \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-Windows-01-GPU0/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-Windows-01-GPU0/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-Windows-01-GPU0/.config \ /usr/libexec/qemu-kvm \ -name guest=Windows-01-GPU0,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-Windows-01-GPU0/master-key.aes"}' \ -blockdev '{"driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/d5ba5e0e-339e-40bb-90d8-e34d3d158261.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine pc-q35-rhel8.4.0,usb=off,dump-guest-core=off,kernel_irqchip=split,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \ -accel kvm \ -cpu host,migratable=on,vmx=on,kvm=off \ -m size=33554432k,slots=16,maxmem=134217728k \ -overcommit mem-lock=off \ -smp 16,maxcpus=256,sockets=16,dies=1,cores=8,threads=2 \ -object '{"qom-type":"iothread","id":"iothread1"}' \ -object '{"qom-type":"memory-backend-ram","id":"ram-node0","size":34359738368}' \ -numa node,nodeid=0,cpus=0-255,memdev=ram-node0 \ -uuid d5ba5e0e-339e-40bb-90d8-e34d3d158261 \ -smbios type=1,manufacturer=oVirt,product=RHEL,version=8.6-1.el8,serial=d7cb7a89-958c-2af6-c6b0-fc349767d7db,uuid=d5ba5e0e-339e-40bb-90d8-e34d3d158261,family=oVirt \ -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=39,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=2022-01-28T09:03:00,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device intel-iommu,intremap=on,caching-mode=on,eim=on \ -device pcie-root-port,port=8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -device pcie-root-port,port=10,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \ -device pcie-root-port,port=11,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \ -device pcie-root-port,port=12,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \ -device pcie-root-port,port=13,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \ -device pcie-root-port,port=14,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \ -device pcie-root-port,port=15,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \ -device pcie-root-port,port=16,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x2 \ -device pcie-root-port,port=17,chassis=10,id=pci.10,bus=pcie.0,addr=0x2.0x1 \ -device pcie-root-port,port=18,chassis=11,id=pci.11,bus=pcie.0,addr=0x2.0x2 \ -device pcie-root-port,port=19,chassis=12,id=pci.12,bus=pcie.0,addr=0x2.0x3 \ -device pcie-root-port,port=20,chassis=13,id=pci.13,bus=pcie.0,addr=0x2.0x4 \ -device pcie-root-port,port=21,chassis=14,id=pci.14,bus=pcie.0,addr=0x2.0x5 \ -device pcie-root-port,port=22,chassis=15,id=pci.15,bus=pcie.0,addr=0x2.0x6 \ -device pcie-root-port,port=23,chassis=16,id=pci.16,bus=pcie.0,addr=0x2.0x7 \ -device pcie-root-port,port=24,chassis=17,id=pci.17,bus=pcie.0,addr=0x3 \ -device pcie-pci-bridge,id=pci.18,bus=pci.4,addr=0x0 \ -device qemu-xhci,p2=8,p3=8,id=ua-2d5407e4-258d-4858-a667-7c8d68e6e079,bus=pci.2,addr=0x0 \ -device virtio-scsi-pci,iothread=iothread1,id=ua-c8f1426f-0f5c-4acb-8c65-f90ae8781eff,bus=pci.7,addr=0x0 \ -device virtio-serial-pci,id=ua-c43f87f4-49fd-409a-abb4-e243eaf3ac8e,max_ports=16,bus=pci.3,addr=0x0 \ -device ide-cd,bus=ide.2,id=ua-87b896ca-ac92-4645-bd7d-ef89927a6989,werror=report,rerror=report \ -blockdev '{"driver":"file","filename":"/rhev/data-center/mnt/localhost:_srv_rhv/bd9ac7ab-b2cc-4588-8fb9-da497907378c/images/adcac7da-7c0a-46a9-815a-8455c3327af0/0346ab51-49c8-450e-86e6-7bbc5f86ae99","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device scsi-hd,bus=ua-c8f1426f-0f5c-4acb-8c65-f90ae8781eff.0,channel=0,scsi-id=0,lun=0,device_id=adcac7da-7c0a-46a9-815a-8455c3327af0,drive=libvirt-2-format,id=ua-adcac7da-7c0a-46a9-815a-8455c3327af0,bootindex=1,write-cache=on,serial=adcac7da-7c0a-46a9-815a-8455c3327af0,werror=stop,rerror=stop \ -blockdev '{"driver":"file","filename":"/rhev/data-center/mnt/localhost:_srv_store_rhv/8de83773-d704-424c-b5b3-101a671c8954/images/8e146613-a4b7-41e8-b2f2-e2fbdf5246d7/e3a5b4e8-7781-4212-b98a-9f2c99e17337","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device scsi-hd,bus=ua-c8f1426f-0f5c-4acb-8c65-f90ae8781eff.0,channel=0,scsi-id=0,lun=1,device_id=8e146613-a4b7-41e8-b2f2-e2fbdf5246d7,drive=libvirt-1-format,id=ua-8e146613-a4b7-41e8-b2f2-e2fbdf5246d7,write-cache=on,serial=8e146613-a4b7-41e8-b2f2-e2fbdf5246d7,werror=stop,rerror=stop \ -netdev tap,fds=41:43:44:45,id=hostua-daac2d8b-38cf-4ac0-b48c-b6fedebe2944,vhost=on,vhostfds=46:47:48:49 \ -device virtio-net-pci,mq=on,vectors=10,host_mtu=1500,netdev=hostua-daac2d8b-38cf-4ac0-b48c-b6fedebe2944,id=ua-daac2d8b-38cf-4ac0-b48c-b6fedebe2944,mac=56:6f:ca:5e:00:00,bus=pci.1,addr=0x0 \ -chardev socket,id=charchannel0,fd=50,server=on,wait=off \ -device virtserialport,bus=ua-c43f87f4-49fd-409a-abb4-e243eaf3ac8e.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0 \ -chardev socket,id=charchannel1,fd=51,server=on,wait=off \ -device virtserialport,bus=ua-c43f87f4-49fd-409a-abb4-e243eaf3ac8e.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 \ -audiodev '{"id":"audio1","driver":"none"}' \ -device vfio-pci,host=0000:04:00.0,id=ua-3f3c1a1c-4b88-47b5-9119-f4056a96e93c,bus=pci.5,addr=0x0 \ -device vfio-pci,host=0000:af:00.1,id=ua-582527a5-a6ea-475a-9c86-816f973ea027,bus=pci.6,addr=0x0 \ -device vfio-pci,host=0000:3c:00.0,id=ua-5c49a9c0-8557-447a-9189-97bb3952a062,bus=pci.10,addr=0x0 \ -device vfio-pci,host=0000:af:00.0,id=ua-d3af7b38-0840-425b-8fcd-773e0d7dd03c,bus=pci.8,addr=0x0 \ -device virtio-balloon-pci,id=ua-71e3afb3-6c29-4201-87a3-5ed93fd873b1,bus=pci.9,addr=0x0 \ -object '{"qom-type":"rng-random","id":"objua-e8be8bc4-167f-4223-8628-b218a38c9ead","filename":"/dev/urandom"}' \ -device virtio-rng-pci,rng=objua-e8be8bc4-167f-4223-8628-b218a38c9ead,id=ua-e8be8bc4-167f-4223-8628-b218a38c9ead,bus=pci.11,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2022-01-28T09:03:08.866199Z qemu-kvm: -device vfio-pci,host=0000:af:00.1,id=ua-582527a5-a6ea-475a-9c86-816f973ea027,bus=pci.6,addr=0x0: VFIO_MAP_DMA failed: Cannot allocate memory 2022-01-28T09:03:08.912870Z qemu-kvm: -device vfio-pci,host=0000:af:00.1,id=ua-582527a5-a6ea-475a-9c86-816f973ea027,bus=pci.6,addr=0x0: VFIO_MAP_DMA failed: Cannot allocate memory 2022-01-28T09:03:08.912990Z qemu-kvm: -device vfio-pci,host=0000:af:00.1,id=ua-582527a5-a6ea-475a-9c86-816f973ea027,bus=pci.6,addr=0x0: vfio 0000:af:00.1: failed to setup container for group 150: memory listener initialization failed: Region ram-node0: vfio_dma_map(0x5630dd4b9af0, 0x0, 0x80000000, 0x7f5a73e00000) = -12 (Cannot allocate memory) 2022-01-28 09:03:09.005+0000: shutting down, reason=failed
Note that this time caching-mode=on is set in the QEMU command line in Comment 13, so we are apparently facing a different, although possibly related, issue here than in the supposed duplicate.
Petr, can you please file a separate issue?
Hi Arik, sure, filled a new bug here: https://bugzilla.redhat.com/show_bug.cgi?id=2048429