Bug 1942820
Summary: | SMBIOS 2.1 table limits error when booting guest with 710 vcpus and 250G memory | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 9 | Reporter: | Yanhui Ma <yama> |
Component: | qemu-kvm | Assignee: | Julia Suvorova <jusual> |
qemu-kvm sub component: | Devices | QA Contact: | Yiqian Wei <yiwei> |
Status: | CLOSED MIGRATED | Docs Contact: | |
Severity: | medium | ||
Priority: | medium | CC: | ailan, chayang, coli, imammedo, jinzhao, jusual, juzhang, leidwang, nilal, virt-maint, yiwei |
Version: | unspecified | Keywords: | MigratedToJIRA, Reopened, TestOnly, Triaged |
Target Milestone: | rc | ||
Target Release: | 9.3 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2023-09-22 17:50:50 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1904267, 1906077 | ||
Bug Blocks: |
Description
Yanhui Ma
2021-03-25 05:08:14 UTC
Tested this case on my host.Results are as follows. Env: qemu-kvm-5.2.0-11.module+el8.4.0+10268+62bcbbed.x86_64 kernel-4.18.0-301.el8.x86_64 qemu command line: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35,kernel-irqchip=split \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 30720 \ -smp N \ -device intel-iommu,intremap=on,eim=on \ -cpu 'Haswell-noTSX',+kvm_pv_unhalt \ -debugcon file:/home/kdump.log -global isa-debugcon.iobase=0x402 \ -chardev socket,server,id=qmp_id_qmpmonitor1,path=/tmp/avocado_e7k47npg/monitor-qmpmonitor1-20210120-212426-JZ06lDY8,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,server,id=qmp_id_catch_monitor,path=/tmp/avocado_e7k47npg/monitor-catch_monitor-20210120-212426-JZ06lDY8,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,id=pvpanic0,ioport=0x0505 -no-shutdown \ -chardev socket,server,id=chardev_serial0,path=/tmp/avocado_e7k47npg/serial-serial0-20210120-212426-JZ06lDY8,nowait \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20210120-212426-JZ06lDY8,path=/tmp/avocado_e7k47npg/seabios-20210120-212426-JZ06lDY8,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20210120-212426-JZ06lDY8,iobase=0x402 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/rhel840-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:ec:0b:00:3a:88,id=idIe1NbO,netdev=idSVIekk,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idSVIekk,vhost=on \ -qmp tcp:127.0.0.1:4445,server,nowait \ -vnc :0 \ -monitor stdio \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=on,strict=on \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 1. smp=10, vm works well. 2. smp=355 # sh 710cpu.sh QEMU 5.2.0 monitor - type 'help' for more information (qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24) virtio-scsi: Failed to set host notifier (-24) qemu-kvm: ../hw/scsi/virtio-scsi-dataplane.c:59: virtio_scsi_data_plane_handle_cmd: Assertion `s->ctx && s->dataplane_started' failed. 3.smp=710 # sh 710cpu.sh QEMU 5.2.0 monitor - type 'help' for more information (qemu) qemu-kvm: warning: ACPI table size 91896 exceeds 65536 bytes, migration may not work Try removing CPUs, NUMA nodes, memory slots or PCI bridges.qemu-kvm: warning: ACPI table size 92012 exceeds 65536 bytes, migration may not work Try removing CPUs, NUMA nodes, memory slots or PCI bridges.qemu-kvm: virtio-scsi: Failed to set guest notifiers (-24), ensure -accel kvm is set. qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower). 4.smp=711 # sh 710cpu.sh qemu-kvm: Invalid SMP CPUs 711. The max CPUs supported by machine 'pc-q35-rhel8.4.0' is 710 (In reply to leidwang from comment #1) > Tested this case on my host.Results are as follows. > > Env: > qemu-kvm-5.2.0-11.module+el8.4.0+10268+62bcbbed.x86_64 > kernel-4.18.0-301.el8.x86_64 > > qemu command line: > /usr/libexec/qemu-kvm \ > -name 'avocado-vt-vm1' \ > -sandbox on \ > -machine q35,kernel-irqchip=split \ > -device > pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1, > chassis=1 \ > -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ > -nodefaults \ > -device VGA,bus=pcie.0,addr=0x2 \ > -m 30720 \ > -smp N \ > -device intel-iommu,intremap=on,eim=on \ > -cpu 'Haswell-noTSX',+kvm_pv_unhalt \ > -debugcon file:/home/kdump.log -global isa-debugcon.iobase=0x402 \ > -chardev > socket,server,id=qmp_id_qmpmonitor1,path=/tmp/avocado_e7k47npg/monitor- > qmpmonitor1-20210120-212426-JZ06lDY8,nowait \ > -mon chardev=qmp_id_qmpmonitor1,mode=control \ > -chardev > socket,server,id=qmp_id_catch_monitor,path=/tmp/avocado_e7k47npg/monitor- > catch_monitor-20210120-212426-JZ06lDY8,nowait \ > -mon chardev=qmp_id_catch_monitor,mode=control \ > -device pvpanic,id=pvpanic0,ioport=0x0505 -no-shutdown \ > -chardev > socket,server,id=chardev_serial0,path=/tmp/avocado_e7k47npg/serial-serial0- > 20210120-212426-JZ06lDY8,nowait \ > -device isa-serial,id=serial0,chardev=chardev_serial0 \ > -chardev > socket,id=seabioslog_id_20210120-212426-JZ06lDY8,path=/tmp/avocado_e7k47npg/ > seabios-20210120-212426-JZ06lDY8,server,nowait \ > -device > isa-debugcon,chardev=seabioslog_id_20210120-212426-JZ06lDY8,iobase=0x402 \ > -device > pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0, > chassis=2 \ > -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ > -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ > -device > pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0, > chassis=3 \ > -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ > -blockdev > node-name=file_image1,driver=file,auto-read-only=on,discard=unmap, > aio=threads,filename=/home/rhel840-64-virtio-scsi.qcow2,cache.direct=on, > cache.no-flush=off \ > -blockdev > node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no- > flush=off,file=file_image1 \ > -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ > -device > pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0, > chassis=4 \ > -device > virtio-net-pci,mac=9a:ec:0b:00:3a:88,id=idIe1NbO,netdev=idSVIekk,bus=pcie- > root-port-3,addr=0x0 \ > -netdev tap,id=idSVIekk,vhost=on \ > -qmp tcp:127.0.0.1:4445,server,nowait \ > -vnc :0 \ > -monitor stdio \ > -rtc base=utc,clock=host,driftfix=slew \ > -boot menu=on,strict=on \ > -enable-kvm \ > -device > pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0, > addr=0x3,chassis=5 > > 1. smp=10, vm works well. > > 2. smp=355 > > # sh 710cpu.sh > QEMU 5.2.0 monitor - type 'help' for more information > (qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event > notifier: Too many open files (-24) > virtio-scsi: Failed to set host notifier (-24) > qemu-kvm: ../hw/scsi/virtio-scsi-dataplane.c:59: > virtio_scsi_data_plane_handle_cmd: Assertion `s->ctx && > s->dataplane_started' failed. > You need to increase the file descriptor limit on host. vim /etc/security/limits.conf, add "* soft nofile 8192", reboot host, then check it via # ulimit -n > 3.smp=710 > > # sh 710cpu.sh > QEMU 5.2.0 monitor - type 'help' for more information > (qemu) qemu-kvm: warning: ACPI table size 91896 exceeds 65536 bytes, > migration may not work > Try removing CPUs, NUMA nodes, memory slots or PCI bridges.qemu-kvm: > warning: ACPI table size 92012 exceeds 65536 bytes, migration may not work > Try removing CPUs, NUMA nodes, memory slots or PCI bridges.qemu-kvm: > virtio-scsi: Failed to set guest notifiers (-24), ensure -accel kvm is set. > qemu-kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower). > > 4.smp=711 > > # sh 710cpu.sh > qemu-kvm: Invalid SMP CPUs 711. The max CPUs supported by machine > 'pc-q35-rhel8.4.0' is 710 Although we will support SMBIOS3.0 Bug 1904267 - Q35: Support SMBIOS 3.0 Entry Point Type, I still open the bug to track whether the issue exists for SMBIOS3.0. I reproduced the issue with following qemu cmd line and packages on host with 448 cpus and 500G memory(lenovo-sr950-02.khw2.lab.eng.bos.redhat.com): kernel-4.18.0-298.el8.x86_64 qemu-kvm-5.2.0-14.module+el8.4.0+10425+ad586fa5.x86_64 # ./cmd QEMU 5.2.0 monitor - type 'help' for more information (qemu) 2021-03-26T05:35:09.364474Z qemu-kvm: warning: ACPI table size 91708 exceeds 65536 bytes, migration may not work Try removing CPUs, NUMA nodes, memory slots or PCI bridges.2021-03-26T05:35:09.390783Z qemu-kvm: SMBIOS 2.1 table length 65599 exceeds 65535 # cat cmd /usr/libexec/qemu-kvm \ -name guest=710_vcpu,debug-threads=on \ -S \ -machine pc-q35-rhel8.4.0,accel=kvm,usb=off,dump-guest-core=off,kernel_irqchip=split,memory-backend=pc.ram \ -cpu host,migratable=on \ -device VGA,bus=pcie.0,addr=0x2 \ -m 256000 \ -object memory-backend-ram,id=pc.ram,size=268435456000 \ -overcommit mem-lock=off \ -smp 710,sockets=710,cores=1,threads=1 \ -uuid 335a662b-6ae7-483a-9fd2-199db3c85d36 \ -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,path=/var/tmp/monitor-qmpmonitor1-20201130-083617-UdoMuUZg,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device intel-iommu,intremap=on,eim=on \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \ -device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \ -device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \ -device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \ -device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \ -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \ -blockdev '{"driver":"file","filename":"/home/rhel840-64-virtio-scsi.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \ -device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk0,bootindex=1 \ -chardev socket,id=charchannel0,path=/var/tmp/serial-serial0-20201130-083617-UdoMuUZg,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \ -object rng-random,id=objrng0,filename=/dev/urandom \ -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.6,addr=0x0 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on -monitor stdio -vnc :0 Hello Eduardo, Could you please tell me whether there is a more easier reproducer to trigger the limit error? I mean I don't need to borrow machine with 250G memory and boot guest with 250G, just need to mimic the behaviour. Thank you. Assigned to Amnon for initial triage per bz process and age of bug created or assigned to virt-maint without triage. Changed from General to Devices... From the referenced bug https://bugzilla.redhat.com/show_bug.cgi?id=1840923#c30 there is an error and comment from Eduardo: > # ./cmd_cpu > QEMU 5.2.0 monitor - type 'help' for more information > (qemu) qemu-kvm: warning: ACPI table size 65958 exceeds 65536 bytes, > migration may not work > Try removing CPUs, NUMA nodes, memory slots or PCI bridges.qemu-kvm: > warning: ACPI table size 66074 exceeds 65536 bytes, migration may not work > Try removing CPUs, NUMA nodes, memory slots or PCI bridges. Thanks for the report. Let's track this warning in a separate BZ, as this is a kernel/KVM BZ, and ACPI table limits are a qemu-kvm issue. I see the error report would appear to come from: commit 451b157041d265f94a4be668b9cd234b3e34fabb Author: Yubo Miao <miaoyubo> Date: Thu Nov 19 09:48:38 2020 +0800 acpi: Align the size to 128k If table size is changed between virt_acpi_build and virt_acpi_build_update, the table size would not be updated to UEFI, therefore, just align the size to 128kb, which is enough and same with x86. It would warn if 64k is not enough and the align size should be updated. Signed-off-by: Yubo Miao <miaoyubo> Signed-off-by: Jiahui Cen <cenjiahui> Message-Id: <20201119014841.7298-7-cenjiahui> Reviewed-by: Michael S. Tsirkin <mst> Signed-off-by: Michael S. Tsirkin <mst> (In reply to Yanhui Ma from comment #5) > Hello Eduardo, > > Could you please tell me whether there is a more easier reproducer to > trigger the limit error? I mean I don't need to borrow machine with 250G > memory and boot guest with 250G, just need to mimic the behaviour. Thank you. It should be possible to create VMs with a huge amount of CPUs and RAM on smaller hosts if you have enough swap space available. The system may become very slow, but it might be enough for checking if SMBIOS limits aren't being hit. You can also trigger the SMBIOS limit error by adding custom SMBIOS tables using -smbios. That would be enough to check if SMBIOS 3.0 is really increasing the table size limit, but not sufficient to mark this BZ as verified. (In reply to Eduardo Habkost from comment #7) > (In reply to Yanhui Ma from comment #5) > > Hello Eduardo, > > > > Could you please tell me whether there is a more easier reproducer to > > trigger the limit error? I mean I don't need to borrow machine with 250G > > memory and boot guest with 250G, just need to mimic the behaviour. Thank you. > > It should be possible to create VMs with a huge amount of CPUs and RAM on > smaller hosts if you have enough swap space available. The system may > become very slow, but it might be enough for checking if SMBIOS limits > aren't being hit. > Yes, the SMBIOS limits can be hit via enlarging swap space. Before enlarging swap # free -h total used free shared buff/cache available Mem: 62Gi 481Mi 61Gi 12Mi 286Mi 61Gi Swap: 31Gi 0B 31Gi Enlarging swap via following steps: # dd if=/dev/zero of=/home/swapfile bs=1G count=220 220+0 records in 220+0 records out 236223201280 bytes (236 GB, 220 GiB) copied, 1110.56 s, 213 MB/s # mkswap /home/swapfile mkswap: /home/swapfile: insecure permissions 0644, 0600 suggested. Setting up swapspace version 1, size = 220 GiB (236223197184 bytes) no label, UUID=b322da5d-febc-46b6-879b-fea9b2bfc52a # swapon /home/swapfile swapon: /home/swapfile: insecure permissions 0644, 0600 suggested. After enlarging swap # free -h total used free shared buff/cache available Mem: 62Gi 799Mi 1.2Gi 155Mi 60Gi 60Gi Swap: 251Gi 1.0Mi 251Gi Then guest with 710 vcpus and 250G memory can be booted, and there will be SMBIOS limits error. > You can also trigger the SMBIOS limit error by adding custom SMBIOS tables > using -smbios. That would be enough to check if SMBIOS 3.0 is really > increasing the table size limit, but not sufficient to mark this BZ as > verified. (In reply to Yanhui Ma from comment #8) > Then guest with 710 vcpus and 250G memory can be booted, and there will be > SMBIOS limits error. Do you mean QEMU runs but exits with the SMBIOS limit error, or that the guest really boots? (In reply to Eduardo Habkost from comment #9) > (In reply to Yanhui Ma from comment #8) > > Then guest with 710 vcpus and 250G memory can be booted, and there will be > > SMBIOS limits error. > > Do you mean QEMU runs but exits with the SMBIOS limit error, or that the > guest really boots? I mean QEMU runs but exits with the SMBIOS limit error. Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release. Reassigning to Nitesh. Note that this is a TestOnly BZ that can be moved to ON_QA as soon as bug 1904267 is on ON_QA. Removing the TestOnly keyword here as it seems there is a dev work required. Julia, please correct me if that is not the case. Thanks After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. Re-opening the BZ as Julia is working on it. Reading upstream: https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg01640.html posting and it seems this bug would be fixed there as well (e.g. bug 2091166) Setting ITR=9.2 - let's try to get this into qemu-7.2 for RHEL 9.2 via rebase. The patch set mentioned above is merged, but it will not fix the issue. It will be probably fixed with bug 1906077 Based on Julia's comment above making this TestOnly. Removing this bug from the current release - the depends on bug was already removed and thus the expectation is work will be planned for a future release. Julia, what the status of this BZ? Have it been fixed upstream already? After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. Reopening this as the dependent bug 1906077 state has changed. Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |