Bug 2126623

Hi Igor,


Would you please help to take a look this bug?
Is this cause related to cpu hotplug?

Thanks a lot!


Best regards
Nana Liu

(In reply to liunana from comment #1)
> Hi Igor,
> 
> 
> Would you please help to take a look this bug?
> Is this cause related to cpu hotplug?

it could be or it could be something else.

Anyways BZ doesn't have enough info to reproduce,
you shall always provide used complete QEMU command line (all of it).

So far with seabios/q35 it works for me.


> 
> Thanks a lot!
> 
> 
> Best regards
> Nana Liu

(In reply to Igor Mammedov from comment #2)
> (In reply to liunana from comment #1)
> > Hi Igor,
> > 
> > 
> > Would you please help to take a look this bug?
> > Is this cause related to cpu hotplug?
> 
> it could be or it could be something else.
> 
> Anyways BZ doesn't have enough info to reproduce,
> you shall always provide used complete QEMU command line (all of it).
> 
My apologies.

> So far with seabios/q35 it works for me.
> 
I can reproduce this issue both with a OVMF vm and a q35/seabios vm.

Test Env:
Host:
    5.14.0-163.el9.x86_64
    qemu-kvm-7.1.0-1.el9.x86_64
    edk2-ovmf-20220526git16779ede2d36-3.el9.noarch
    Model name:            Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
    dell-per750-05.lab.eng.pek2.redhat.com
Guest: 
    ovmf +  5.14.0-163.el9.x86_64
    q35/seabios + 5.14.0-162.2.1.el9.x86_64


[1] QEMU command line
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
    -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kar/workspace/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel920-64-virtio-scsi_qcow2_filesystem_VARS.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
    -machine q35,kernel-irqchip=split,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device intel-iommu,intremap=on,device-iotlb=on \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 14336 \
    -object memory-backend-ram,size=14336M,id=mem-machine_mem  \
    -smp 1,maxcpus=24,cores=1,threads=1,dies=1,sockets=24 \
    -cpu 'Icelake-Server',hle=off,rtm=off,kvm_pv_unhalt=on \
    -chardev socket,server=on,wait=off,path=/tmp/monitor-qmp,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -device pvpanic,ioport=0x505,id=idKkFI4J \
    -chardev socket,server=on,wait=off,path=/tmp/serial-serial0,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel920-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:6a:9c:ed:24:f7,id=idAduWDJ,netdev=idW7JxBY,bus=pcie-root-port-3,addr=0x0 \
    -netdev tap,id=idW7JxBY,vhost=on \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -monitor stdio \


[2]
1. Can't recover vm with qemu command 'system_reset'. VM still is in panic status.

2. Can't reproduce this issue if I remove the IOMMU device '-device intel-iommu,intremap=on,device-iotlb=on'.

3. When booting ovmf vm with ' -smp 1,maxcpus=12,cores=1,threads=1,dies=1,sockets=12' to reproduce this issue, guest will keep loading at booting phase after rebooting and can't boot up successfully, but without Call Trace messages.

4. Full guest panic log please refer to Additional info --> [Full guest kernel log] in Description.


[3] Hotpulg vcpu scripts, execute command "./loop.sh 23"  (./loop.sh $number_vcpus)
# cat loop.sh 
#!/bin/sh
declare -i sockets=$1
echo -e "\nGenerate topolopy for vcpu hotplug $1 sockets! \n"
declare -i a=1

for i in `seq 0 $sockets`;do 
	if [ "$a" -eq "$sockets" ];then
                echo -e '\n{ "execute": "qmp_capabilities" }{ "execute": "device_add","arguments":{"driver":"Icelake-Server-x86_64-cpu","core-id": 0, "thread-id": 0, "socket-id": '$a', "die-id": 0,"id":"cpu'$a'"}}'
                echo '{ "execute": "qmp_capabilities" }{ "execute": "device_add","arguments":{"driver":"Icelake-Server-x86_64-cpu","core-id": 0, "thread-id": 0, "socket-id": '$a', "die-id": 0,"id":"cpu'$a'"}}' | nc -U /tmp/monitor-qmp
		exit 0
	fi

        echo -e "\nhotplug vcpu id : cpu$a"
        echo -e '{ "execute": "device_add","arguments":{"driver":"Icelake-Server-x86_64-cpu","core-id": 0, "thread-id": 0, "socket-id": '$a', "die-id": 0,"id":"cpu'$a'"}}'
        echo '{ "execute": "qmp_capabilities" }{ "execute": "device_add","arguments":{"driver":"Icelake-Server-x86_64-cpu","core-id": 0, "thread-id": 0, "socket-id": '$a', "die-id": 0,"id":"cpu'$a'"}}' | nc -U /tmp/monitor-qmp
	sleep 3
	a=$a+1
done
echo "topolopy generate done"


Could you please help to check this? 
Any unclear information please let me know, thanks for your time.


Best regards
Nana Liu

I sort of able to reproduce hang at boot time, simplified CLI
        -enable-kvm \
        -nodefaults \
	-m 4G \
	-M q35,kernel-irqchip=split \
       	-cpu Nehalem,hle=off,rtm=off,kvm_pv_unhalt=on \
       	-smp 9 \
       	-device intel-iommu,intremap=on \
	-serial file:/tmp/serial-serial0 \
	-drive if=virtio,file=rhel9.2.qcow2

i.e. it doesn't depend on CPU hotplug nor CPU model,
it seems that irq remapping is broken when guest is booted
with more than 8 vCPUs.

It's typically stuck at storage initialization or accessing it.

Host kernel:
5.14.0-162.el9.x86_64
guest kernel:
5.14.0-162.el9.x86_64

The same happens with RHEL8 host kernel:
4.18.0-415.el8.x86_64

On QEMU side (upstream) it bisects to:
commit 77250171bdc02aee1 (intel_iommu: Fix irqchip / X2APIC configuration checks)

It's IOMMU stuff and I'm not sure if it's kernel or QEMU to blame (CCing IOMMU folks),
Peter could you take over this BZ?

(In reply to Igor Mammedov from comment #4)
> On QEMU side (upstream) it bisects to:
> commit 77250171bdc02aee1 (intel_iommu: Fix irqchip / X2APIC configuration
> checks)
> 
> It's IOMMU stuff and I'm not sure if it's kernel or QEMU to blame (CCing
> IOMMU folks),
> Peter could you take over this BZ?

Thanks for digging into it, Igor.

I think 77250171bdc02aee1 is probably wrong in that it skipped calling kvm_enable_x2apic() even if intr_eim==ON, so no matter whether n_vcpus>255 we need to enable x2apic in KVM, afaict.  It means we'll declare EIM capability everywhere (e.g. in VT-d ECAP register) but KVM didn't really enable EIM at all.  It doesn't really sound correct.

For example, if we attach eim=off to the intel-iommu cmdline it'll start to work for 9-255 vcpus again for me.

I'll post a patch shortly to partly revert the patch.

https://lore.kernel.org/qemu-devel/20220921161227.57259-1-peterx@redhat.com/T/#u

*** This bug has been marked as a duplicate of bug 2126095 ***