Hide Forgot
Description of problem: qemu-kvm with vfio Killed when boot with "numactl -m 1" Version-Release number of selected component (if applicable): qemu-kvm-1.5.3-19.el7.x86_64 kernel-3.10.0-50.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. boot guest with numactl numactl \ -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu \ -name 'virt-tests-vm1' \ -sandbox on \ -M pc \ -nodefaults \ -vga cirrus \ -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20131119-170324-eboEDTcp,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20131119-170324-eboEDTcp,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20131119-170324-eboEDTcp,path=/tmp/seabios-20131119-170324-eboEDTcp,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20131119-170324-eboEDTcp,iobase=0x402 \ -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \ -drive id=drive_image1,if=none,cache=none,snapshot=on,aio=native,file=/root/staf-kvm-devel/autotest/client/tests/virt/shared/data/images/RHEL-Server-6.5-64-virtio.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04 \ -device vfio-pci,host=0000:05:00.0,id=id_0000.05.00.0,bus=pci.0,addr=05 \ -device vfio-pci,host=0000:05:00.1,id=id_0000.05.00.1,bus=pci.0,addr=06 \ -m 4096 \ -smp 8,maxcpus=8,cores=4,threads=1,sockets=2 \ -cpu 'Nehalem' \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off \ -no-kvm-pit-reinjection \ -enable-kvm \ -monitor stdio 2. 3. Actual results: (qemu) Killed Expected results: guest boot up successful Additional info: processor : 15 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping : 5 microcode : 0x19 cpu MHz : 2393.991 cache size : 8192 KB physical id : 0 siblings : 8 core id : 3 cpu cores : 4 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid bogomips : 4787.23 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-50.el7.x86_64 root=/dev/mapper/rhel_intel--e5530--8--2-root ro console=tty0 vconsole.keymap=us crashkernel=auto reboot=pci console=ttyS0,115200 rd.lvm.lv=rhel_intel-e5530-8-2/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=rhel_intel-e5530-8-2/root biosdevname=0 crashkernel=256M rhgb quiet intel_iommu=on # numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 2 4 6 8 10 12 14 node 0 size: 4096 MB node 0 free: 3880 MB node 1 cpus: 1 3 5 7 9 11 13 15 node 1 size: 4083 MB node 1 free: 3622 MB node distances: node 0 1 0: 10 20 1: 20 10 trace the qemu-kvm: # strace numactl -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu -name 'virt-tests-vm1' -sandbox off -M pc -nodefaults -vga cirrus -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20131119-170324-eboEDTcp,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20131119-170324-eboEDTcp,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20131119-170324-eboEDTcp,path=/tmp/seabios-20131119-170324-eboEDTcp,server,nowait -device isa-debugcon,chardev=seabioslog_id_20131119-170324-eboEDTcp,iobase=0x402 -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 -drive id=drive_image1,if=none,cache=none,snapshot=on,aio=native,file=/root/staf-kvm-devel/autotest/client/tests/virt/shared/data/images/RHEL-Server-6.5-64-virtio.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04 -device vfio-pci,host=0000:05:00.0,id=id_0000.05.00.0,bus=pci.0,addr=05 -device vfio-pci,host=0000:05:00.1,id=id_0000.05.00.1,bus=pci.0,addr=06 -m 4096 -smp 8,maxcpus=8,cores=4,threads=1,sockets=2 -cpu 'Nehalem' -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off -no-kvm-pit-reinjection -enable-kvm -monitor stdio ... ioctl(23, VFIO_GET_API_VERSION, 0x17) = 0 ioctl(23, VFIO_CHECK_EXTENSION, 0x1) = 1 ioctl(22, VFIO_GROUP_SET_CONTAINER, 0x7fff988bf2a0) = 0 ioctl(23, VFIO_SET_IOMMU, 0x1) = 0 ioctl(23, VFIO_IOMMU_MAP_DMA, 0x7fff988bf110) = 0 ioctl(23, VFIO_IOMMU_MAP_DMA, 0x7fff988bf110) = 0 ioctl(23, VFIO_IOMMU_MAP_DMA, 0x7fff988bf110) = 0 ioctl(23, VFIO_IOMMU_MAP_DMA <unfinished ...> +++ killed by SIGKILL +++ Killed
Created attachment 827004 [details] strace debug file
(In reply to ShupingCui from comment #0) > Steps to Reproduce: > 1. boot guest with numactl > numactl \ > -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu \ ... > -m 4096 \ ... > # numactl --hardware > available: 2 nodes (0-1) > node 0 cpus: 0 2 4 6 8 10 12 14 > node 0 size: 4096 MB > node 0 free: 3880 MB > node 1 cpus: 1 3 5 7 9 11 13 15 > node 1 size: 4083 MB > node 1 free: 3622 MB ^^^^ From numactl man page: --membind=nodes, -m nodes Only allocate memory from nodes. Allocation will fail when there is not enough memory available on these nodes. nodes may be specified as noted above. So you're trying to start a 4G VM, forcing memory to be allocated from a node that does not have 4G available and the allocation fails as the numactl man page says it will... sounds like notabug. Re-open if there's something I'm missing. I've tested using membind within the size available on my host and it works fine.
(In reply to Alex Williamson from comment #3) > (In reply to ShupingCui from comment #0) > > Steps to Reproduce: > > 1. boot guest with numactl > > numactl \ > > -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu \ > ... > > -m 4096 \ > ... > > # numactl --hardware > > available: 2 nodes (0-1) > > node 0 cpus: 0 2 4 6 8 10 12 14 > > node 0 size: 4096 MB > > node 0 free: 3880 MB > > node 1 cpus: 1 3 5 7 9 11 13 15 > > node 1 size: 4083 MB > > node 1 free: 3622 MB > ^^^^ > > From numactl man page: > > --membind=nodes, -m nodes > Only allocate memory from nodes. Allocation will fail > when > there is not enough memory available on these nodes. nodes > may > be specified as noted above. > > So you're trying to start a 4G VM, forcing memory to be allocated from a > node that does not have 4G available and the allocation fails as the numactl > man page says it will... sounds like notabug. Re-open if there's something > I'm missing. I've tested using membind within the size available on my host > and it works fine. scui, pls retry w/ lower ram, and update the result here, thx.
(In reply to Xiaoqing Wei from comment #4) > (In reply to Alex Williamson from comment #3) > > (In reply to ShupingCui from comment #0) > > > Steps to Reproduce: > > > 1. boot guest with numactl > > > numactl \ > > > -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu \ > > ... > > > -m 4096 \ > > ... > > > # numactl --hardware > > > available: 2 nodes (0-1) > > > node 0 cpus: 0 2 4 6 8 10 12 14 > > > node 0 size: 4096 MB > > > node 0 free: 3880 MB > > > node 1 cpus: 1 3 5 7 9 11 13 15 > > > node 1 size: 4083 MB > > > node 1 free: 3622 MB > > ^^^^ > > > > From numactl man page: > > > > --membind=nodes, -m nodes > > Only allocate memory from nodes. Allocation will fail > > when > > there is not enough memory available on these nodes. nodes > > may > > be specified as noted above. > > > > So you're trying to start a 4G VM, forcing memory to be allocated from a > > node that does not have 4G available and the allocation fails as the numactl > > man page says it will... sounds like notabug. Re-open if there's something > > I'm missing. I've tested using membind within the size available on my host > > and it works fine. > > scui, pls retry w/ lower ram, and update the result here, thx. tried with "-m 2048", no killed. Thanks, Shuping