Bug 1032874

Summary: qemu-kvm with vfio Killed when boot with "numactl -m 1"
Product: Red Hat Enterprise Linux 7 Reporter: ShupingCui <scui>
Component: qemu-kvmAssignee: Alex Williamson <alex.williamson>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: acathrow, bdas, chayang, hhuang, juzhang, michen, scui, shuang, virt-maint, xuhan, xwei
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-22 19:31:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
strace debug file none

Description ShupingCui 2013-11-21 05:49:46 UTC
Description of problem:
qemu-kvm with vfio Killed when boot with "numactl -m 1"

Version-Release number of selected component (if applicable):
qemu-kvm-1.5.3-19.el7.x86_64
kernel-3.10.0-50.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. boot guest with numactl
numactl \
    -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu \
    -name 'virt-tests-vm1'  \
    -sandbox on \
    -M pc  \
    -nodefaults  \
    -vga cirrus  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20131119-170324-eboEDTcp,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20131119-170324-eboEDTcp,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20131119-170324-eboEDTcp,path=/tmp/seabios-20131119-170324-eboEDTcp,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20131119-170324-eboEDTcp,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \
    -drive id=drive_image1,if=none,cache=none,snapshot=on,aio=native,file=/root/staf-kvm-devel/autotest/client/tests/virt/shared/data/images/RHEL-Server-6.5-64-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04 \
    -device vfio-pci,host=0000:05:00.0,id=id_0000.05.00.0,bus=pci.0,addr=05 \
    -device vfio-pci,host=0000:05:00.1,id=id_0000.05.00.1,bus=pci.0,addr=06  \
    -m 4096  \
    -smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
    -cpu 'Nehalem' \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off  \
    -no-kvm-pit-reinjection \
    -enable-kvm \
    -monitor stdio

2.
3.

Actual results:
(qemu) Killed

Expected results:
guest boot up successful

Additional info:
processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           E5530  @ 2.40GHz
stepping	: 5
microcode	: 0x19
cpu MHz		: 2393.991
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 4787.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-50.el7.x86_64 root=/dev/mapper/rhel_intel--e5530--8--2-root ro console=tty0 vconsole.keymap=us crashkernel=auto reboot=pci console=ttyS0,115200 rd.lvm.lv=rhel_intel-e5530-8-2/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=rhel_intel-e5530-8-2/root biosdevname=0 crashkernel=256M rhgb quiet intel_iommu=on

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14
node 0 size: 4096 MB
node 0 free: 3880 MB
node 1 cpus: 1 3 5 7 9 11 13 15
node 1 size: 4083 MB
node 1 free: 3622 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

trace the qemu-kvm:
# strace numactl -m 1  /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu     -name 'virt-tests-vm1'      -sandbox off     -M pc      -nodefaults      -vga cirrus      -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20131119-170324-eboEDTcp,server,nowait     -mon chardev=qmp_id_qmpmonitor1,mode=control      -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20131119-170324-eboEDTcp,server,nowait     -device isa-serial,chardev=serial_id_serial0      -chardev socket,id=seabioslog_id_20131119-170324-eboEDTcp,path=/tmp/seabios-20131119-170324-eboEDTcp,server,nowait     -device isa-debugcon,chardev=seabioslog_id_20131119-170324-eboEDTcp,iobase=0x402     -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03     -drive id=drive_image1,if=none,cache=none,snapshot=on,aio=native,file=/root/staf-kvm-devel/autotest/client/tests/virt/shared/data/images/RHEL-Server-6.5-64-virtio.qcow2     -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04     -device vfio-pci,host=0000:05:00.0,id=id_0000.05.00.0,bus=pci.0,addr=05     -device vfio-pci,host=0000:05:00.1,id=id_0000.05.00.1,bus=pci.0,addr=06      -m 4096      -smp 8,maxcpus=8,cores=4,threads=1,sockets=2      -cpu 'Nehalem'     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1      -vnc :0      -rtc base=utc,clock=host,driftfix=slew      -boot order=cdn,once=c,menu=off      -no-kvm-pit-reinjection     -enable-kvm     -monitor stdio

...
ioctl(23, VFIO_GET_API_VERSION, 0x17)   = 0
ioctl(23, VFIO_CHECK_EXTENSION, 0x1)    = 1
ioctl(22, VFIO_GROUP_SET_CONTAINER, 0x7fff988bf2a0) = 0
ioctl(23, VFIO_SET_IOMMU, 0x1)          = 0
ioctl(23, VFIO_IOMMU_MAP_DMA, 0x7fff988bf110) = 0
ioctl(23, VFIO_IOMMU_MAP_DMA, 0x7fff988bf110) = 0
ioctl(23, VFIO_IOMMU_MAP_DMA, 0x7fff988bf110) = 0
ioctl(23, VFIO_IOMMU_MAP_DMA <unfinished ...>
+++ killed by SIGKILL +++
Killed

Comment 1 ShupingCui 2013-11-21 05:50:36 UTC
Created attachment 827004 [details]
strace debug file

Comment 3 Alex Williamson 2013-11-22 19:31:21 UTC
(In reply to ShupingCui from comment #0)
> Steps to Reproduce:
> 1. boot guest with numactl
> numactl \
>     -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu \
...
>     -m 4096  \
...
> # numactl --hardware
> available: 2 nodes (0-1)
> node 0 cpus: 0 2 4 6 8 10 12 14
> node 0 size: 4096 MB
> node 0 free: 3880 MB
> node 1 cpus: 1 3 5 7 9 11 13 15
> node 1 size: 4083 MB
> node 1 free: 3622 MB
               ^^^^

From numactl man page:

       --membind=nodes, -m nodes
              Only allocate memory from  nodes.   Allocation  will  fail  when
              there  is not enough memory available on these nodes.  nodes may
              be specified as noted above.

So you're trying to start a 4G VM, forcing memory to be allocated from a node that does not have 4G available and the allocation fails as the numactl man page says it will... sounds like notabug.  Re-open if there's something I'm missing.  I've tested using membind within the size available on my host and it works fine.

Comment 4 Xiaoqing Wei 2013-11-27 08:48:57 UTC
(In reply to Alex Williamson from comment #3)
> (In reply to ShupingCui from comment #0)
> > Steps to Reproduce:
> > 1. boot guest with numactl
> > numactl \
> >     -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu \
> ...
> >     -m 4096  \
> ...
> > # numactl --hardware
> > available: 2 nodes (0-1)
> > node 0 cpus: 0 2 4 6 8 10 12 14
> > node 0 size: 4096 MB
> > node 0 free: 3880 MB
> > node 1 cpus: 1 3 5 7 9 11 13 15
> > node 1 size: 4083 MB
> > node 1 free: 3622 MB
>                ^^^^
> 
> From numactl man page:
> 
>        --membind=nodes, -m nodes
>               Only allocate memory from  nodes.   Allocation  will  fail 
> when
>               there  is not enough memory available on these nodes.  nodes
> may
>               be specified as noted above.
> 
> So you're trying to start a 4G VM, forcing memory to be allocated from a
> node that does not have 4G available and the allocation fails as the numactl
> man page says it will... sounds like notabug.  Re-open if there's something
> I'm missing.  I've tested using membind within the size available on my host
> and it works fine.

scui, pls retry w/ lower ram, and update the result here, thx.

Comment 5 ShupingCui 2013-11-27 08:54:07 UTC
(In reply to Xiaoqing Wei from comment #4)
> (In reply to Alex Williamson from comment #3)
> > (In reply to ShupingCui from comment #0)
> > > Steps to Reproduce:
> > > 1. boot guest with numactl
> > > numactl \
> > >     -m 1 /root/staf-kvm-devel/autotest/client/tests/virt/qemu/qemu \
> > ...
> > >     -m 4096  \
> > ...
> > > # numactl --hardware
> > > available: 2 nodes (0-1)
> > > node 0 cpus: 0 2 4 6 8 10 12 14
> > > node 0 size: 4096 MB
> > > node 0 free: 3880 MB
> > > node 1 cpus: 1 3 5 7 9 11 13 15
> > > node 1 size: 4083 MB
> > > node 1 free: 3622 MB
> >                ^^^^
> > 
> > From numactl man page:
> > 
> >        --membind=nodes, -m nodes
> >               Only allocate memory from  nodes.   Allocation  will  fail 
> > when
> >               there  is not enough memory available on these nodes.  nodes
> > may
> >               be specified as noted above.
> > 
> > So you're trying to start a 4G VM, forcing memory to be allocated from a
> > node that does not have 4G available and the allocation fails as the numactl
> > man page says it will... sounds like notabug.  Re-open if there's something
> > I'm missing.  I've tested using membind within the size available on my host
> > and it works fine.
> 
> scui, pls retry w/ lower ram, and update the result here, thx.

tried with "-m 2048", no killed.

Thanks,
Shuping