Description of problem: A VM which is configured to run in a specific numa node isn't able to start any more, e.g.: <numatune> <memory mode='strict' nodeset='3'/> </numatune> The startup results in the following message: kvm_create_vcpu: Cannot allocate memory Failed to create vCPU. Check the -smp parameter. Version-Release number of selected component (if applicable): qemu-kvm-0.12.1.2-2.479.el6.x86_64 libvirt-0.10.2-54.el6.x86_64 How reproducible: Always Steps to Reproduce: 1. Have a system with at least 2 Numa Nodes 2. Configure a VM with strict NUMA pinning and excluding NUMA Node 0, e.g. <numatune> <memory mode='strict' nodeset='1'/> </numatune> 3. Start the VM Actual results: The VM does not start with OOM condition for the vcpu threads: kvm_create_vcpu: Cannot allocate memory Failed to create vCPU. Check the -smp parameter. Expected results: VM should start, as it did with RHEL 6.6 Additional info: As soon as Numa Node 0 is used, the VM does start. I am currently trying to build a reproducer in house, but I think the recent changes for numaset to work correctly might have caused the issue, esp. the fix for BZ #1198645 might be causing this.
Some test from my side: [root@cisco-b200m1-01 ~]# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 2 4 6 8 10 12 14 node 0 size: 6080 MB node 0 free: 5445 MB node 1 cpus: 1 3 5 7 9 11 13 15 node 1 size: 6144 MB node 1 free: 5595 MB node distances: node 0 1 0: 10 21 1: 21 10 [root@cisco-b200m1-01 ~]# [root@cisco-b200m1-01 ~]# virsh dumpxml numa1 <domain type='kvm'> <name>numa1</name> <uuid>5b91caaf-7c02-0108-b5d5-12d1e3063403</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static' cpuset='1,3,5,7,9,11,13,15' current='2'>8</vcpu> <numatune> <memory mode='strict' nodeset='1'/> </numatune> <os> <type arch='x86_64' machine='rhel6.6.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/var/lib/libvirt/images/bsul0471.img'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <interface type='network'> <mac address='52:54:00:3a:c4:b7'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain> [root@cisco-b200m1-01 ~]# So this one is running on NUMA Node 1 only. Now start the VM: [root@cisco-b200m1-01 ~]# virsh start numa1 error: Failed to start domain numa1 error: Unable to read from monitor: Connection reset by peer [root@cisco-b200m1-01 ~]# The log shows: 2015-09-15 13:21:08.914+0000: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name numa1 -S -M rhel6.6.0 -enable-kvm -m 4096 -realtime mlock=off -smp 2,maxcpus=8,sockets=8,cores=1,threads=1 -uuid 5b91caaf-7c02-0108-b5d5-12d1e3063403 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/numa1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -drive file=/var/lib/libvirt/images/bsul0471.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3a:c4:b7,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on char device redirected to /dev/pts/2 kvm_create_vcpu: Cannot allocate memory Failed to create vCPU. Check the -smp parameter. 2015-09-15 13:21:09.325+0000: shutting down Doing a small change to Nodeset 0 works: [root@cisco-b200m1-01 ~]# diff numa1.xml numa1_new.xml 8c8 < <memory mode='strict' nodeset='1'/> --- > <memory mode='strict' nodeset='0'/> [root@cisco-b200m1-01 ~]# virsh start numa1 Domain numa1 started [root@cisco-b200m1-01 ~]#
I just tried downgrading qemu and libvirt to RHEL 6.6 versions. Interestingly this did not change the behaviour. So I completely downgraded my system to a RHEL 6.6 level and reran the tests. this time all works as expected: <vcpu placement='static' cpuset='1,3,5,7,9,11,13,15' current='2'>8</vcpu> <numatune> <memory mode='strict' nodeset='1'/> </numatune> [root@cisco-b200m1-01 ~]# virsh start numa1 Domain numa1 started [root@cisco-b200m1-01 ~]# rpm -q -a qemu\* libvirt\* libvirt-java-devel-0.4.9-1.el6.noarch libvirt-0.10.2-46.el6.x86_64 libvirt-devel-0.10.2-46.el6.x86_64 libvirt-python-0.10.2-46.el6.x86_64 qemu-kvm-0.12.1.2-2.445.el6.x86_64 libvirt-java-0.4.9-1.el6.noarch qemu-img-0.12.1.2-2.445.el6.x86_64 libvirt-client-0.10.2-46.el6.x86_64 [root@cisco-b200m1-01 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.6 (Santiago) [root@cisco-b200m1-01 ~]#
Updating libvirt and qemu from RHEL 6.6 only, still kept the system working: ================================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================================ Updating: libvirt x86_64 0.10.2-54.el6 beaker-Server 2.4 M qemu-kvm x86_64 2:0.12.1.2-2.479.el6 beaker-Server 1.6 M Updating for dependencies: libvirt-client x86_64 0.10.2-54.el6 beaker-Server 4.1 M libvirt-devel x86_64 0.10.2-54.el6 beaker-Server 910 k libvirt-python x86_64 0.10.2-54.el6 beaker-Server 500 k qemu-img x86_64 2:0.12.1.2-2.479.el6 beaker-Server 830 k Transaction Summary ================================================================================================================================================================ [root@cisco-b200m1-01 yum.repos.d]# service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: [ OK ] [root@cisco-b200m1-01 yum.repos.d]# virsh start numa1 Domain numa1 started [root@cisco-b200m1-01 yum.repos.d]# So next I also updated the kernel to a RHEL 6.7 kernel: ================================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================================ Installing: kernel x86_64 2.6.32-573.el6 beaker-Server 30 M Updating for dependencies: kernel-firmware noarch 2.6.32-573.el6 beaker-Server 18 M Transaction Summary ================================================================================================================================================================ Install 1 Package(s) Upgrade 1 Package(s) After rebooting the system, the error showed up: [root@cisco-b200m1-01 ~]# virsh start numa1 error: Failed to start domain numa1 error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1 kvm_create_vcpu: Cannot allocate memory Failed to create vCPU. Check the -smp parameter. [root@cisco-b200m1-01 ~]# So this looks to be a kernel issue and no libvirt/qemu issue then. Still doing some additional tests.
Hi Jan, I guess this issue looks like this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1206424 Wish this will help you Thanks, Luyao
Yes, with the patch from bug 1206424 (or rather its RHEL 7 clone: 1010885) I am able to start the machine with 6.7 kernel: commit 7e72ac787848b7434c9359a57c1e2789d92350f8 Author: Martin Kletzander <mkletzan> CommitDate: 2014-07-16 20:15:46 +0200 qemu: leave restricting cpuset.mems after initialization git describe: v1.2.6-176-g7e72ac7 contains: v1.2.7-rc1~91 As Martin Tessun discovered in comment 4, it seems the kernel change that prompted the libvirt patch was backported to kernel between versions 2.6.32-504.el6.x86_64 and 2.6.32-573.el6.x86_64.
I can reproduce this issue with libvirt-0.10.2-54.el6.x86_64, kernel-2.6.32-573.8.1.el6.x86_64: 1. check DMA location: # cat /proc/zoneinfo | grep DMA Node 0, zone DMA Node 0, zone DMA32 Node 1, zone DMA32 2. prepare a guest which memory bind to node 3: # virsh dumpxml test4 ... <numatune> <memory mode='strict' nodeset='3'/> </numatune> ... 4. start guest: # virsh start test4 error: Failed to start domain test4 error: internal error process exited while connecting to monitor: kvm_create_vcpu: Cannot allocate memory Failed to create vCPU. Check the -smp parameter. And verify this bug with libvirt-0.10.2-55.el6.x86_64, kernel-2.6.32-573.8.1.el6.x86_64: 1. check DMA location: # cat /proc/zoneinfo | grep DMA Node 0, zone DMA Node 0, zone DMA32 Node 1, zone DMA32 2. prepare a guest which memory bind to node 3: # virsh dumpxml test4 ... <numatune> <memory mode='strict' nodeset='3'/> </numatune> ... 3. start guest: # virsh start test4 Domain test4 started And test cpu hot-plug: 1. # virsh vcpucount test4 maximum config 5 maximum live 5 current config 4 current live 4 # cgget -g cpuset /libvirt/qemu/test4 /libvirt/qemu/test4: cpuset.memory_spread_slab: 0 cpuset.memory_spread_page: 0 cpuset.memory_pressure: 0 cpuset.memory_migrate: 1 cpuset.sched_relax_domain_level: -1 cpuset.sched_load_balance: 1 cpuset.mem_hardwall: 0 cpuset.mem_exclusive: 0 cpuset.cpu_exclusive: 0 cpuset.mems: 3 cpuset.cpus: 0-23 2. # virsh setvcpus test4 5 3. check cgroup: # cgget -g cpuset /libvirt/qemu/test4/vcpu4 /libvirt/qemu/test4/vcpu4: cpuset.memory_spread_slab: 0 cpuset.memory_spread_page: 0 cpuset.memory_pressure: 0 cpuset.memory_migrate: 1 cpuset.sched_relax_domain_level: -1 cpuset.sched_load_balance: 1 cpuset.mem_hardwall: 0 cpuset.mem_exclusive: 0 cpuset.cpu_exclusive: 0 cpuset.mems: 3 cpuset.cpus: 2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0738.html
After upgrading libvirt to libvirt-0.10.2-60.el6.x86_64.rpm from libvirt-0.10.2-54.el6.x86_64.rpm on RHEL6.7 the issue has been resolved