Red Hat Bugzilla – Bug 808313
win2008r2 sees only 64 cpus when starting guest wiith 160 vcpus
Last modified: 2012-05-24 05:29:20 EDT
Description of problem:
The current supported limit with Windows 2008 R2 Datacenter Edition is 256 Logical Processors.
Version-Release number of selected component (if applicable):
rpm -qa|grep qemu
Steps to Reproduce:
1./usr/libexec/qemu-kvm run -M rhel6.3.0 --enable-kvm -m 4G -smp 1,maxcpus=161 -name rhel6.3 -uuid ddcbfb49-3411-1701-3c36-6bdbc00bedbb -rtc base=utc,clock=host,driftfix=slew -boot c -drive file=/dev/vg-qzhang/lv-2k8r2,if=none,id=virtio,format=raw,cache=none,werror=stop,rerror=stop,boot=on -device virtio-blk-pci,drive=virtio,id=drive-virtio0-0-0,bootindex=1 -netdev tap,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=32:12:50:a4:32:74 -spice port=5914,disable-ticketing -vga qxl -device sga -chardev socket,id=serial0,path=/var/test3,server,nowait -device isa-serial,chardev=serial0 -monitor unix:/tmp/monitor4,server,nowait -monitor stdio -qmp tcp:0:6667,server,nowait
2.hotplug with script
while [ $i -lt 160 ]
echo "cpu_set $i online"|nc -U /tmp/monitor4
4.check logical processors in guest
Opening Task Manager and going to the Tab 'Performance'
show 64 Logical Processors
show 160 logical processors
guest is win2008r2 datacenter
I tested two scenarios again.
1. boot win2008r2 guest with -smp 160
result: guest only can find 64 vcpus from performance(in Task Manager).
2. boot win2008r2 guest with -smp 160,cores=4,sockets=40,threads=1
It takes more than 22 minutes to boot guest. but guest find 160 vcpus in performance tab.
I am not sure if management tool can auto match sockets when use -smp 160 without sockets option to boot guest? If so, this maybe not be a problem.
why guest boot need to spend >22 minutes?
This needs to be either a libvirt or virt-install bug, or just a kbase entry.
May I switch component to libvirt?
Igor and Paolo, I'm puzzled as to why this might be a libvirt BZ. It appears from the description that the reporter isn't using libvirt at all. Can you explain?
(In reply to comment #5)
> Igor and Paolo, I'm puzzled as to why this might be a libvirt BZ. It appears
> from the description that the reporter isn't using libvirt at all. Can you
If libvirt starts guest with 160 cpus for win2008r2 as:
... -smp 160,cores=4,sockets=40,threads=1 ...
Then we could close it as not a bug. Otherwise libvirt should specify 'correct' topology for win2008r2 to be happy and see all 160 cpus.
If you find libvirt starting a guest with an inappropriate cmdline, then certainly file a BZ, or move this one to libvirt. AFAIK, though, libvirt does the right thing when properly requested by the domain xml, but I don't start guests on the scale of this one so I'm not entirely certain of that.
Can you retest using libvirt to manage vm, instead of running qemu manually?
(In reply to comment #8)
> Can you retest using libvirt to manage vm, instead of running qemu manually?
I tested libvirt to manage vm, It has the same issue.
1.virt-manager start a win2kr2 guest with 'Max allocation 160' and 'Current allocation 1'
2.use virtsh to hotplug vcpu to guest
# virsh setvcpus hotplug 80
# virsh setvcpus hotplug 160
3.check guest 'devive manager'
show 160 vcpus
4. reboot guest
show 64 vpus from guest Task Manager
*** Bug 808316 has been marked as a duplicate of this bug. ***
Since CPU hotplug is tech-preview and this is a corner case, postponing this to 6.4.
According to comments 2 and 9, libvirt does not start qemu with proper arguments.
I've created windows vm and tried to run it with 160 vcpus.
As result qemu was started with following cpu options:
... -smp 160,maxcpus=160,sockets=160,cores=1,threads=1 ...
as result windows sees only 64 cpus.
Probably testing support for 160 does not include testing with win2008r2 so this bug wasn't caught earlier. Moving bug to libvirt.
Versions of used components:
This doesn't really look like a libvirt bug. Libvirt doesn't try to invent CPU topology, it just uses what the user asked for. You should be able to add
<topology sockets='40' cores='4' threads='1'/>
to get the desired qemu command line.
@xfu, can you retest with the XML provided by Jiri and comment with the qemu commandline?
I am retesting it, I will update result to bz as soon as possible.
(In reply to comment #14)
> @xfu, can you retest with the XML provided by Jiri and comment with the qemu
1.virsh edit hotplug(domain name)
<type arch='x86_64' machine='rhel6.3.0'>hvm</type>
<topology sockets='40' cores='4' threads='1'/>
2. boot guest with virt-manager
3. get qemu commandline
/usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 2048 -smp 160,sockets=40,cores=4,threads=1 -name hotplug -uuid 1bb0784d-9765-4b51-fd42-26e17e880ad6 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/hotplug.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/mapper/vg--qzhang-lv--2k8r2,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=native -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:64:19:d7,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
I can not reproduce this bug with the following pkgs:
# virsh edit guest
# virsh start guest
make sure guest can start successfully, check the current cpu and max cpu value display correctly, as 1 and 160.
# ps -ef|grep kvm
root 1422 2 0 16:20 ? 00:00:00 [kvm-irqfd-clean]
qemu 16691 1 99 17:20 ? 00:24:34 /usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 1024 -smp 1,maxcpus=160,sockets=160,cores=1,threads=1 -name win2008r2 -uuid c53d6254-11e5-68b2-04af-252a8bd93745 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008r2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/mnt/Windows_KVM_images/win2008r2-64-virtio.raw,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=28,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c0:3c:80,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
# virsh setvcpus guest 80
check the processor below device manager in guest, it is 80.
# virsh setvcpus guest 160
check the processor below device manager in guest, it is 160.
and reboot the guest, check again, it is still 160
> # virsh setvcpus guest 80
> check the processor below device manager in guest, it is 80.
> # virsh setvcpus guest 160
> check the processor below device manager in guest, it is 160.
> and reboot the guest, check again, it is still 160
device manager show correctly. but it is wrong from guest Task Manager. guest are using vcpu from Task Manager.
> device manager show correctly. but it is wrong from guest Task Manager. guest
> are using vcpu from Task Manager.
Then what's your result for guest Task Manager when you setvcpus as 160 immediately, not reboot?
The same as you. I can not get it correctly (160) in guest Task Manager whenever setvcpus immediately or after reboot. It display 8 after setvcpus 160 immediately. After reboot it display 64.
Microsoft limits the number of CPU sockets that are usable on their OSes. For W2008 Server Datacenter Edition the limit is 64 sockets.
- Support for up to 64 x64 physical processors (up to 256 logical processors) and 2 terabytes of RAM
To use more than this you need to specify a multicore CPU topology that is not done by default in libvirt. You need to follow Jiri's advice (comment 13) to specify the topology manually otherwise you'll get 160 singlecore singlethread processors that exceed the limit. You may specify the topology even if you don't enable all the processors (eg. for hot-plugging later)
If you specify a correct CPU topology as Jiri suggested in Comment 13, you get the desired qemu commandline as Igor suggests in Comment 6 (that is needed for W2k8 server to recognize all the processors due to a software limit), therefore I'm closing this as NOTABUG.