Bug 808313 - win2008r2 sees only 64 cpus when starting guest wiith 160 vcpus
win2008r2 sees only 64 cpus when starting guest wiith 160 vcpus
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.3
x86_64 Windows
medium Severity medium
: rc
: ---
Assigned To: Peter Krempa
Virtualization Bugs
:
: 808316 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-30 02:12 EDT by FuXiangChun
Modified: 2012-05-24 05:29 EDT (History)
26 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-05-24 05:29:20 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description FuXiangChun 2012-03-30 02:12:36 EDT
Description of problem:
The current supported limit with Windows 2008 R2 Datacenter Edition is 256 Logical Processors.

Version-Release number of selected component (if applicable):
rpm -qa|grep qemu
qemu-kvm-0.12.1.2-2.267.el6.x86_64

uname -r
2.6.32-257.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1./usr/libexec/qemu-kvm run -M rhel6.3.0 --enable-kvm -m 4G -smp 1,maxcpus=161 -name rhel6.3 -uuid ddcbfb49-3411-1701-3c36-6bdbc00bedbb -rtc base=utc,clock=host,driftfix=slew -boot c -drive file=/dev/vg-qzhang/lv-2k8r2,if=none,id=virtio,format=raw,cache=none,werror=stop,rerror=stop,boot=on -device virtio-blk-pci,drive=virtio,id=drive-virtio0-0-0,bootindex=1 -netdev tap,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=32:12:50:a4:32:74 -spice port=5914,disable-ticketing -vga qxl -device sga -chardev socket,id=serial0,path=/var/test3,server,nowait -device isa-serial,chardev=serial0 -monitor unix:/tmp/monitor4,server,nowait -monitor stdio -qmp tcp:0:6667,server,nowait

2.hotplug with script
i=1
while [ $i -lt 160 ]
do
sleep 2
echo "cpu_set $i online"|nc -U /tmp/monitor4
i=$(($i+1))
done

3.reboot guest

4.check logical processors in guest
Opening Task Manager and going to the Tab 'Performance'
  
Actual results:
show 64 Logical Processors

Expected results:
show 160 logical processors

Additional info:

guest is win2008r2 datacenter
Comment 2 FuXiangChun 2012-03-30 03:45:57 EDT
I tested two scenarios again.

1. boot win2008r2 guest with -smp 160 
result: guest only can find 64 vcpus from performance(in Task Manager). 

2. boot win2008r2 guest with -smp 160,cores=4,sockets=40,threads=1
result: 
It takes more than 22 minutes to boot guest. but guest find 160 vcpus in performance tab.

Notes:
I am not sure if management tool can auto match sockets when use -smp 160 without sockets option to boot guest?   If so, this maybe not be a problem. 

why guest boot need to spend >22 minutes?
Comment 3 Paolo Bonzini 2012-04-02 07:58:17 EDT
This needs to be either a libvirt or virt-install bug, or just a kbase entry.
Comment 4 Igor Mammedov 2012-04-03 09:24:51 EDT
Dave,
May I switch component to libvirt?
Comment 5 Dave Allan 2012-04-03 09:51:42 EDT
Igor and Paolo, I'm puzzled as to why this might be a libvirt BZ.  It appears from the description that the reporter isn't using libvirt at all.  Can you explain?
Comment 6 Igor Mammedov 2012-04-03 10:16:47 EDT
(In reply to comment #5)
> Igor and Paolo, I'm puzzled as to why this might be a libvirt BZ.  It appears
> from the description that the reporter isn't using libvirt at all.  Can you
> explain?

If libvirt starts guest with 160 cpus for win2008r2 as:
 ... -smp 160,cores=4,sockets=40,threads=1 ...

Then we could close it as not a bug. Otherwise libvirt should specify 'correct' topology for win2008r2 to be happy and see all 160 cpus.
Comment 7 Dave Allan 2012-04-04 23:25:37 EDT
If you find libvirt starting a guest with an inappropriate cmdline, then certainly file a BZ, or move this one to libvirt.  AFAIK, though, libvirt does the right thing when properly requested by the domain xml, but I don't start guests on the scale of this one so I'm not entirely certain of that.
Comment 8 Igor Mammedov 2012-04-05 04:48:41 EDT
FuXiangChun,

Can you retest using libvirt to manage vm, instead of running qemu manually?
Comment 9 FuXiangChun 2012-04-06 00:49:07 EDT
(In reply to comment #8)
> FuXiangChun,
> 
> Can you retest using libvirt to manage vm, instead of running qemu manually?

I tested libvirt to manage vm, It has the same issue.

steps:
1.virt-manager start a win2kr2 guest with 'Max allocation 160' and 'Current allocation 1'
2.use virtsh to hotplug vcpu to guest
 # virsh setvcpus hotplug 80
 # virsh setvcpus hotplug 160
3.check guest 'devive manager'
   show 160 vcpus 
4. reboot guest
   show 64 vpus from guest Task Manager
Comment 10 Ademar Reis 2012-04-09 10:42:30 EDT
*** Bug 808316 has been marked as a duplicate of this bug. ***
Comment 11 Ademar Reis 2012-04-09 10:45:11 EDT
Since CPU hotplug is tech-preview and this is a corner case, postponing this to 6.4.
Comment 12 Igor Mammedov 2012-04-10 08:59:26 EDT
According to comments 2 and 9, libvirt does not start qemu with proper arguments.

I've created windows vm and tried to run it with 160 vcpus.
As result qemu was started with following cpu options:
...  -smp 160,maxcpus=160,sockets=160,cores=1,threads=1 ...
as result windows sees only 64 cpus.

Probably testing support for 160 does not include testing with win2008r2 so this bug wasn't caught earlier. Moving bug to libvirt.

Versions of used components:
kernel-2.6.32-259.el6.x86_64

libvirt-0.9.10-10.el6.x86_64
libvirt-python-0.9.10-10.el6.x86_64
virt-manager-0.9.0-11.el6.x86_64
python-virtinst-0.600.0-8.el6.noarch
libvirt-client-0.9.10-10.el6.x86_64
Comment 13 Jiri Denemark 2012-04-10 10:15:00 EDT
This doesn't really look like a libvirt bug. Libvirt doesn't try to invent CPU topology, it just uses what the user asked for. You should be able to add

<cpu>
  <topology sockets='40' cores='4' threads='1'/>
</cpu>

to get the desired qemu command line.
Comment 14 Dave Allan 2012-04-10 14:23:39 EDT
@xfu, can you retest with the XML provided by Jiri and comment with the qemu commandline?
Comment 15 FuXiangChun 2012-04-11 07:31:10 EDT
I am retesting it, I will update result to bz as soon as possible.
Comment 16 FuXiangChun 2012-04-16 07:03:56 EDT
(In reply to comment #14)
> @xfu, can you retest with the XML provided by Jiri and comment with the qemu
> commandline?

1.virsh edit hotplug(domain name)

<vcpu>160</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu>
    <topology sockets='40' cores='4' threads='1'/>
  </cpu>

2. boot guest with virt-manager

3. get qemu commandline
/usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 2048 -smp 160,sockets=40,cores=4,threads=1 -name hotplug -uuid 1bb0784d-9765-4b51-fd42-26e17e880ad6 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/hotplug.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/mapper/vg--qzhang-lv--2k8r2,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=native -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:64:19:d7,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Comment 17 min zhan 2012-04-20 05:44:40 EDT
I can not reproduce this bug with the following pkgs:

kernel-2.6.32-262.el6.x86_64
qemu-kvm-0.12.1.2-2.277.el6.x86_64
libvirt-0.9.10-13.el6.x86_64

steps:
# virsh edit guest
...
  <vcpu current='1'>160</vcpu>

# virsh start guest
make sure guest can start successfully, check the current cpu and max cpu value display correctly, as 1 and 160. 

# ps -ef|grep kvm
root      1422     2  0 16:20 ?        00:00:00 [kvm-irqfd-clean]
qemu     16691     1 99 17:20 ?        00:24:34 /usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 1024 -smp 1,maxcpus=160,sockets=160,cores=1,threads=1 -name win2008r2 -uuid c53d6254-11e5-68b2-04af-252a8bd93745 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win2008r2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/mnt/Windows_KVM_images/win2008r2-64-virtio.raw,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=28,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:c0:3c:80,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4


# virsh setvcpus guest 80
check the processor below device manager in guest, it is 80.

# virsh setvcpus guest 160 
check the processor below device manager in guest, it is 160.

and reboot the guest, check again, it is still 160
Comment 18 FuXiangChun 2012-04-20 06:24:34 EDT
> # virsh setvcpus guest 80
> check the processor below device manager in guest, it is 80.
> 
> # virsh setvcpus guest 160 
> check the processor below device manager in guest, it is 160.
> 
> and reboot the guest, check again, it is still 160


device manager show correctly. but it is wrong from guest Task Manager. guest are using vcpu from Task Manager.
Comment 19 min zhan 2012-04-23 05:38:45 EDT
 
> device manager show correctly. but it is wrong from guest Task Manager. guest
> are using vcpu from Task Manager.

Then what's your result for guest Task Manager when you setvcpus as 160 immediately, not reboot?

The same as you. I can not get it correctly (160) in guest Task Manager whenever setvcpus immediately or after reboot. It display 8 after setvcpus 160 immediately. After reboot it display 64.
Comment 20 Peter Krempa 2012-05-18 09:04:56 EDT
Microsoft limits the number of CPU sockets that are usable on their OSes. For W2008 Server Datacenter Edition the limit is 64 sockets. 
( http://www.microsoft.com/en-us/server-cloud/windows-server/2008-r2-datacenter.aspx 
- Support for up to 64 x64 physical processors (up to 256 logical processors) and 2 terabytes of RAM
)

To use more than this you need to specify a multicore CPU topology that is not done by default in libvirt. You need to follow Jiri's advice (comment 13) to specify the topology manually otherwise you'll get 160 singlecore singlethread processors that exceed the limit. You may specify the topology even if you don't enable all the processors (eg. for hot-plugging later)
Comment 21 Peter Krempa 2012-05-24 05:29:20 EDT
If you specify a correct CPU topology as Jiri suggested in Comment 13, you get the desired qemu commandline as Igor suggests in Comment 6 (that is needed for W2k8 server to recognize all the processors due to a software limit), therefore I'm closing this as NOTABUG.

Note You need to log in before you can comment on or make changes to this bug.