Bug 949547

Created attachment 732666 [details]
perf report

for qemu-kvm (using 5% cpu at idle) process running windows

This seems to be a much worse problem now. With 10% usage inside the guest, qemu-kvm process usage rises to 50% and more; and in just half an hour increasing the load average to 5. This is getting worse! 

Would someone mind checking this bug report?

Created attachment 733157 [details]
latest perf report

latest perf report attached

Reproduce this problem on the latest kernel and qemu-kvm version. The qemu-kvm process consumes about 7% cpu when the windows guest is idle inside and 50% when guest uses 5% cpu inside.

Host info:
kernel-2.6.32-358.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.359.el6.x86_64
seabios-0.6.1.2-26.el6.x86_64
guest info:
win7-64bit

Steps:
1.configure the environment.
Optimizations:
* removed tablet device
* apic/acpi enabled
  append '-global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0' in cli.
* localtime with bcdedit's useplatformclock inside guest
   C:\User\Administrator>bcdedit /set {default} useplatformclock true
   The operation completed successfully.
* topology: cores per socket instead of default all-sockets
  e.g:-smp 4,sockets=1,cores=4,threads=1
* cpu -host
* enabled hyperv's hv_relaxed timing interrupt
2.boot a win7 guest with the optimization of step1, do not start CPU at startup with '-S'.
e.g:# /usr/libexec/qemu-kvm -name sluo-testing -S -M rhel6.4.0 -cpu host,hv_relaxed,+rdtscp,+pdpe1gb,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 20480 -smp 4,sockets=1,cores=4,threads=1 -uuid 6172da17-4b79-4786-937d-463c616d69d8 -nodefconfig -nodefaults -rtc base=localtime -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/win7-64.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-0-0,readonly=on,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0,vhost=off,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=5C:F3:FC:DB:B4:C1,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -vnc :1 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -monitor stdio
3.record its profile into perf.data.
# /usr/bin/perf record -p `pidof qemu-kvm`
4.read perf.data (created by perf record) and display the profile.
# /usr/bin/perf report -i perf.data > /home/perf-report.txt

Results:
I will attach the perf-report later after step 4.
Qemu-kvm process for win7 64bit guests idles at 7% cpu (after optimizations) while task manager inside the guest shows 0% cpu consumption and qemu-kvm process consume about 50% when guest uses 5% cpu inside.
e.g1: Qemu-kvm process 7% cpu while 0% cpu inside guest
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
20542 root      20   0 24.0g  19g 4600 S  6.9 62.4   3:00.47 qemu-kvm 
e.g2: Qemu-kvm process 50% cpu while 5% cpu inside guest
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
20542 root      20   0 24.5g  19g 4600 S 47.6 62.4   3:24.33 qemu-kvm

Created attachment 733473 [details]
perf-report of my testing.

(In reply to comment #5)
> 3.record its profile into perf.data.
> # /usr/bin/perf record -p `pidof qemu-kvm`
add one step here that continue the CPU to start up VM via 'cont' in monitor.
(qemu) cont
after observe both the qemu-kvm process consumes cpu and inside the guest for while, then do step 4.
> 4.read perf.data (created by perf record) and display the profile.
> # /usr/bin/perf report -i perf.data > /home/perf-report.txt
>

Any thoughts what exactly is triggering this high cpu.. ? That's the tricky question.

The usage jumps as high as 230%. Makes me think if moving to kvm was as bad choice after all :-/

I figured that this is being caused by high disk i/o. It can be reproduced by any benchmark tools. Starting benchmark after guest startup, shoots cpu to 120%+. Stopping it brings back cpu to 40% but it stays there forever cause of 20+ threads in iotop, for a single guest.

Attached is the iotop report.

Created attachment 734512 [details]
iotop showing lots of threads from qemu

I know there are no imminent answers about bugs but truth be told, kvm sucks, since the problem is reproduced and is there!

Moving off Xen was a really terrible mistake by Red Hat. 

Sadly, cannot afford to lose more time on this kvm shit so off my way back to Xen. 

Lesson learned, never trust a corporate decision that's been made for you! Stupid fuggly kvm.

Thank you for taking the time to enter a bug report with us.  We appreciate the feedback and look to use reports such as this to guide our efforts at improving our products.  That being said, this bug tracking system is not a mechanism for requesting support, and we are not able to  guarantee the timeliness or suitability of a resolution.

If this issue is critical or in any way time sensitive, please raise a ticket through your regular Red Hat support channels to make certain  it receives the proper attention and prioritization to assure a timely resolution.

For information on how to contact the Red Hat production support team, please visit   https://www.redhat.com/support/process/production/#howto

yes whatever!

I have observed the same issue. For me, Qemu-kvm process for windows guests idles at 7%+ cpu, while task manager inside the guest shows 0% cpu consumption. This indeed a performance issue needs to be analyzed and corrected.

But interestingly, RHEL 5.X vms also showing more idle cpu usage. Please see top output below: 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29274 root      20   0 3606m 1.4g 5036 S 11.2  4.5   1980:55 /usr/libexec/qemu-kvm -name VM4-RHEL5U8-X86-64 -S -M rhel6.3.0 -enable-kvm -m 2048 -smp 2,sockets=2
28985 root      20   0 3896m 1.6g 5092 S 10.2  5.0   2410:17 /usr/libexec/qemu-kvm -name VM3-RHEL5U9-X86-64 -S -M rhel6.3.0 -enable-kvm -m 2048 -smp 2,sockets=2
31398 root      20   0 3129m 2.0g 5060 R  9.3  6.5   1675:01 /usr/libexec/qemu-kvm -name VM9-WIN7-X86-64 -S -M rhel6.3.0 -enable-kvm -m 2048 -smp 2,sockets=2,co
28884 root      20   0 5818m 4.0g 5084 S  8.9 12.9   1555:35 /usr/libexec/qemu-kvm -name VM5-WIN2K3-X86-64 -S -M rhel6.3.0 -enable-kvm -m 4096 -smp 2,sockets=2,
21222 root      20   0 1165m  24m 5724 S  8.3  0.1   1027:39 libvirtd --daemon
14978 root      20   0 4667m 4.0g 5076 S  7.9 12.9 296:10.44 /usr/libexec/qemu-kvm -name VM6-WIN2K8-X86-64 -S -M rhel6.3.0 -enable-kvm -m 4096 -smp 2,sockets=2,
23938 root      20   0 4591m 3.5g 5060 S  7.6 11.3 273:04.54 /usr/libexec/qemu-kvm -name VM7-WINXP-X86-32 -S -M rhel6.3.0 -enable-kvm -m 4096 -smp 2,sockets=2,c
31242 root      20   0 5186m 2.0g 5052 S  6.9  6.5   1437:21 /usr/libexec/qemu-kvm -name VM8-WINVISTA-X86-64 -S -M rhel6.3.0 -enable-kvm -m 4096 -smp 2,sockets=
21463 root      20   0 1494m 191m  17m S  1.7  0.6 951:24.77 /usr/bin/python /usr/share/virt-manager/virt-manager.py
32740 root      20   0 1038m 164m  16m S  1.7  0.5 696:15.51 /usr/bin/python /usr/share/virt-manager/virt-manager.py
23976 root      20   0 5606m 1.2g 4588 S  1.3  3.7 931:34.99 /usr/libexec/qemu-kvm -name VM2-RHEL6U4-X86-64 -S -M rhel6.4.0 -enable-kvm -m 2048 -smp 1,sockets=1
16227 root      20   0 5252m 999m 4608 S  0.3  3.1 312:09.88 /usr/libexec/qemu-kvm -name VM1-RHEL6U3-X86-64 -S -M rhel6.3.0 -enable-kvm -m 2048 -smp 4,sockets=4

So only RHEL6 Vms show less idle time cpu usage.

> But interestingly, RHEL 5.X vms also showing more idle cpu usage. Please see
> top output below: 

> So only RHEL6 Vms show less idle time cpu usage.

Idle load depends on guest behavior, so different guests showing different idle loads isn't a bug.  Some background:

RHEL-6 supports tickless idle, so there is no periodic timer interrupt running in case the guest is idle, leading to a very low idle load in virtual machines (and also longer sleep times in deep C states on physical hardware).

Guests which don't have that will be woken up frequently by the timer interrupt. And even if they just figure they have still nothing to do and go sleep again instantly this is still something qemu-kvm must handle many times per second, summing up to a mensurable overhead.

Does anyone see a regression here?  i.e. behavior in RHEL-6.4 being worse than in older versions?

I am affected by this bug as well.
Red Hat ignores that many people need to virtualize windows as well.

(In reply to Alves from comment #31)
> I am affected by this bug as well.
> Red Hat ignores that many people need to virtualize windows as well.

!. Are you using virtio net and storage devices?
2. If you boot up Windows in safe-mode, do you see the
same CPU consumption?

Thanks,
Vadim.

The answer is yes, I am 100% virtio.
recently I started using a new feature that brought CPU under control.
 <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='4096'/>
    </hyperv>

but the sad side-effect is that my clock is roughly 1/2 of what it should me. In one hour my internal clock thinks only one half has elapsed. The virtual machine is virtually useless for all serious purposes, databases, etc. I tried the hack 
localtime with bcdedit's useplatformclock inside guest

but it makes no difference. This Windows Server 2008 R2 Datacenter. I am forced to look for a different solution for Windows virtualization. I want to complain about Red Hat not taking seriously Windows Virtual machines, while Microsoft does indeed take seriously virtualizing Linux.
If somebody has any idea how to pass the clock from the host to the guest, please let me know.

(In reply to Alves from comment #33)
> The answer is yes, I am 100% virtio.
> recently I started using a new feature that brought CPU under control.
>  <hyperv>
>       <relaxed state='on'/>
>       <vapic state='on'/>
>       <spinlocks state='on' retries='4096'/>
>     </hyperv>
> 
> but the sad side-effect is that my clock is roughly 1/2 of what it should
> me. In one hour my internal clock thinks only one half has elapsed. The
> virtual machine is virtually useless for all serious purposes, databases,
> etc. I tried the hack 
> localtime with bcdedit's useplatformclock inside guest
> 
> but it makes no difference. This Windows Server 2008 R2 Datacenter. I am
> forced to look for a different solution for Windows virtualization. I want
> to complain about Red Hat not taking seriously Windows Virtual machines,
> while Microsoft does indeed take seriously virtualizing Linux.
> If somebody has any idea how to pass the clock from the host to the guest,
> please let me know.

Could it be that your VM is running with HPET enabled?

How would I know that? This is a windows guest.
In my guest definition file I am using

 <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup' track='guest'>
      <catchup  threshold='123' slew='120' limit='10000'/>
    </timer>
    <timer name='pit' tickpolicy='delay'/>
  </clock>

 <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='4096'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>core2duo</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='cx16'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
  </cpu>

rpm -qa | grep qemu
qemu-common-1.6.0-8.fc19.x86_64
qemu-system-x86-1.6.0-8.fc19.x86_64
ipxe-roms-qemu-20130517-3.gitc4bce43.fc19.noarch
qemu-guest-agent-1.6.0-8.fc19.x86_64
qemu-img-1.6.0-8.fc19.x86_64
qemu-kvm-1.6.0-8.fc19.x86_64
libvirt-daemon-driver-qemu-1.1.3-1.fc19.x86_64

The host is a Dell r900 with 4 Intel X7350 Quad CPUS.
Much appreciate you help.

My host is using hpet
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

 dmesg | grep hpet
[    0.000000] hpet clockevent registered
[    0.613108] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.613112] hpet0: 3 comparators, 64-bit 14.318180 MHz counter
[    0.617084] Switched to clocksource hpet
[    0.960929] rtc_cmos 00:03: alarms up to one day, y3k, 242 bytes nvram, hpet irqs
[    2.133194] CE: hpet increased min_delta_ns to 20113 nsec

Maybe the host should be using tsc. How do I go around it? This Fedora 19.
uname -r
3.11.2-201.fc19.x86_64

This is another indication of a problem
 dmesg | grep tsc
[    0.000000] tsc: Fast TSC calibration failed
[    0.000000] tsc: PIT calibration matches HPET. 1 loops
[    0.000000] tsc: Detected 2925.858 MHz processor
[    0.882661] tsc: Marking TSC unstable due to TSC halts in idle

Is the hardware bad?

(In reply to Alves from comment #37)
> This is another indication of a problem
>  dmesg | grep tsc
> [    0.000000] tsc: Fast TSC calibration failed
> [    0.000000] tsc: PIT calibration matches HPET. 1 loops
> [    0.000000] tsc: Detected 2925.858 MHz processor
> [    0.882661] tsc: Marking TSC unstable due to TSC halts in idle
> 
> Is the hardware bad?

My question was about guest (you can check it from Device Manager dialog inside on your VM. Just open it, go to "System devices" and check if can see "High precision event timer" item there.)

But in any case the dmesg output doesn't look good to me. CC'ing Gleb.

Best,
Vadim.

I solved my adding the parameter processor.max-cstate=1 to grub, and disabled hpet in the guest's XML file.
But in my BIOS the C-states were disabled. Furthermore, the X7350 can provide a constant TSC even in a deep C state. By adding that parameter I am spending a lot of money in power. Please consider this a new bug.

(In reply to Vadim Rozenfeld from comment #38)
> (In reply to Alves from comment #37)
> > This is another indication of a problem
> >  dmesg | grep tsc
> > [    0.000000] tsc: Fast TSC calibration failed
> > [    0.000000] tsc: PIT calibration matches HPET. 1 loops
> > [    0.000000] tsc: Detected 2925.858 MHz processor
> > [    0.882661] tsc: Marking TSC unstable due to TSC halts in idle
> > 
> > Is the hardware bad?
> 
> My question was about guest (you can check it from Device Manager dialog
> inside on your VM. Just open it, go to "System devices" and check if can see
> "High precision event timer" item there.)
> 
> But in any case the dmesg output doesn't look good to me. CC'ing Gleb.
> 
We need to implement Hyper-V reference tsc & counter, without it Windows performance will be subpar since Windows love to use timers and without Hyper-V extensions it uses PM which goes all the way to userspace many times per second.

(In reply to Gleb Natapov from comment #40)
> (In reply to Vadim Rozenfeld from comment #38)
> > (In reply to Alves from comment #37)
> > > This is another indication of a problem
> > >  dmesg | grep tsc
> > > [    0.000000] tsc: Fast TSC calibration failed
> > > [    0.000000] tsc: PIT calibration matches HPET. 1 loops
> > > [    0.000000] tsc: Detected 2925.858 MHz processor
> > > [    0.882661] tsc: Marking TSC unstable due to TSC halts in idle
> > > 
> > > Is the hardware bad?
> > 
> > My question was about guest (you can check it from Device Manager dialog
> > inside on your VM. Just open it, go to "System devices" and check if can see
> > "High precision event timer" item there.)
> > 
> > But in any case the dmesg output doesn't look good to me. CC'ing Gleb.
> > 
> We need to implement Hyper-V reference tsc & counter, without it Windows
> performance will be subpar since Windows love to use timers and without
> Hyper-V extensions it uses PM which goes all the way to userspace many times
> per second.

yes, it's on the way.

I don't know if this bug system accepts images. In any case, these are my devices.
http://www.minixel.com/kvm_devices.png
I don't see the High Precision Timers

Hello,

I have observed the same issue. For me, Qemu-kvm process for windows guests idles at 7%+ cpu without virtserial install, and from 11% to 25% with virtioserial, qxl combinaison.

Can you tell us how configure clock, timer , virtio, hyperv extension to virtualise windows guest with kvm ?

Regards, 
Nicolas Prochazka.

(In reply to nicolas from comment #43)
> Hello,
> 
> I have observed the same issue. For me, Qemu-kvm process for windows guests
> idles at 7%+ cpu without virtserial install, and from 11% to 25% with
> virtioserial, qxl combinaison.
> 
> Can you tell us how configure clock, timer , virtio, hyperv extension to
> virtualise windows guest with kvm ?
> 
> Regards, 
> Nicolas Prochazka.

My personal recommendations:
 - disable hpet (--no-hpet) if it is not disable by default,
 - enable relaxed timing (hv_relaxed) on Win7/W2K8/W2K8R2,
 - enable Hyper-V vapic (hv_vapic) and Hyper-V spinlocks (hv_spinlock=0xFFF),
 - don't use UP configuration, (use at least -smp 2),
 - make sure that netkvm and virtio-blk are operating in MSI mode,
 - if you are brave enough and don't need migration - search kvm and
   qemu forums for Hyper-V reference counters and invariant TSC patches 
   and try applying any one of them.
Again, it is my personal recommendation, so if you want to try any on of them,
do it your own risk.

Cheers,
Vadim.

Guys guys, don't you get it? KVM is not perfect for virtualizing Windows. It is not there yet. Ask me in five years and they may be!

Search the web. See the major KVM based product vendors' forums. Windows idling in KVM is an issue that every haunts every administrator.

You don't wanna run a crappy node unless you don't care about the life snap of your hardware cause the way KVM runs Windows virtualization, it will just wear out the CPUs in just a year!

Go figure! ;)

(In reply to Bassu from comment #45)
> Guys guys, don't you get it? KVM is not perfect for virtualizing Windows. It
> is not there yet. Ask me in five years and they may be!
> 
> Search the web. See the major KVM based product vendors' forums. Windows
> idling in KVM is an issue that every haunts every administrator.
> 
> You don't wanna run a crappy node unless you don't care about the life snap
> of your hardware cause the way KVM runs Windows virtualization, it will just
> wear out the CPUs in just a year!
> 
> Go figure! ;)

Thank you for sharing your opinion.
If you are RH customer, I would encourage you to contact GSS team to 
discuss any problem you have, running Windows guest on top of RHEL 
platforms, running on top of RH certified  hardware.
If you are a Fedora user - please use bugzilla to submit bugs you have
discovered.
If you have any technical problems - feel free to ask question and discuss problems on qemu or/and kvm forums.
If you have any technical problems related to running Windows VM on top of KVM - feel free to ping me or Yan in addition to qemu and kvm forums.

But if you have a crappy HW, or if you use non-RH linux distributions - it can be your own problem, and you might be on your own trying to fix it.

Vadim.

Thank you for your response, Vadim.
I might have few dozens of crappy machines but what you are implying is that everyone who complained here has this problem because of their hardware. To me that is just insane, unless either the vendor is blind or if there is something seriously wrong with the software itself. 

The only purpose of reporting this bug was just to help you guys figure out the crap that's likely keeping you guys behind in the race to virtualization. I have already moved on -- to something better. I never needed your help and never will in future as I was just sharing my thoughts. Though I do believe it is good for everyone to not to depend on vendors of crappy, untested and unproved technologies from the start!

Hope this helps clear your confusion.

Thanks for reporting the bug, Bassu.  I quite agree that people
shouldn't depend on vendors of crappy, untested and unproven
technologies, and I'm glad to hear you found a vendor who can sell you
what you need for a price you consider fair, be it software, help with
figuring out problems, or taking abuse.  I'm also glad to hear that
you never needed our help, and never will in future.  If that should
turn out to be wrong, I trust you'll find us.  Thanks again, and take
care!

With that out of the way, let's go back to fixing this bug.

hello, 
After applying vadim recommendations : 

- standard windows guest idle ~ 5% cpu 
- virtio serial windoows guest idle : ~ 13 %
- virtioserial, qxl, usb bus ( usbredirc) idle: ~ 20%

Regards, 
Nicolas Prochazka.

Hi Nicolas,
Is it inside of VM or from the host side?
Thanks,
Vadim.

( % cpu in comment 50 ) is qemu process under host
inside the guest vm, windows monitor show 0-1% 

This differents tests are on : 
Linux 3.11.4  / qemu 1.5.3 
Linux 3.11.4   / qemu 1.6.0

Hi Ronen,

Can we ask QE/performance teams to do some benchmark testing for us?


Thanks,
Vadim.

Hello, 
For you intention
using  custom cpu mode from libvirt, the gain is about 3% over using -host-passthrough

with different host cpu models  ( CPU E5-2650,  E5345 , ) 

Regards, 
Nicolas Prochazka

Dear Nicolas
When I go to "Processor" in Virt-Manager, I select "Copy host CPU Configuration". According to your  findings, what exactly should I select? There are dozens of options, and one of them is kvm64. Which option should give me the highest performance and the lowest idle CPU consumption?
Right now I am using
<features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='4096'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>core2duo</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='cx16'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
  </cpu>

Hello
Is there any news about this bug ?
Regards, 
Nicolas Prochazka.

Hello,

I believe I've run afoul of this bug as well. My Windows and FreeBSD VMs are using 50% of the available CPU. If it's a single CPU VM, 50% of a CPU is used. If it's a dual CPU VM, a full CPU is used.

Now, apologies first off. I am not using RedHat, I'm just hoping that the information I provide will help in nailing down the problem. I'm running Ubuntu 13.04. virsh -v shows version 1.1.1.

Originally I thought this was an issue of high CPU load due to context switching. This was a big issue in the Proxmox distribution, and the solution was to disable the tablet interface. That did not help here.

I moved on to using an e1000 virtual NIC rather than VirtIO. No love. Then I changed to a SATA type rather than VirtIO for the virtual hard disk. That solved my issue, at least in Windows. I have not tried FreeBSD yet. If it makes any difference, in Windows I used the drivers on the VirtIO 0-1.65 ISO.

I hope this info gives you a thread to pull on. I'm a huge fan of KVM!

(In reply to Michael Bellerue from comment #57)
> I moved on to using an e1000 virtual NIC rather than VirtIO. No love. Then I
> changed to a SATA type rather than VirtIO for the virtual hard disk. That
> solved my issue, at least in Windows. I have not tried FreeBSD yet. If it
> makes any difference, in Windows I used the drivers on the VirtIO 0-1.65 ISO.

Hello Michael,
Could you please give more details regarding tp your Windows guest configuration,
like guest OS and bitness as well as qemu command line?

Thanks,
Vadim.

Sure thing. The Windows guest is Windows Server 2008 R2 64bit. I'm not sure about the qemu command line, however. The only way I've interacted with KVM has been through libvirt tools like virsh and Virtual Machine Manager. I could dump the XML config, if that would be useful.

Also, we can forget about the FreeBSD portion of my post. Turns out there really was something eating up half the CPU in that VM.

Michael, the XML config of your troublesome guest would be most appreciated!  Suggest to use the "Add an attachment" link.

Created attachment 824235 [details]
sqlserver VM is showing 15-50% CPU usage in Virtual Machine Manager. Task Manager in the VM shows 0-3%

This VM was originally built with VirtIO disks, NIC, and a Tablet interface. I changed the disks to SATA, the NIC to e1000, and removed the Tablet interface one by one to try to get the CPU usage to come down. At this point, the VM still has the issue, which is odd, because another VM (which I will post next) was fixed when these options were changed.

Created attachment 824236 [details]
sqlserver VM is showing 15-50% CPU usage in Virtual Machine Manager. Task Manager in the VM shows 0-3%

This VM was originally built with VirtIO disks, NIC, and a Tablet interface. I changed the disks to SATA, the NIC to e1000, and removed the Tablet interface one by one to try to get the CPU usage to come down. At this point, the VM still has the issue, which is odd, because another VM (which I will post next) was fixed when these options were changed.

Created attachment 824237 [details]
svr2k8r2 VM showed the same CPU issue, but was fixed.

Unfortunately I don't have very good notes on this VM. I created it quickly to try to test the problem. However, I do know that it was showing the same problem, and making the same or similar changes to the SQL Server VM brought the physical CPU usage and the virtual CPU usage back in line with each other.

I will recreate the issue on another VM, take better notes, and get the info uploaded.

Created attachment 824273 [details]
svr2k8r2-v2, same problem, better notes.

Notes are in the tar file. I included everything I could think that may be important.

Thanks Michael.

I re-checked this issue on my Win7-32 VM:

 - ide + e1000 gives me 3.7% host CPU usage;
 - viostor + e1000 goes down to 2.4%
 - viostor + e1000 + "usbdevice tablet" jumps up to 13.3%

my qemu command line is:

sudo /home/vrozenfe/work/hyperv/qemu/x86_64-softmmu/qemu-system-x86_64 -cpu qemu64,+x2apic,family=0xf,hv_vapic,hv_spinlocks=0xfff,hv_relaxed -m 1G -smp 4 -usbdevice tablet -drive file=/home/vrozenfe/work/images/w7.qcow2,if=none,id=drive-virtio0,cache=none,werror=stop,rerror=stop,boot=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio-blk-pci0,bootindex=1 -netdev user,id=hostnet0 -device e1000,netdev=hostnet0,mac=22:3A:40:3F:2F:58,id=net0 -boot c -uuid e32f9a6f-5b9e-4419-97b4-da6fe8fb7062 -monitor stdio --enable-kvm --snapshot 

Mike, can we please try allocating some QE resources to get more precise statistics on host CPU usage?

Thanks,
Vadim.

I'm game for anything. Unfortunately I don't know what you mean by QE resources.

(In reply to Michael Bellerue from comment #66)
> I'm game for anything. Unfortunately I don't know what you mean by QE
> resources.

Vadim wrote this comment to Mike Cao from our QE team.
Thanks.

Mike Cao, and Vadim,

IIUC the suspects are "USB tables", viostor, and most of all the guest clocks
How about testing QEMU CPU (host side) with the following:
1. E1000+IDE
2. E1000+IDE+"USB tablet"
3. E1000+viostor (not virtio-SCSI)
4. NETKVM+viostor+"USB tablet"
Now Take the configuration that consumes the most CPU, and test the clocks
1. Disable HPET (guest)
2. Vadim, can you provide a host with iTSC  Enlightenment support

Feel free to improve this suggestion, Ronen.

(In reply to Vadim Rozenfeld from comment #34)
> (In reply to Alves from comment #33)
> > The answer is yes, I am 100% virtio.
> > recently I started using a new feature that brought CPU under control.
> >  <hyperv>
> >       <relaxed state='on'/>
> >       <vapic state='on'/>
> >       <spinlocks state='on' retries='4096'/>
> >     </hyperv>
> > 
> > but the sad side-effect is that my clock is roughly 1/2 of what it should
> > me. In one hour my internal clock thinks only one half has elapsed. The
> > virtual machine is virtually useless for all serious purposes, databases,
> > etc. I tried the hack 
> > localtime with bcdedit's useplatformclock inside guest
> > 
> > but it makes no difference. This Windows Server 2008 R2 Datacenter. I am
> > forced to look for a different solution for Windows virtualization. I want
> > to complain about Red Hat not taking seriously Windows Virtual machines,
> > while Microsoft does indeed take seriously virtualizing Linux.
> > If somebody has any idea how to pass the clock from the host to the guest,
> > please let me know.
> 
> Could it be that your VM is running with HPET enabled?

it shouldn't be ,HPET is disable in RHEL6.x

(In reply to Ronen Hod from comment #67)
> (In reply to Michael Bellerue from comment #66)
> > I'm game for anything. Unfortunately I don't know what you mean by QE
> > resources.
> 
> Vadim wrote this comment to Mike Cao from our QE team.
> Thanks.
> 
> Mike Cao, and Vadim,
> 
> IIUC the suspects are "USB tables", viostor, and most of all the guest clocks
> How about testing QEMU CPU (host side) with the following:

I also think usb-tablet should be blamed ,noting related to virtio-win

Following is my testing results:
                                               (percent of CPU usage on the host)
guest 0% CPU + VirtIO NIC + VirtIO-BLOCK + usb tablet  --->10%
guest 0% CPU + VirtIO NIC + VirtIO-BLOCK               --->3.7%

guest 50% CPU + VirtIO NIC + VirtIO-BLOCK + usb tablet --->108.1%
guest 50% CPU + VirtIO NIC + VirtIO-BLOCK              --->103.%


guest 0% CPU + e1000 NIC + IDE-BLOCK + usb tablet      --->10%
guest 0% CPU + e1000 NIC + IDE-BLOCK                   --->3.7%


BTW ,From https://bugzilla.redhat.com/show_bug.cgi?id=889222 there are some performance enhancement for this issue ,Is that possible to backport to RHEL6 ?
What's more I remember Jason talked this issue long time ago 

Mike

> -cpu qemu64,+x2apic,family=0xf,hv_vapic,hv_spinlocks=0xfff,hv_relaxed

Has libvirt support for these?
At least in the virt-manager gui I see x2apci only ...

(In reply to Gerd Hoffmann from comment #70)
> > -cpu qemu64,+x2apic,family=0xf,hv_vapic,hv_spinlocks=0xfff,hv_relaxed
> 
> Has libvirt support for these?
> At least in the virt-manager gui I see x2apci only ...

there is no hv_vapic hv_spinlocks support in RHEL6.x  

Mike

Test packages are at:
http://people.redhat.com/ghoffman/bz949547/

Note this isn't complete yet, it's a guest-visible change so we'll need some compat fluff so it'll be only activated for the rhel6.6.0 machine type.

With this windows should automatically suspend the usb tablet when idle, which should bring down the cpu load.  Best tested by doing a fresh windows install with the new qemu-kvm packages.  Windows caches usb device properties in the registry, therefore making this fly with existing windows images requires some manual registry fiddling.

Some more background info:
http://www.kraxel.org/blog/2014/03/qemu-and-usb-tablet-cpu-consumtion/

(In reply to Gerd Hoffmann from comment #73)
> Test packages are at:
> http://people.redhat.com/ghoffman/bz949547/
> 
> Note this isn't complete yet, it's a guest-visible change so we'll need some
> compat fluff so it'll be only activated for the rhel6.6.0 machine type.
> 
> With this windows should automatically suspend the usb tablet when idle,
> which should bring down the cpu load.  Best tested by doing a fresh windows
> install with the new qemu-kvm packages.  Windows caches usb device
> properties in the registry, therefore making this fly with existing windows
> images requires some manual registry fiddling.
> 
> Some more background info:
> http://www.kraxel.org/blog/2014/03/qemu-and-usb-tablet-cpu-consumtion/

Hi, Gerd

Do you mean we should see some results that bring down the cpu load in your private build even with the current "-M rhel6.5.0"?  And in future once the fixes go into official build, we need the new "-M rhel6.6.0" to activate the change? 

btw, do we plan to backport the bug 889222 to RHEL6? If so, we will clone the bug.

Thanks,
Qunfang

> Hi, Gerd
> 
> Do you mean we should see some results that bring down the cpu load in your
> private build even with the current "-M rhel6.5.0"?  And in future once the
> fixes go into official build, we need the new "-M rhel6.6.0" to activate the
> change? 

Exactly.

> btw, do we plan to backport the bug 889222 to RHEL6?

With remote wakeup being active in windows guests by default it should not be needed any more, so no.

(In reply to Gerd Hoffmann from comment #75)
> > Hi, Gerd
> > 
> > Do you mean we should see some results that bring down the cpu load in your
> > private build even with the current "-M rhel6.5.0"?  And in future once the
> > fixes go into official build, we need the new "-M rhel6.6.0" to activate the
> > change? 
> 
> Exactly.
> 
> > btw, do we plan to backport the bug 889222 to RHEL6?
> 
> With remote wakeup being active in windows guests by default it should not
> be needed any more, so no.

Okay, thanks for the feedback. We will give a test with your build soon.

Tried this issue according the comment #69. According to my testing usb-tablet should be blamed.

host info:
# uname -r && rpm -q qemu-kvm
2.6.32-447.el6.x86_64
qemu-kvm-0.12.1.2-2.423.el6.x86_64
guest info:
win7-64bit
virtio-win-prewhql-0.1-76

Following is my testing results:
usb-tablet not support the EHCI controller in rhel6, so here i just tried the with and without UHCI controller.

e.g:# /usr/libexec/qemu-kvm -M pc -enable-kvm -m 2048 -smp 2...-nodefconfig -nodefaults...-usb -device usb-tablet...
                                                                        *(%CPU)*
1.guest 0% CPU + VirtIO NIC + VirtIO-BLOCK + UHCI: usb-tablet      -----> 32.4
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22154 root      20   0 2696m 2.0g 5848 S 32.4  8.7   1:27.61 qemu-kvm  

2.guest 0% CPU + VirtIO NIC + VirtIO-BLOCK                         -----> 12.9
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22181 root      20   0 2697m 2.0g 5840 S 12.9  8.7   3:04.33 qemu-kvm 

3.guest 50% CPU + VirtIO NIC + VirtIO-BLOCK + UHCI: usb-tablet     -----> 144.4
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22267 root      20   0 2696m 2.1g 5852 R 144.4  8.8   4:52.90 qemu-kvm      

4.guest 50% CPU + VirtIO NIC + VirtIO-BLOCK                        -----> 120.6
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22321 root      20   0 2770m 2.0g 5840 S 120.6  8.7   4:43.04 qemu-kvm  

5.guest 0% CPU + e1000 NIC + IDE-BLOCK + UHCI: usb-tablet          -----> 24.5
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22005 root      20   0 2583m 2.0g 5848 S 24.5  8.7   3:18.44 qemu-kvm

6.guest 0% CPU + e1000 NIC + IDE-BLOCK                               ---->9.6
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22044 root      20   0 2583m 2.0g 5840 S  9.6  8.7   2:18.76 qemu-kvm

Best Regards,
sluo

(In reply to Sibiao Luo from comment #77)
> Tried this issue according the comment #69. According to my testing
> usb-tablet should be blamed.

> qemu-kvm-0.12.1.2-2.423.el6.x86_64

As expected, patches are not merged yet.
Can you check the test builds (see comment 73) too and compare please?

(In reply to Gerd Hoffmann from comment #78)
> (In reply to Sibiao Luo from comment #77)
> > Tried this issue according the comment #69. According to my testing
> > usb-tablet should be blamed.
> 
> > qemu-kvm-0.12.1.2-2.423.el6.x86_64
> 
> As expected, patches are not merged yet.
> Can you check the test builds (see comment 73) too and compare please?

Yes, thanks for your kindly reminds, I tried your private build as your instruction. 

host info:
# uname -r && rpm -q qemu-kvm
2.6.32-447.el6.x86_64
qemu-kvm-0.12.1.2-2.422.el6.bz949547.1.x86_64
guest info:
win7-64bit
virtio-win-prewhql-0.1-76

Following is my testing results:
usb-tablet not support the EHCI controller in rhel6, so here i just tried the with and without UHCI controller.

e.g:# /usr/libexec/qemu-kvm -M pc -enable-kvm -m 2048 -smp 2...-nodefconfig -nodefaults...-usb -device usb-tablet...
                                                                        *(%CPU)*
1.guest 0% CPU + VirtIO NIC + VirtIO-BLOCK + UHCI: usb-tablet    -------> 13.5
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22537 root      20   0 2695m 2.0g 5836 R 13.5  8.7   1:39.50 qemu-kvm 

2.guest 0% CPU + VirtIO NIC + VirtIO-BLOCK                       -------> 13.5
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22571 root      20   0 2696m 2.0g 5832 S 13.5  8.7   1:11.14 qemu-kvm 

3.guest 50% CPU + VirtIO NIC + VirtIO-BLOCK + UHCI: usb-tablet   -------> 126.6
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22537 root      20   0 2695m 2.0g 5836 S 126.6  8.7   2:18.94 qemu-kvm

4.guest 50% CPU + VirtIO NIC + VirtIO-BLOCK                      -------> 128.3
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22571 root      20   0 2696m 2.0g 5832 S 128.3  8.7   1:33.09 qemu-kvm  

5.guest 0% CPU + e1000 NIC + IDE-BLOCK + UHCI: usb-tablet        -------> 13.9
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22454 root      20   0 2588m 2.0g 5848 S 13.9  8.7   2:44.34 qemu-kvm  

6.guest 0% CPU + e1000 NIC + IDE-BLOCK                           -------> 12.9
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                      
22504 root      20   0 2584m 2.0g 5844 S 12.9  8.7   2:23.20 qemu-kvm 

According above test results, this issue has been fixed correctly. Please correct me if any mistake, thanks.

Best Regards,
sluo

Kicked new test build:
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=7324365

This time with the compatibility bits added.

Expected results:  With "-M rhel6.5.0" it should behave like rhel 6.5 qemu (same results as in comment #77).  With "-M rhel6.6.0" the bugfix is activated (should see same results as in comment #79).

Please verify.

And remember to use fresh windows installs (because of windows caching device properties in the registry).

(In reply to Gerd Hoffmann from comment #80)
> Kicked new test build:
> http://brewweb.devel.redhat.com/brew/taskinfo?taskID=7324365
> 
> This time with the compatibility bits added.
> 
> Expected results:  With "-M rhel6.5.0" it should behave like rhel 6.5 qemu
> (same results as in comment #77).  With "-M rhel6.6.0" the bugfix is
> activated (should see same results as in comment #79).
> 
> Please verify.
> 
> And remember to use fresh windows installs (because of windows caching
> device properties in the registry).

Okay. 

Hi, Sibiao

Could you give a help to verify the new build?  Please pay attention to the machine type and fresh guest image stuff.

Thanks,
Qunfang

(In reply to Gerd Hoffmann from comment #80)
> Kicked new test build:
> http://brewweb.devel.redhat.com/brew/taskinfo?taskID=7324365
> 
> This time with the compatibility bits added.
sorry to reply it later, but this patch has been closed, could you help recreate it again and i will try it, thanks.
> Expected results:  With "-M rhel6.5.0" it should behave like rhel 6.5 qemu
> (same results as in comment #77).  With "-M rhel6.6.0" the bugfix is
> activated (should see same results as in comment #79).
> 
> Please verify.
> 
> And remember to use fresh windows installs (because of windows caching
> device properties in the registry).

Best Regards,
sluo

new scratch build:
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=7356201

Verbose background info:

https://www.kraxel.org/blog/2014/03/qemu-and-usb-tablet-cpu-consumtion/

RHEL note: microsoft os descriptor patches are backported to rhel7.0 and planned for rhel6.6, therefore the pc-i440fx-rhel7.0.0+, pc-q35-rhel7.0.0+ and rhel6.6.0+ machine types will have support for this.

patches posted.

Dear Friends
I could not use RHEL7 beta because since there is no way to attach it to a subscription, yum cannot install any package. Whoever designed the RHEL beta needs to be fired. Any way, now I feel better. So I got Ubuntu 14.04, with Qemu 2.0 and moved my windows VM to this new box, made sure that this was achieved
https://www.kraxel.org/blog/2014/03/qemu-and-usb-tablet-cpu-consumtion/
I can attest that it works following the instructions, erasing the registry, etc.
Unfortunately, with 4 cpus as below, I still see 60% CPU outside as shown by "Top" versus 0% CPU inside. My Kernel is 3.15.0-031500rc4-generic
If some developer wants to log in and take a look, I am happy to help. The box is not in production and I take full responsibility. Until this is solved, KVM is not going to compete with Hyper-V or Vmware. 

qemu-system-x86_64 -enable-kvm -name Production -S -machine pc-i440fx-2.0,accel=kvm,usb=off -cpu kvm64,+rdtscp,+pdpe1gb,+x2apic,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,hv_relaxed,hv_vapic,hv_spinlocks=0xfff -m 4024 -realtime mlock=off -smp 4,sockets=1,cores=4,threads=1 -uuid e8701c5c-b542-0199-fd2a-1047df24770e -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Production.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/Production.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3a:d2:cd:ea,bus=pci.0,addr=0x3 -netdev tap,fd=35,id=hostnet1,vhost=on,vhostfd=36 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:70:fe:54,bus=pci.0,addr=0x4 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -device VGA,id=video0,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x5 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7

(In reply to Alves from comment #89)
> Dear Friends
> I could not use RHEL7 beta because since there is no way to attach it to a
> subscription, yum cannot install any package. Whoever designed the RHEL beta
> needs to be fired. Any way, now I feel better. So I got Ubuntu 14.04, with
> Qemu 2.0 and moved my windows VM to this new box, made sure that this was
> achieved
> https://www.kraxel.org/blog/2014/03/qemu-and-usb-tablet-cpu-consumtion/
> I can attest that it works following the instructions, erasing the registry,
> etc.
> Unfortunately, with 4 cpus as below, I still see 60% CPU outside as shown by
> "Top" versus 0% CPU inside. My Kernel is 3.15.0-031500rc4-generic
> If some developer wants to log in and take a look, I am happy to help. The
> box is not in production and I take full responsibility. Until this is
> solved, KVM is not going to compete with Hyper-V or Vmware. 
> 

Alves,

Please raise a ticket through your regular Red Hat support channels to make certain it receives the proper attention and prioritization to assure a timely resolution.

Thanks.

Also: thank you very much for doing whatever it took to test this, Alves!

Fix included in qemu-kvm-0.12.1.2-2.429.el6

Created attachment 919566 [details]
pic

Hello,
I am trying verify this bug,
using versions below:
kernel-2.6.32-488.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.430.el6.x86_64

but both rhel6.5.0 and rhel6.6.0 machine types showing not difference
CPU utilization(%) from "top" almost steadily at 4%

VM register database has been updated per(and rebooted)
from https://www.kraxel.org/blog/2014/03/qemu-and-usb-tablet-cpu-consumtion/

----------------
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\>reg DELETE HKLM\SYSTEM\CurrentControlSet\Control\usbflags
Permanently delete the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\usbflags (Yes/No)? Yes
The operation completed successfully.

C:\>reg DELETE HKLM\SYSTEM\CurrentControlSet\Enum\USB
Permanently delete the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USB (Yes/No)? Yes
ERROR: Access is denied.
----------------------------
and the 2nd cmd denied by OS(With Administrator privilige dont help)


cmd:
/usr/bin/qemu-kvm -monitor stdio \
    -S  \
    -name 'virt-tests-vm1' \
    -M rhel6.5.0  \                        ## tried rhel6.6.0 then rhel6.5.0
    -nodefaults  \
    -vga std \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20140721-235321-cV49PqXP,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140721-235321-cV49PqXP,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20140721-235321-cV49PqXP,path=/tmp/seabios-20140721-235321-cV49PqXP,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20140721-235321-cV49PqXP,iobase=0x402 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win7-64-sp1-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=05 \
    -device virtio-net-pci,mac=9a:2c:2d:2e:2f:30,id=idkX614i,vectors=4,netdev=iddJZBjl,bus=pci.0,addr=06  \
    -netdev tap,id=iddJZBjl,vhost=on  \
    -m 4096  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'SandyBridge',hv_relaxed \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \
    -device ide-drive,id=cd1,drive=drive_cd1,bootindex=1,bus=ide.0,unit=0 \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off \
    -enable-kvm  \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \




Is this expected and means the BZ fixed ?

Thanks,
Xiaoqing.

(In reply to Xiaoqing Wei from comment #94)
> Created attachment 919566 [details]

Good, so the guest configured correctly.

> but both rhel6.5.0 and rhel6.6.0 machine types showing not difference
> CPU utilization(%) from "top" almost steadily at 4%

The configuration caching done by windows goes both ways.  If you clear the registry + boot the guest with rhel6.6.0 windows will re-detect the device and turn on the power saving option, save it to the registry.  When you reboot the guest (without clearing the registry again) on rhel6.5.0 you'll get the savings too (also the "power management" tab) because windows uses the values cached on the rhel6.6.0 boot.  Which I think is the reason you see the same load on 6.5 and 6.6.

The *important* thing to test is that removing/adding the usb-tablet should not change the idle load much, with the power management being properly enabled for the tablet.  If you boot the same guest without usb tablet (and otherwise identical configuration) you should get 4% idle load too.

With usb-tablet connected and powermanagement disabled there should be noticable difference.  Exact value depends on the host of course, I'd expect at least 7%.  You should see this when simply unticking the "Allow the computer ..." checkbox.  Clearing the registry before booting into rhel6.5.0 should give simliar results.  Also the "power management" tab should be gone when booting into 6.5 with a cleared registry.

(In reply to Xiaoqing Wei from comment #94)

>     -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
>     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
> 

also tried commented out above two lines, result still 4% on verion 430.

Seems the -M rhel6.5.0 still has the fix

BTW, I tried 428(un-fix version).

the test result is similiar to C#85, cpu% much more higher when w/ usb tablet.
guest task manager: 0%    -> host top: 7%~25%(not so steadily).


Hello Gerd,

Could you pls have a look ?

Thx.

(In reply to Gerd Hoffmann from comment #95)

> The configuration caching done by windows goes both ways.  If you clear the

Hello Gerd,

Yes, the reg delete cmd in https://bugzilla.redhat.com/show_bug.cgi?id=949547#c94
are executed before every test.

and the 1st delete always finish w/o error,
but the 2nd delete alwyas got a denial msg.

now I only got reproducer on unfix build:
https://bugzilla.redhat.com/show_bug.cgi?id=949547#c96

Can you help to see whether my steps are correct ?

Thank you !

> and the 1st delete always finish w/o error,
> but the 2nd delete alwyas got a denial msg.

The registry paths in the command are correct.

I havn't used the reg command line utility but the regedit gui editor.  Got a error message too, but it was worded a bit different, more along the lines "some keys could not be deleted".  So maybe regedit and reg behave differently in the error case: regedit deletes everything it can, and reg aborts on the first error.

> now I only got reproducer on unfix build:
> https://bugzilla.redhat.com/show_bug.cgi?id=949547#c96

Hmm.  Looks like I have to dig a bit deeper.  unfixed version and fixed version with -M rhel6.5.0 should have the same behavior.

Hello Gerd,
no knowing if I understand the code correctly

hw/qdev.h
245 #define DEFINE_PROP_BIT(_name, _state, _field, _bit, _defval) {  \
246         .name      = (_name),                                    \
247         .info      = &(qdev_prop_bit),                           \
248         .bitnr    = (_bit),                                      \
249         .offset    = offsetof(_state, _field)                    \
250             + type_check(uint32_t,typeof_field(_state, _field)), \
251         .defval    = (bool[]) { (_defval) },                     \
252         }

usb-bus.c
 14 static struct BusInfo usb_bus_info = {
 15     .name      = "USB",
 16     .size      = sizeof(USBBus),
 17     .print_dev = usb_bus_dev_print,
 18     .get_dev_path = usb_get_dev_path,
 19     .get_fw_dev_path = usb_get_fw_dev_path,
 20     .props      = (Property[]) {
 21         DEFINE_PROP_STRING("port", USBDevice, port_path),
 22         DEFINE_PROP_UINT32("create_unique_serial", USBDevice, create_unique_serial, 1),
 23         DEFINE_PROP_BIT("msos-desc", USBDevice, flags,
 24                         USB_DEV_FLAG_MSOS_DESC_ENABLE, true),
 25         DEFINE_PROP_END_OF_LIST()
 26     },
 27 };

hw/pc.c
1626 #define PC_RHEL6_5_COMPAT \
1627         {\
1628             .driver   = "USB",\
1629             .property = "msos-desc",\
1630             .value    = "off",\
1631         }
                             ^^^^^ this value should be Bool as the qdev.h and usb-bus.c defined ?

if I am wrong, just ignore my comment :-)


Best Regards,
Xiaoqing Wei.

  Hi,

> hw/pc.c
> 1626 #define PC_RHEL6_5_COMPAT \
> 1627         {\
> 1628             .driver   = "USB",\
> 1629             .property = "msos-desc",\
> 1630             .value    = "off",\
> 1631         }
>                              ^^^^^ this value should be Bool as the qdev.h
> and usb-bus.c defined ?
> 
> if I am wrong, just ignore my comment :-)

I've checked out the property too, so you are digging in the correct direction.

The .value goes through an parser which translates strings into bool, so that isn't the bug.

Setting the property according to the machine type works correctly in my testing:

[root@rincewind ~]# /usr/libexec/qemu-kvm --version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2-2.430.el6), Copyright (c) 2003-2008 Fabrice Bellard
[root@rincewind ~]# (sleep 1; echo "info qtree"; sleep 1; echo "quit") | /usr/libexec/qemu-kvm -M rhel6.6.0 -usb -device usb-tablet -monitor stdio | grep "msos"
            bus-prop: msos-desc = on
[root@rincewind ~]# (sleep 1; echo "info qtree"; sleep 1; echo "quit") | /usr/libexec/qemu-kvm -M rhel6.5.0 -usb -device usb-tablet -monitor stdio | grep "msos"
            bus-prop: msos-desc = off
[root@rincewind ~]# 

Puzzling.

Could you try with a fresh windows install?  Best is probably to do the install without usb tablet connected.  Then use that image (which has never seen a usb-tablet device) as readonly base image and do the individual tests with qcow2 snapshots of that base image.  That way the tests can be done without installing over and over again and also without error-prove registry fiddeling.

(In reply to Gerd Hoffmann from comment #100)

> Could you try with a fresh windows install?  Best is probably to do the
> install without usb tablet connected.  Then use that image (which has never
> seen a usb-tablet device) as readonly base image and do the individual tests
> with qcow2 snapshots of that base image.  That way the tests can be done
> without installing over and over again and also without error-prove registry
> fiddeling.

:-( using a fresh installed vm(no using any usb device during install):
(qemu) info usb
USB support not enabled

then delete the registry usd cmd as before, then shutdown

uncomment 
#    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \
#    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
and boot, after vm says driver installed, check the device manger, hid dont have a power management tab, reboot wont help.

win7 64

do you like to access the vm ?

(In reply to Xiaoqing Wei from comment #101)
> (In reply to Gerd Hoffmann from comment #100)
> 
> > Could you try with a fresh windows install?  Best is probably to do the
> > install without usb tablet connected.  Then use that image (which has never
> > seen a usb-tablet device) as readonly base image and do the individual tests
> > with qcow2 snapshots of that base image.  That way the tests can be done
> > without installing over and over again and also without error-prove registry
> > fiddeling.
> 
> :-( using a fresh installed vm(no using any usb device during install):
> (qemu) info usb
> USB support not enabled

As expected.

> then delete the registry usd cmd as before, then shutdown

Should not be needed if the vm has never seen the usb-tablet.

Looks like I should have explained the snapshot idea in more detail...
At this point you don't change the disk image of the fresh win7 install any more.  You can create qcow2 files with the fresh install as backing storage, this way:

qemu-img create -f qcow2 -o backing_file=win7-fresh.img win7-test.img

Then use -drive file=win7-test.img for testing.  Any disk writes done by the guest will go to win7-test.img, win7-fresh.img will not be modified.

So you can:
  (1) boot win7 with -M rhel6.5.0, check device manager + cpu load.
  (2) delete and re-create win7-test.img, giving you a fresh windows
      install without actually reinstalling.
  (3) boot win7 with -M rhel6.6.0, check device manager + cpu load.

> uncomment 
> #    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \
> #    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
> and boot, after vm says driver installed, check the device manger, hid dont
> have a power management tab, reboot wont help.

Which machine type?  6.5 or 6.6?

(In reply to Gerd Hoffmann from comment #102)

> 
> Looks like I should have explained the snapshot idea in more detail...
> At this point you don't change the disk image of the fresh win7 install any
> more.  You can create qcow2 files with the fresh install as backing storage,
> this way:
> 
> qemu-img create -f qcow2 -o backing_file=win7-fresh.img win7-test.img
> 
> Then use -drive file=win7-test.img for testing.  Any disk writes done by the
> guest will go to win7-test.img, win7-fresh.img will not be modified.

Yes, I did used snapshots, I am using -b option, but I dont think this makes different.

  205  qemu-img create -f qcow2 -b /root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win7-64-sp1-virtio.qcow2 /home/w7-on-6.6.qcow2
  206  qemu-img create -f qcow2 -b /root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win7-64-sp1-virtio.qcow2 /home/w7-on-6.5.qcow2



> 
> So you can:
>   (1) boot win7 with -M rhel6.5.0, check device manager + cpu load.
>   (2) delete and re-create win7-test.img, giving you a fresh windows
>       install without actually reinstalling.
>   (3) boot win7 with -M rhel6.6.0, check device manager + cpu load.

Yea, the w7-on-6.5.qcow2 is booted w/ -M rhel6.5.0,
and w7-on-6.6.qcow2 w/ -M rhel6.6.0.



> 
> > uncomment 
> > #    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \
> > #    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
> > and boot, after vm says driver installed, check the device manger, hid dont
> > have a power management tab, reboot wont help.
> 
> Which machine type?  6.5 or 6.6?

Both :-(

also tried del registry entries by 'reg delete' cmd as above(bit to bit identical, by copy-and-paste,so not typo is made).

(In reply to Xiaoqing Wei from comment #103)

Oh, forgot to tell the good news:

on the fresh installed img(actually a snapshot to the fresh img).
with -M rhel6.5.0, %CPU is steadily at 14%, which seems disabled the fix.

Oh, I've got the desired result on -M rhel6.6.0 !!!!

no knowing why this way could get thing right, but it does.
trying login the VM to delete the registry by GUI, not remote cmd.
open "regedit", then delete the entries we want, though error still happen,

but reboot takes the effect ! "Power management" appears,
and after several min of observation, the %CPU is reduced to 4~5% from host side, while guest Task Manager shown 0%, the expected result

output as below(threads shown)

top -Hp `pidof qemu-kvm` -n 1
Tasks:   5 total,   0 running,   5 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  0.1%sy,  0.0%ni, 99.4%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   7936488k total,  7758112k used,   178376k free,    75076k buffers
Swap: 58720248k total,    14044k used, 58706204k free,  3135004k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                       
 2691 root      20   0 4884m 4.0g 5792 S  4.0 52.5   1:08.02 qemu-kvm                                                                                                                                      
 2711 root      20   0 4884m 4.0g 5792 S  2.0 52.5   0:39.28 qemu-kvm                                                                                                                                      
 2708 root      20   0 4884m 4.0g 5792 S  0.0 52.5   0:55.20 qemu-kvm                                                                                                                                      
 2709 root      20   0 4884m 4.0g 5792 S  0.0 52.5   0:38.35 qemu-kvm                                                                                                                                      
 2710 root      20   0 4884m 4.0g 5792 S  0.0 52.5   0:27.91 qemu-kvm

So the patch works, setting verified.

on host:
kernel-2.6.32-491.el6.x86_64
qemu-kvm-0.12.1.2-2.430.el6.x86_64

boot the VM and wait for its fully initialized, then open the "Task Manager" inside.
and "top -Hp `pidof qemu-kvm` " on host to check the %CPU utilization

-M rhel6.6.0:
    with usb tablet: 4~5%
    w/o  usb tablet: 4~5%

-M rhel6.5.0:
    with usb tablet: 14~15%
    w/o  usb tablet: 5~7%

Thx Gerd for the fix

So, based on comment #93, the fix is in: qemu-kvm-0.12.1.2-2.429.el6

However, I can't figure out if the fix is also in the latest stable version of qemu, which is qemu 2.0.2. Is it?

Ever if it is, how can I get qemu-kvm-0.12.1.2-2.429.el6? When I go to:http://sourceforge.net/projects/kvm/files/qemu-kvm/0.12.1.2/

it only has reverence to 0.12.1.2 (not 0.12.1.2-2....); moreover, the date modified time is for the year 2009 and even after downloading and building that version, nothing indicates that the version has changed to 0.12.1.2-2....

The upstream version of QEMU lives at http://qemu-project.org/

If you have difficulties updating your RHEL-6 machines to qemu-kvm-0.12.1.2-2.429.el6 or later, please contact Red Hat technical support.

(In reply to Markus Armbruster from comment #108)
> The upstream version of QEMU lives at http://qemu-project.org/
> 
Ok, so I'm assuming that the later versions have this fix too?

> If you have difficulties updating your RHEL-6 machines to
> qemu-kvm-0.12.1.2-2.429.el6 or later, please contact Red Hat technical
> support.

Would this fix also appear in the centos updates or is this strictly a RedHat fix?

(In reply to Gaurav Saxena from comment #107)
> So, based on comment #93, the fix is in: qemu-kvm-0.12.1.2-2.429.el6
> 
> However, I can't figure out if the fix is also in the latest stable version
> of qemu, which is qemu 2.0.2. Is it?

Latest stable upstream release is 2.1.2 not 2.0.2.
Fix is included in upstream 2.0 & newer.

> Ever if it is, how can I get qemu-kvm-0.12.1.2-2.429.el6?

RHEL-6.6 will ship a qemu-kvm version with the fix.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1490.html

hitting this in 2018 on rhel 7.5 and ovirt (qemu 2.10).
installed spice guest tools and kept the tablet, usage dropped down to 3-4%. switched from i440 to q35 chipset and gone down to 2.7%