Red Hat Bugzilla – Bug 903123
The value of steal time in "top" command always is "0.0% st" after guest migration
Last modified: 2013-11-21 01:30:44 EST
Description of problem: In source, boot two guests and bond them to a same cpu. Add 100% cpu load in both guests, the value of steal time are both about "50% st". In destination, migrate the two guests from source and also bond them to a same cpu. After migration, the value of steal time in both destination guest always is "0.0% st". Version-Release number of selected component (if applicable): # uname -r 2.6.32-355.el6.x86_64 # rpm -qa | grep qemu-kvm qemu-kvm-0.12.1.2-2.352.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1. In source, boot two guest and bond them to a same cpu: # taskset -c 5 /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 1 -name rhel6.4 -uuid 6afa5f93-2d4f-420f-81c6-e5fdddbd1c83 -drive file=rhel6-64.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:42:0c:00,bus=pci.0,addr=0x3 -vnc :1 -monitor stdio QEMU 0.12.1 monitor - type 'help' for more information (qemu) # taskset -c 5 /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 1 -name rhel6.4 -uuid 6afa5f93-2d4f-420f-81c6-e5fdddbd1c83 -drive file=rhel6-64_2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:42:0d:00,bus=pci.0,addr=0x3 -vnc :2 -monitor stdio QEMU 0.12.1 monitor - type 'help' for more information (qemu) 2. Add 100% cpu load in both guest running the following command: # for ((;;)) do x=1; done 3. In both two guests, use "top" command to check the steal time 4. In destination, migrate the two guests and also bond them to a same cpu: #taskset -c 5 /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 1 -name rhel6.4 -uuid 6afa5f93-2d4f-420f-81c6-e5fdddbd1c83 -drive file=rhel6-64_2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:42:0d:00,bus=pci.0,addr=0x3 -vnc :3 -monitor stdio -incoming tcp:0:5800 QEMU 0.12.1 monitor - type 'help' for more information (qemu) # taskset -c 5 /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 1 -name rhel6.4 -uuid 6afa5f93-2d4f-420f-81c6-e5fdddbd1c83 -drive file=rhel6-64_2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:42:0d:00,bus=pci.0,addr=0x3 -vnc :4 -monitor stdio -incoming tcp:0:5801 QEMU 0.12.1 monitor - type 'help' for more information (qemu) 5. In both two guests, use "top" command to check the steal time Actual results: Step 3: the steal time of both guests is correct. In both guests, "top" command shows about "50% st". Please refer to "guest_1_src.png" and "guest_2_src.png". Step 5: After migration, the steal time of both guests are "0.0% st". Please refer to "guest_1_dst.png" and "guest_2_dst.png". Expected results: The steal time in Step 3 and Step5 should be about "50% st" showed in top command.
Created attachment 685748 [details] The result of top command in the first source guest.
Created attachment 685750 [details] The result of top command in the second source guest.
Created attachment 685751 [details] After migration, the result of top command in the first guest.
Created attachment 685752 [details] After migration, the result of top command in the second guest.
"qemu-kvm -M rhel6.4.0" has also this problem.
Before migration, in source guest, we check /proc/stat twice: # cat /proc/stat cpu 34653 0 2602 1196 748 4 0 33433 0 cpu0 34653 0 2602 1196 748 4 0 33433 0 # cat /proc/stat cpu 37273 0 2766 1196 748 4 0 37123 0 cpu0 37273 0 2766 1196 748 4 0 37123 0 We can see the steal time is updated from 33433 to 37123. After migration, in destination guest, we check /proc/stat twice, found that steal time is not updated and always "42168". # cat /proc/stat cpu 56982 0 4128 1196 748 8 16 42168 0 cpu0 56982 0 4128 1196 748 8 16 42168 0 # cat /proc/stat cpu 62499 0 4506 1196 748 8 16 42168 0 cpu0 62499 0 4506 1196 748 8 16 42168 0
This should not have anything to do with qemu-kvm, IIRC steal time is calculated entirely between the guest and the host kernel with no qemu intervention.
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate, in the next release of Red Hat Enterprise Linux.
(In reply to comment #9) > This should not have anything to do with qemu-kvm, IIRC steal time is > calculated entirely between the guest and the host kernel with no qemu > intervention. Don't see MSR_KVM_STEAL_TIME being saved/restored in qemu-kvm, as its done for other kvmclock MSRs, in target-i386/kvm.c: #endif msrs[n++].index = MSR_KVM_SYSTEM_TIME; msrs[n++].index = MSR_KVM_WALL_CLOCK; if (has_msr_async_pf_en) {
Reproduce with qemu-kvm-0.12.1.2-2.392.el6.x86_64 Steps to Reproduce: 1. In source, boot two guest and bond them to a same cpu: taskset -c 5 /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 1 -name rhel6.51 -uuid 6afa5f93-2d4f-420f-81c6-e5fdddbd1c83 -drive file=/mnt/RHEL6.5-64-IDE.raw,if=none,id=drive-virtio-disk0,format=raw,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:42:0c:00,bus=pci.0,addr=0x3 -vnc :1 -monitor stdio QEMU 0.12.1 monitor - type 'help' for more information taskset -c 5 /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 1 -name rhel6.52 -uuid 6afa5f93-2d4f-420f-81c6-e5fdddbd1c83 -drive file=/mnt/RHEL222.raw,if=none,id=drive-virtio-disk0,format=raw,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:42:0d:00,bus=pci.0,addr=0x3 -vnc :2 -monitor stdio QEMU 0.12.1 monitor - type 'help' for more information 2. Add 100% cpu load in both guest running the following command: # for ((;;)) do x=1; done 3. In both two guests, use "top" command to check the steal time 4. In destination, migrate the two guests and also bond them to a same cpu: taskset -c 5 /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 1 -name rhel6.51 -uuid 6afa5f93-2d4f-420f-81c6-e5fdddbd1c83 -drive file=/mnt/RHEL6.5-64-IDE.raw,if=none,id=drive-virtio-disk0,format=raw,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:42:0c:00,bus=pci.0,addr=0x3 -vnc :1 -monitor stdio -incoming tcp:0:5888 QEMU 0.12.1 monitor - type 'help' for more information taskset -c 5 /usr/libexec/qemu-kvm -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 1 -name rhel6.52 -uuid 6afa5f93-2d4f-420f-81c6-e5fdddbd1c83 -drive file=/mnt/RHEL222.raw,if=none,id=drive-virtio-disk0,format=raw,serial=40c061dd-5d60-4fc5-865f-55db700407f0,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:42:0d:00,bus=pci.0,addr=0x3 -vnc :2 -monitor stdio -incoming tcp:0:6999 5. In both two guests, use "top" command to check the steal time Actual results: Step 3: the steal time of both guests is correct. In both guests, "top" command shows about "50% st". Step 5: After migration, the steal time of both guests are "0.0% st". Expected results: The steal time in Step 3 and Step5 should be about "50% st" showed in top command. Verify this issue with qemu-kvm-0.12.1.2-2.407.el6.x86_64 Actual results: The steal time in Step 3 and Step5 are about "50% st" showed in top command. Based on above information, so this issue has been fixed.
Possibly related: bug 1022821
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-1553.html