Hide Forgot
Description of problem: start a windows server 2008 dc guest with -m 400G -cpu 64,the progress of booting up spent me more than 25mins Version-Release number of selected component (if applicable): # uname -r 2.6.32-122.el6.x86_64 [root@intel-x7550-512-1 ~]# rpm -q qemu-kvm qemu-kvm-0.12.1.2-2.151.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1.start a windows guest with -m 400G -cpu 64, eg:perf record -a -g /usr/libexec/qemu-kvm -enable-kvm -m 400G -smp 64,sockets=4,cores=4,threads=4 -name rhel5U6 -uuid ddcbfb49-3411-1701-3c36-6bdbc00bedba -rtc base=localtime,clock=host,driftfix=slew -boot c -drive file=/mnt/mike1/win08_64/win08_64,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-ide0-0-0,id=drive-ide0-0-0 -netdev tap,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=32:12:50:a4:32:71 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc :5 -device virtio-balloon-pci,id=ballooning -monitor stdio Actual results: the bootup of windows costs more or less 25mins Expected results: the bootup of windows should less than 10mins Additional info: # cat /sys/module/kvm_intel/parameters/ept Y host info: # free -g total used free shared buffers cached Mem: 441 167 273 0 0 14 -/+ buffers/cache: 153 287 Swap: 32 0 32 #cat /proc/cpuinfo processor : 63 vendor_id : GenuineIntel cpu family : 6 model : 46 model name : Intel(R) Xeon(R) CPU X7550 @ 2.00GHz stepping : 6 cpu MHz : 1994.907 cache size : 18432 KB physical id : 3 siblings : 16 core id : 11 cpu cores : 8 apicid : 119 initial apicid : 119 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 x2apic popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid bogomips : 3989.53 clflush size : 64 cache_alignment : 64 address sizes : 44 bits physical, 48 bits virtual power management:
Perf data shows high access rate to cr4 register. Please provide trace of only CR accesses. To do that set trace event like that: "echo kvm:kvm_cr > set_event"
This is probably cr4.pge being toggled by the guest. Upstream should already be fast here.
(In reply to comment #9) > This is probably cr4.pge being toggled by the guest. Upstream should already > be fast here. That is what I am trying to confirm with a trace. If this is the case and backport is not trivial probably we should push this to rhel7.
(In reply to comment #10) > (In reply to comment #9) > > This is probably cr4.pge being toggled by the guest. Upstream should already > > be fast here. > > That is what I am trying to confirm with a trace. If this is the case and > backport is not trivial probably we should push this to rhel7. Yes bit 7 of cr4 (CR4.PGE) is flipped to often.
Can't we add this to vmx_vcpu_setup? Or will this break stuff without more changes to vmx? --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -2434,7 +2434,8 @@ static int vmx_vcpu_setup(struct vcpu_vmx *vmx) vmcs_write32(VM_ENTRY_CONTROLS, vmcs_config.vmentry_ctrl); vmcs_writel(CR0_GUEST_HOST_MASK, ~0UL); - vmcs_writel(CR4_GUEST_HOST_MASK, KVM_GUEST_CR4_MASK); + vmcs_writel(CR4_GUEST_HOST_MASK, + KVM_GUEST_CR4_MASK & ~(enable_ept ? X86_CR4_PGE : 0)); kvm_write_tsc(&vmx->vcpu, 0);
I meet this issue as well when testing RHEL6.1 SVVP. And seems it is related the option "vhost=on" Host: AMD, 512G mem, 48 cpus Boot a win2k8-r2 guest with 256G mem and 32 smp, with vhost=on, guset nearly hangs and can not boot up after half an hour. Boot with the same v-mem and vcpu, but withouth vhost, guest can boot up successfully within few minutes. Command line: /usr/libexec/qemu-kvm -m 256G -smp 32 -cpu cpu64-rhel6,+x2apic,family=0xf -usbdevice tablet -drive file=win2k8-r2-sp1.raw,format=raw,if=none,id=drive-virtio0,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio0,id=virtio-blk-pci0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,mac=00:10:20:1a:2a:41,bus=pci.0,addr=0x4 -boot c -uuid 1d4dfe1b-39ed-4d3e-881e-a2b400b63d54 -rtc-td-hack -no-kvm-pit-reinjection -monitor stdio -name win2k8-r2-amd-sut -vnc :10
(In reply to comment #14) > I meet this issue as well when testing RHEL6.1 SVVP. And seems it is related > the option "vhost=on" > Host: AMD, 512G mem, 48 cpus > Boot a win2k8-r2 guest with 256G mem and 32 smp, with vhost=on, guset nearly > hangs and can not boot up after half an hour. > Boot with the same v-mem and vcpu, but withouth vhost, guest can boot up > successfully within few minutes. > Looks like totally unrelated issue to me. Open separate bug against vhost.
(In reply to comment #13) > Can't we add this to vmx_vcpu_setup? Or will this break stuff without more > changes to vmx? > > > --- a/arch/x86/kvm/vmx.c > +++ b/arch/x86/kvm/vmx.c > @@ -2434,7 +2434,8 @@ static int vmx_vcpu_setup(struct vcpu_vmx *vmx) > vmcs_write32(VM_ENTRY_CONTROLS, vmcs_config.vmentry_ctrl); > > vmcs_writel(CR0_GUEST_HOST_MASK, ~0UL); > - vmcs_writel(CR4_GUEST_HOST_MASK, KVM_GUEST_CR4_MASK); > + vmcs_writel(CR4_GUEST_HOST_MASK, > + KVM_GUEST_CR4_MASK & ~(enable_ept ? X86_CR4_PGE : 0)); > > kvm_write_tsc(&vmx->vcpu, 0); Not that simple. Following commit should be backported: 87bc3bf972af0585ba5415aebbc8bd09b6a2ee94 ce03e4f21a33b97c5d20ce597f64c361bb247904 4c38609ac569483152f9cb7e5a66f17355e563b2 fc78f51938e1ea866daa2045851b2e5681371668 May be more.
For the Actual Results in comment #0: booting up windows 2008 64 bit on this host only need 2mins (w/o host hardware checking)
(In reply to comment #14) > I meet this issue as well when testing RHEL6.1 SVVP. And seems it is related > the option "vhost=on" > Host: AMD, 512G mem, 48 cpus > Boot a win2k8-r2 guest with 256G mem and 32 smp, with vhost=on, guset nearly > hangs and can not boot up after half an hour. > Boot with the same v-mem and vcpu, but withouth vhost, guest can boot up > successfully within few minutes. > > Command line: > > /usr/libexec/qemu-kvm -m 256G -smp 32 -cpu cpu64-rhel6,+x2apic,family=0xf > -usbdevice tablet -drive > file=win2k8-r2-sp1.raw,format=raw,if=none,id=drive-virtio0,cache=none,werror=stop,rerror=stop > -device virtio-blk-pci,drive=drive-virtio0,id=virtio-blk-pci0 -netdev > tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device > virtio-net-pci,netdev=hostnet0,mac=00:10:20:1a:2a:41,bus=pci.0,addr=0x4 -boot c > -uuid 1d4dfe1b-39ed-4d3e-881e-a2b400b63d54 -rtc-td-hack -no-kvm-pit-reinjection > -monitor stdio -name win2k8-r2-amd-sut -vnc :10 Please open another bug for it, it might not be related. To prove whether it is related to vhost or not (contradicts w/ the current kvm stat we have), we'll need to redo the above tests.
Referring to https://bugzilla.redhat.com/show_bug.cgi?id=691704
(In reply to comment #13) > Can't we add this to vmx_vcpu_setup? Or will this break stuff without more > changes to vmx? > > @@ -2434,7 +2434,8 @@ static int vmx_vcpu_setup(struct vcpu_vmx *vmx) > vmcs_write32(VM_ENTRY_CONTROLS, vmcs_config.vmentry_ctrl); > > vmcs_writel(CR0_GUEST_HOST_MASK, ~0UL); > - vmcs_writel(CR4_GUEST_HOST_MASK, KVM_GUEST_CR4_MASK); > + vmcs_writel(CR4_GUEST_HOST_MASK, > + KVM_GUEST_CR4_MASK & ~(enable_ept ? X86_CR4_PGE : 0)); > > kvm_write_tsc(&vmx->vcpu, 0); No, that kills save/restore of cr4.pge.
Please also try the original kernel, but set vpid=0: # rmmod kvm-intel # modprobe kvm-intel vpid=0
(In reply to comment #24) > Please also try the original kernel, but set vpid=0: > > # rmmod kvm-intel > # modprobe kvm-intel vpid=0 Tried on qemu-kvm-0.12.1.2-2.160.el6.x86_64 2.6.32-156.el6.x86_64 Results: the time of guest booing up is less than 5 mins Tried on qemu-kvm-0.12.1.2-2.160.el6.x86_64 2.6.32-156.el6.x86_64 with modprobe kvm-intel vpid=0 The time of guest booting up is less than 4 mins bcao--->avi any idea why the booting time improved a lot ? the scratch builds is expired ,could you rebuild it for me to test ?
Nothing relevant changed. Can you still reproduce with the original kernel?
(In reply to comment #27) > Nothing relevant changed. > > Can you still reproduce with the original kernel? Tried on 2.6.32-122.el6.x86_64 ,still can reproduce.the time of booting up costs more than 20mins . BTW ,does the booting time (5 mins ) on kernel 2.6.32-156.el6.x86_64 is acceptable ?Do we have Optimization to make the time of booting smaller ?
Interesting. How long does it take to boot on bare metal?
(In reply to comment #29) > Interesting. > > How long does it take to boot on bare metal? Referring to comment #17 ,less than 2mins
Marking as resolved; while 5 minutes isn't perfect, at least it's the same order of magnitude as bare metal.