RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 740221 - it took too long time to boot when starting guest windows 2003 with 512G memory
Summary: it took too long time to boot when starting guest windows 2003 with 512G memory
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.6
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Marcelo Tosatti
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1094855
TreeView+ depends on / blocked
 
Reported: 2011-09-21 10:32 UTC by FuXiangChun
Modified: 2014-05-06 15:40 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1094855 (view as bug list)
Environment:
Last Closed: 2014-01-08 19:01:09 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
memory usage for win2003 (34.57 KB, image/png)
2013-03-21 01:30 UTC, FuXiangChun 		no flags Details
View All

Description FuXiangChun 2011-09-21 10:32:22 UTC 
Description of problem:
 guest need about 15 minutes to boot if boot guest with 512G memory. it will take more time if boot guest with 1T memory.. 

Version-Release number of selected component (if applicable):
host info:
# uname -r
2.6.32-195.el6.x86_64
# rpm -qa|grep kvm
qemu-kvm-0.12.1.2-2.189.el6.x86_64

#cat /proc/meminfo
memory 1071644672 kB
#cat /proc/cpuinfo
cpu 160 core(processor    : 159)

guest info:
windows server 2003 enterprise x64 Edition

How reproducible:
always

Steps to Reproduce:
1./usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -name win2003 -smp 64 -m 512G -uuid 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -drive file=/home/win2003.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none,aio=threads -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=22,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:19:3b:80,bus=pci.0,addr=0x4 -balloon none -monitor stdio -spice id=on,disable-ticketing,port=5912
2.
3.
  
Actual results:
spend about 15 minutes to boot

Expected results:
guest boot successfully within 5 minutes

Additional info:
guest windows 2008 only need 5 minutes to boot
Comment 2 Andrew Cathrow 2011-09-21 12:33:33 UTC 
Do we know how long a physical Windows 2003 server in that configuration would take to boot?

This BZ compares win2k8 and win2k3 we need to compare physical to virtual.
Comment 3 Dor Laor 2011-09-22 08:26:40 UTC 
Does your hardware support nested paging?
Can you try without spice (w/ vnc)? We had issues in the past with mapping of the memory slots of spice. Lastly, win2k8R2 is more important than win2k3 which is very old
Comment 4 FuXiangChun 2011-09-22 10:41:19 UTC 
(In reply to comment #2)
> Do we know how long a physical Windows 2003 server in that configuration would
> take to boot?
> 
> This BZ compares win2k8 and win2k3 we need to compare physical to virtual.

I am finding machine with 512G memory.  i will update testing result to bz as soon as possible.
Comment 6 FuXiangChun 2011-12-15 03:48:26 UTC 
(In reply to comment #3)
> Does your hardware support nested paging?
> Can you try without spice (w/ vnc)? We had issues in the past with mapping of
> the memory slots of spice. Lastly, win2k8R2 is more important than win2k3 which
> is very old

Hi Dor
   physical host support nested paging, without spice have the same issue too. 
 In addition,win2008-64 have the same issue too. it take 1.5 hours to boot completely.
Comment 7 Dor Laor 2012-01-10 09:23:31 UTC 
Marcelo, it looks it worth to test for 6.3 since its uses nested paging.
Comment 11 Marcelo Tosatti 2012-03-21 00:44:26 UTC 
The host has 504GB memory, starting a 512G guest exceeds its RAM size. Since Win2003 clears memory on startup, a 512G guest on this host goes deep into swap.

Booting guest with 480GB takes 5 minutes.

Closing as NOTABUG.
Comment 12 FuXiangChun 2012-03-21 01:43:04 UTC 
(In reply to comment #11)
> The host has 504GB memory, starting a 512G guest exceeds its RAM size. Since
> Win2003 clears memory on startup, a 512G guest on this host goes deep into
> swap.
> 
> Booting guest with 480GB takes 5 minutes.
> 
> Closing as NOTABUG.

If I can reproduce this bug in 1T host then I need to reopen it?
Comment 13 Marcelo Tosatti 2012-03-22 02:57:39 UTC 
(In reply to comment #12)
> (In reply to comment #11)
> > The host has 504GB memory, starting a 512G guest exceeds its RAM size. Since
> > Win2003 clears memory on startup, a 512G guest on this host goes deep into
> > swap.
> > 
> > Booting guest with 480GB takes 5 minutes.
> > 
> > Closing as NOTABUG.
> 
> If I can reproduce this bug in 1T host then I need to reopen it?

Yes, you can then reopen, but please make sure the host is not swapping (see vmstat -a 1 output or cat /proc/swaps).
Comment 14 FuXiangChun 2013-01-28 09:06:05 UTC 
Since still can reproduce it in 1T memory big machine. So re-open this bug. 

This are detail testing steps:

1. reboot host - (in order to let all applications to release memory)

2. check free memory and swap in host

# free -g
             total       used       free     shared    buffers     cached
Mem:           991          5        986          0          0          0
-/+ buffers/cache:          5        986
Swap:            3          0          3

# cat /proc/swaps(no use swap) 
Filename			Type		Size	 Used	Priority
/dev/dm-1                       partition	4194296	  0	-1


3. boot win2003-64 guest with 512G memory and 64 cores vcpus

result:
Guest will take >20 mins to boot.

host info:
cpu:
processor	: 159
vendor_id	: GenuineIntel
cpu family	: 6
model		: 47
model name	: Intel(R) Xeon(R) CPU E7- 2860  @ 2.27GHz

memory:
# cat /proc/meminfo
MemTotal:       1040136796 kB(1T)
Comment 19 FuXiangChun 2013-02-05 07:12:42 UTC 
If host is rhel6.4, please use -M rhel6.4.0
Comment 20 Marcelo Tosatti 2013-02-19 23:34:31 UTC 
FuXiangChun,

Profiled Windows 2003 guest boot on the machine with perf, thanks for setting it up.

Windows is clearing all of the memory during bootup (512G). There is little KVM interference, so it should take approximately the same amount of time to boot Windows 2003 on real HW.

Unless it can be shown that on real hardware Windows takes significantly less time to boot with such large amount of memory, this ticket should be closed as NOTABUG.
Comment 21 FuXiangChun 2013-02-20 10:26:09 UTC 
(In reply to comment #20)
> FuXiangChun,
> 
> Profiled Windows 2003 guest boot on the machine with perf, thanks for
> setting it up.
> 
> Windows is clearing all of the memory during bootup (512G). There is little
> KVM interference, so it should take approximately the same amount of time to
> boot Windows 2003 on real HW.
> 
> Unless it can be shown that on real hardware Windows takes significantly
> less time to boot with such large amount of memory, this ticket should be
> closed as NOTABUG.

I will reserve a 512G memory machine(since hasn't 1T memory machine in local) to install windows 2003, and update testing result to bz.
Comment 25 FuXiangChun 2013-03-20 11:01:08 UTC 
Tested this issue on 512G bare machine. windows2003 os only took 2 minutes to boot. and os work well.  

host:512G memory
cpu:48 cores ,PowerEdge R815
Comment 28 FuXiangChun 2013-03-21 01:30:44 UTC 
Created attachment 713555 [details]
memory usage for win2003
Comment 31 Ademar Reis 2013-04-17 21:24:44 UTC 
Please retest with RHEL7 packages (or upstream). If the problem persists, we'll move this bug to RHEL7. If it's solved, we'll close it as WONTFIX in RHEL6 due to capacity.
Comment 32 FuXiangChun 2013-04-18 08:09:41 UTC 
I think RHEL7.0 still has this issue.  It took 8 minutes to boot guest. If boot guest in 1T host. It will take more time.

Host: AMD 6172(Memory 512G, cpu: 48 cores)

# uname -r
3.9.0-0.rc6.51.el7.x86_64
qemu-kvm-1.4.0-2.el7.x86_64

Boot win2003 64bit guest with (free -g) memory
/usr/libexec/qemu-kvm -M pc-i440fx-1.4 -enable-kvm -m 504G -smp 4,sockets=2,cores=2,threads=1 -name windows -uuid 745fe449-aac8-29f1-0c2d-5042a707263b -nodefconfig -nodefaults -drive file=/home/win2003-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-blk-pci0 -usb -device usb-tablet,id=input0 -vnc :0 -vga qxl -monitor stdio
Comment 35 Marcelo Tosatti 2013-08-14 02:43:25 UTC 
(In reply to FuXiangChun from comment #32)
> I think RHEL7.0 still has this issue.  It took 8 minutes to boot guest. If
> boot guest in 1T host. It will take more time.
> 
> Host: AMD 6172(Memory 512G, cpu: 48 cores)
> 
> # uname -r
> 3.9.0-0.rc6.51.el7.x86_64
> qemu-kvm-1.4.0-2.el7.x86_64
> 
> Boot win2003 64bit guest with (free -g) memory
> /usr/libexec/qemu-kvm -M pc-i440fx-1.4 -enable-kvm -m 504G -smp
> 4,sockets=2,cores=2,threads=1 -name windows -uuid
> 745fe449-aac8-29f1-0c2d-5042a707263b -nodefconfig -nodefaults -drive
> file=/home/win2003-64-virtio.qcow2,if=none,id=drive-virtio-disk0,
> format=qcow2 -device
> virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-blk-pci0 -usb -device
> usb-tablet,id=input0 -vnc :0 -vga qxl -monitor stdio

Is it possible to perform the test with -cpu host?
Comment 36 FuXiangChun 2013-08-14 11:18:02 UTC 
I　don't reserve AMD 6172 host. Maybe need to update result later.  I have contacted the current user of host. I will start to test this issue as soon as owner of host released machine.
Comment 38 FuXiangChun 2014-05-06 15:20:35 UTC 
(In reply to Ademar Reis from comment #31)
> Please retest with RHEL7 packages (or upstream). If the problem persists,
> we'll move this bug to RHEL7. If it's solved, we'll close it as WONTFIX in
> RHEL6 due to capacity.

Adermar,

Reproduced this issue with 1T memory & 48 cores on AMD machine (not found 1T memory amd machine before). According to comment31 & comment32 & comment37. For RHEL7.0 host, Is this issue handled the same as RHEL6 host?  Do QE need to clone this bug to RHEL7.0? 

Another, I tested several key points and checked host's memory usage.
1. Tested -M host & AMD machine 

2. Boot RHEL7.0 guest with -m 512G

3. free -g on host
# free -g
             total       used       free     shared    buffers     cached
Mem:           994        463        531          0          0         14
-/+ buffers/cache:        449        545
Swap:            3          0          3

4.# cat /proc/swaps 
Filename				Type		Size	Used	Priority
/dev/dm-0                               partition	4194300	0	-1

5.host info
processor	: 47
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 2
model name	: AMD Opteron(tm) Processor 6344

6.hostname
hp-dl585g7-01.rhts.eng.nay.redhat.com

7. kernel & qemu version
3.10.0-121.el7.x86_64 & qemu-kvm-1.5.3-60.el7.x86_64
Note You need to log in before you can comment on or make changes to this bug.