Description of problem: When trying to install hvm/pv rhel5.3 guests on a rhel5.3 ia64 dom0, even though there is enough free memory on the system, installation fails with memory allocations error: [root@hp-merlion-01 ~]# uname -a Linux hp-merlion-01.rhts.bos.redhat.com 2.6.18-117.el5xen #1 SMP Mon Sep 29 22:46:20 EDT 2008 ia64 ia64 ia64 GNU/Linux [root@hp-merlion-01 ~]# free total used free shared buffers cached Mem: 3849216 623600 3225616 0 30400 241536 -/+ buffers/cache: 351664 3497552 Swap: 2031584 0 2031584 [root@hp-merlion-01 ~]# virt-install --name rhel5.3_pv_guest --location nfs:bigpapi.bos.redhat.com:/vol/engineering/redhat/rel-eng/RHEL5.3-Server-20081001.0/5/ia64/os --nonsparse --paravirt --file /var/lib/xen/images/rhel5.3_pv_guest.img -s 10 -r 512 --nographics Starting install... virDomainCreateLinux() failed POST operation failed: (xend.err 'Error creating domain: (4, \'Out of memory\', "xc_dom_boot_mem_init: can\'t allocate low memory for domain\\n")') Domain installation may not have been successful. If it was, you can restart your domain by running 'virsh start rhel5.3_pv_guest'; otherwise, please restart your installation. Mon, 06 Oct 2008 09:19:25 ERROR virDomainCreateLinux() failed POST operation failed: (xend.err 'Error creating domain: (4, \'Out of memory\', "xc_dom_boot_mem_init: can\'t allocate low memory for domain\\n")') Traceback (most recent call last): File "/usr/sbin/virt-install", line 559, in ? main() File "/usr/sbin/virt-install", line 491, in main dom = guest.start_install(conscb, progresscb, wait=(not wait)) File "/usr/lib/python2.4/site-packages/virtinst/Guest.py", line 822, in start_install return self._do_install(consolecb, meter, wait) File "/usr/lib/python2.4/site-packages/virtinst/Guest.py", line 843, in _do_install self.domain = self.conn.createLinux(install_xml, 0) File "/usr/lib/python2.4/site-packages/libvirt.py", line 573, in createLinux if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self) libvirtError: virDomainCreateLinux() failed POST operation failed: (xend.err 'Error creating domain: (4, \'Out of memory\', "xc_dom_boot_mem_init: can\'t allocate low memory for domain\\n")') [root@hp-merlion-01 ~]# free total used free shared buffers cached Mem: 3543040 644992 2898048 0 32080 258128 -/+ buffers/cache: 354784 3188256 Swap: 2031584 0 2031584 [root@hp-merlion-01 ~]# # virsh list Id Name State ---------------------------------- 0 Domain-0 running libvir: Xen error : Domain not found: xenUnifiedDomainLookupByID Version-Release number of selected component (if applicable): # uname -a Linux hp-merlion-01.rhts.bos.redhat.com 2.6.18-117.el5xen #1 SMP Mon Sep 29 22:46:20 EDT 2008 ia64 ia64 ia64 GNU/Linux # rpm -qa | grep xen kmod-gnbd-xen-0.1.5-2.el5 kernel-xen-2.6.18-117.el5 kmod-gfs2-xen-1.116-1.12.el5 xen-devel-3.0.3-73.el5 kmod-cmirror-xen-0.1.18-1.el5 xen-ia64-guest-firmware-1.0.13-1 xen-libs-3.0.3-73.el5 xen-3.0.3-73.el5 kernel-xen-devel-2.6.18-117.el5 kmod-gfs-xen-0.1.26-1.el5 How reproducible: Everytime. Steps to Reproduce: 1. Get an ia64 box, install 5.3 dom0 and try to install guest. I can reliably reproduce this issue on hp-merlion-01.rhts.bos.redhat.com which is a part of rhts so you can use it to reproduce as well. Additional info: Here is some info from the xend log file: [2008-10-06 23:24:14 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name', 'rhel5.3_hvm_guest'], ['memory', '512'], ['maxmem', '512'], ['vcpus', '1'], ['uuid', 'edf77845-2357-cbe6-54d3-3138c66f400d'], ['on_poweroff', 'destroy'], ['on_reboot', 'destroy'], ['on_crash', 'destroy'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'], ['device_model', '/usr/lib/xen/bin/qemu-dm'], ['vcpus', '1'], ['boot', 'd'], ['acpi', '1'], ['apic', '1'], ['usb', '1'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1']]], ['device', ['vbd', ['dev', 'hda:disk'], ['uname', 'file:/var/lib/xen/images/rhel5.3_hvm_guest.img'], ['mode', 'w']]], ['device', ['vbd', ['dev', 'hdc:cdrom'], ['uname', 'file:/var/lib/xen/images/rhel5.3_ia64.iso'], ['mode', 'r']]], ['device', ['vif', ['mac', '00:16:3e:01:88:fd'], ['bridge', 'xenbr0']]]]) [2008-10-06 23:24:14 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:312) parseConfig: config is ['vm', ['name', 'rhel5.3_hvm_guest'], ['memory', '512'], ['maxmem', '512'], ['vcpus', '1'], ['uuid', 'edf77845-2357-cbe6-54d3-3138c66f400d'], ['on_poweroff', 'destroy'], ['on_reboot', 'destroy'], ['on_crash', 'destroy'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'], ['device_model', '/usr/lib/xen/bin/qemu-dm'], ['vcpus', '1'], ['boot', 'd'], ['acpi', '1'], ['apic', '1'], ['usb', '1'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1']]], ['device', ['vbd', ['dev', 'hda:disk'], ['uname', 'file:/var/lib/xen/images/rhel5.3_hvm_guest.img'], ['mode', 'w']]], ['device', ['vbd', ['dev', 'hdc:cdrom'], ['uname', 'file:/var/lib/xen/images/rhel5.3_ia64.iso'], ['mode', 'r']]], ['device', ['vif', ['mac', '00:16:3e:01:88:fd'], ['bridge', 'xenbr0']]]] [2008-10-06 23:24:14 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:417) parseConfig: result is {'shadow_memory': None, 'start_time': None, 'uuid': 'edf77845-2357-cbe6-54d3-3138c66f400d', 'on_crash': 'destroy', 'on_reboot': 'destroy', 'localtime': None, 'image': ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'], ['device_model', '/usr/lib/xen/bin/qemu-dm'], ['vcpus', '1'], ['boot', 'd'], ['acpi', '1'], ['apic', '1'], ['usb', '1'], ['serial', 'pty'], ['vnc', '1'], ['vncunused', '1']], 'on_poweroff': 'destroy', 'bootloader_args': None, 'cpus': None, 'name': 'rhel5.3_hvm_guest', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 512, 'device': [('vbd', ['vbd', ['dev', 'hda:disk'], ['uname', 'file:/var/lib/xen/images/rhel5.3_hvm_guest.img'], ['mode', 'w']]), ('vbd', ['vbd', ['dev', 'hdc:cdrom'], ['uname', 'file:/var/lib/xen/images/rhel5.3_ia64.iso'], ['mode', 'r']]), ('vif', ['vif', ['mac', '00:16:3e:01:88:fd'], ['bridge', 'xenbr0']])], 'bootloader': None, 'cpu': None, 'maxmem': 512} [2008-10-06 23:24:14 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:1358) XendDomainInfo.construct: None [2008-10-06 23:24:14 xend 3093] DEBUG (balloon:143) Balloon: 524512 KiB free; need 2048; done. [2008-10-06 23:24:14 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:1406) XendDomainInfo.initDomain: 4 1.0 [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: boot, val: d [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: fda, val: None [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: fdb, val: None [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: soundhw, val: None [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: localtime, val: None [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: serial, val: pty [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: std-vga, val: None [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: isa, val: None [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: vcpus, val: 1 [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: acpi, val: 1 [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: usb, val: 1 [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: usbdevice, val: None [2008-10-06 23:24:14 xend 3093] DEBUG (image:328) args: k, val: None [2008-10-06 23:24:14 xend 3093] DEBUG (image:388) No VNC passwd configured for vfb access [2008-10-06 23:24:14 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:1445) _initDomain:shadow_memory=0x0, maxmem=0x200, memory=0x200. [2008-10-06 23:24:14 xend 3093] DEBUG (balloon:149) Balloon: 524432 KiB free; 0 to scrub; need 540736; retries: 20. [2008-10-06 23:24:14 xend 3093] DEBUG (balloon:164) Balloon: setting dom0 target to 3444 MiB. [2008-10-06 23:24:14 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:1087) Setting memory target of domain Domain-0 (0) to 3444 MiB. [2008-10-06 23:24:14 xend 3093] DEBUG (balloon:143) Balloon: 540816 KiB free; need 540736; done. [2008-10-06 23:24:14 xend 3093] INFO (image:137) buildDomain os=None dom=4 vcpus=1 [2008-10-06 23:24:14 xend 3093] INFO (image:500) Creating NVRAM file UUID record for rhel5.3_hvm_guest: edf77845-2357-cbe6-54d3-3138c66f400d [2008-10-06 23:24:14 xend 3093] DEBUG (image:273) domid = 4 [2008-10-06 23:24:14 xend 3093] DEBUG (image:274) image = /usr/lib/xen/boot/hvmloader [2008-10-06 23:24:14 xend 3093] DEBUG (image:275) store_evtchn = 1 [2008-10-06 23:24:14 xend 3093] DEBUG (image:276) memsize = 512 [2008-10-06 23:24:14 xend 3093] DEBUG (image:277) vcpus = 1 [2008-10-06 23:24:14 xend 3093] DEBUG (image:278) pae = 0 [2008-10-06 23:24:14 xend 3093] DEBUG (image:279) acpi = 1 [2008-10-06 23:24:14 xend 3093] DEBUG (image:280) apic = 1 [2008-10-06 23:24:16 xend.XendDomainInfo 3093] ERROR (XendDomainInfo:212) Domain construction failed Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 205, in create vm.initDomain() File "/usr/lib/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1482, in initDomain raise VmError(str(exn)) VmError: (1, 'Internal error', 'Error constructing guest OS') [2008-10-06 23:24:16 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:1568) XendDomainInfo.destroy: domid=4 [2008-10-06 23:24:16 xend.XendDomainInfo 3093] DEBUG (XendDomainInfo:1576) XendDomainInfo.destroyDomain(4) [2008-10-06 23:24:17 xend 3093] ERROR (SrvBase:88) Request create failed. Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/xen/web/SrvBase.py", line 85, in perform return op_method(op, req) File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line 82, in op_create raise XendError("Error creating domain: " + str(ex)) XendError: Error creating domain: (1, 'Internal error', 'Error constructing guest OS') [2008-10-06 23:27:42 xend 3093] DEBUG (XendDomain:208) Cannot recreate information for dying domain 4. Xend will ignore this domain from now on. [2008-10-06 23:27:49 xend 3093] DEBUG (XendDomain:208) Cannot recreate information for dying domain 4. Xend will ignore this domain from now on. [2008-10-06 23:40:28 xend 3093] DEBUG (XendDomain:208) Cannot recreate information for dying domain 4. Xend will ignore this domain from now on. [2008-10-06 23:41:46 xend 3093] DEBUG (XendDomain:208) Cannot recreate information for dying domain 4. Xend will ignore this domain from now on.
Here is the patch Intel mentioned in their previous update: Attach Intel developer's patch for this issue. For each vcpu of xen/ia64, it requires 16M contiguous memory for vhpt, but balloon driver didn't consider this case when it balloons memory. Signed-off-by Xiantao Zhang <xiantao.zhang> --- XendDomainInfo.py.old 2008-11-25 05:55:34.000000000 -0500 +++ XendDomainInfo.py 2008-11-25 06:57:05.000000000 -0500 @@ -1450,8 +1450,14 @@ # set memory limit xc.domain_setmaxmem(self.domid, maxmem) + # Ensure hypervisor can allocate 16M contiguous memory for vcpu's vhpt. + if arch.type == "ia64": + extra_balloon_kb = self.image.vm.getVCpuCount() * (64 * 1024) + else: + extra_balloon_kb = 0 + # Make sure there's enough RAM available for the domain - balloon.free(memory + shadow) + balloon.free(memory + shadow + extra_balloon_kb) # Set up the shadow memory shadow_cur = xc.shadow_mem_control(self.domid, shadow / 1024)
Couple things... first of, i couldn't get workaround with dom0_mem working... I did this: image=vmlinuz-2.6.18-125.el5xen vmm=xen.gz-2.6.18-125.el5 label=linux initrd=initrd-2.6.18-125.el5xen.img read-only root=/dev/VolGroup00/LogVol00 append="dom0_mem=3500M -- " Yet, i still got memory allocation errors when trying to create guests. Second of, I have tried out the patch gcase posted, and i verified to solve my problem. Is it too late to pull this in one of the 5.3 trees we'll be building? Thanks.
This bugzilla has Keywords: Regression. Since no regressions are allowed between releases, it is also being proposed as a blocker for this release. Please resolve ASAP.
Tetsu, can you see comment #4 from Gurhan?
I can not see comment #4. Is it a private comment?
Tetsu, changed the permissions you should be able to see it now.
Thanks. And if possible, could you change the permission so that I can see the patch you mentioned on comment #4?
Gurhan, I can not reproduce the problem on Fujitsu PRIMEQUEST. So could you provide me the xend.log? I'd like to see the log in the each case of hvm and pv. And could you let me on which boxes you reproduced the problem?
I'm making the patch public.
Tetsu, Here are the information you have requested. First of , I am booting the box with dom0_mem argument: # cat /etc/elilo.conf prompt timeout=20 default=linux relocatable image=vmlinuz-2.6.18-125.el5xen vmm=xen.gz-2.6.18-125.el5 label=linux initrd=initrd-2.6.18-125.el5xen.img read-only root=/dev/VolGroup00/LogVol00 append="dom0_mem=3500M -- " # free total used free shared buffers cached Mem: 3504128 2779248 724880 0 39104 2313296 -/+ buffers/cache: 426848 3077280 Swap: 6094816 352 6094464 With this setting, hvm installations still error with memory allocation error: # virt-install --name rhel5.3_hvm_guest2 --cdrom /var/lib/xen/images/rhel5.3_hvm_guest.iso --nonsparse --hvm --file /var/lib/xen/images/rhel5.3_hvm_guest2.img -s 6 --extra-args ks=http://lab.rhts.bos.redhat.com/kickstarts/hosts/guest-81-142.rhts.bos.redhat.com/ks.cfg --noreboot How much RAM should be allocated (in megabytes)? 512 Would you like to enable graphics support? (yes or no) yes Starting install... Creating storage file... | 6.0 GB 01:28 virDomainCreateLinux() failed POST operation failed: (xend.err "Error creating domain: (12, 'Cannot allocate memory')") Domain installation may not have been successful. If it was, you can restart your domain by running 'virsh start rhel5.3_hvm_guest2'; otherwise, please restart your installation. Sun, 14 Dec 2008 03:20:20 ERROR virDomainCreateLinux() failed POST operation failed: (xend.err "Error creating domain: (12, 'Cannot allocate memory')") Traceback (most recent call last): File "/usr/sbin/virt-install", line 560, in ? main() File "/usr/sbin/virt-install", line 492, in main dom = guest.start_install(conscb, progresscb, wait=(not wait)) File "/usr/lib/python2.4/site-packages/virtinst/Guest.py", line 822, in start_install return self._do_install(consolecb, meter, wait) File "/usr/lib/python2.4/site-packages/virtinst/Guest.py", line 843, in _do_install self.domain = self.conn.createLinux(install_xml, 0) File "/usr/lib/python2.4/site-packages/libvirt.py", line 573, in createLinux if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self) libvirtError: virDomainCreateLinux() failed POST operation failed: (xend.err "Error creating domain: (12, 'Cannot allocate memory')") I need to clarify something... PV guests install ok, but hvm guests do not. I will be attaching xend.log s for each PV and HVM installations, along with dmesg, cpuinfo, dmidecode info on the box.
Created attachment 326850 [details] dmesg
Created attachment 326851 [details] dmidecode
Created attachment 326852 [details] cpuinfo
Created attachment 326853 [details] xend.log during hvm install
Created attachment 326854 [details] xend.log during pv install
Created attachment 326855 [details] xend-debug.log during hvm install
Ping Tetsu, Any update on this?
It seems to need a fix for hvm domain, so I'm asking the detail of the patch on comment #3 to Intel.
Intel suggests the following workarounds: 1. specify dom0_mem=3000M to leave space for creating guest. Or, 2. Use "xm mem-set 1024M" to trim down Dom0 before creating guest And they say the patch on comment #3 can fix this problem, so I'll work on it for 5.4.
Created attachment 334020 [details] proposed patch
Hi, I have created a patch for BZ #250762 that should correct some memory allocation issues for this bug as well. Allocating 64M extra memory for IA64 FV guest is not enough when using 128G physical memory machine with 100G allocated to dom0 and trying to allocate 28G to domU. Patch has been sent to Tetsu and we've having e-mail conversation about changing extra memory allocation so if having any troubles when using Tetsu's patch, please use my patch posted in BZ #250762. Michal
Sorry, this is still failing with 5.4 nightly trees: [root@hp-merlion-01 ~]# free total used free shared buffers cached Mem: 3525632 3465264 60368 0 121472 2914208 -/+ buffers/cache: 429584 3096048 Swap: 6094816 0 6094816 [root@hp-merlion-01 ~]# virt-install --name rhel4_ia64_hvm_guest_2 --cdrom /var/lib/xen/images/rhel4_ia64_hvm_guest.iso --nonsparse --hvm --file /var/lib/xen/images/rhel4_ia64_hvm_guest_2.img -s 5 --extra-args ks=http://lab.rhts.bos.redhat.com/kickstarts/hosts/guest-81-58.rhts.bos.redhat.com/ks.cfg --prompt --noreboot How much RAM should be allocated (in megabytes)? 512 Starting install... Creating storage file... | 5.0 GB 01:23 POST operation failed: xend_post: error from xen daemon: (xend.err "Error creating domain: (12, 'Cannot allocate memory')") Domain installation may not have been successful. If it was, you can restart your domain by running 'virsh start rhel4_ia64_hvm_guest_2'; otherwise, please restart your installation. ERROR POST operation failed: xend_post: error from xen daemon: (xend.err "Error creating domain: (12, 'Cannot allocate memory')") Traceback (most recent call last): File "/usr/sbin/virt-install", line 865, in ? main() File "/usr/sbin/virt-install", line 763, in main start_time, guest.start_install) File "/usr/sbin/virt-install", line 818, in do_install dom = install_func(conscb, progresscb, wait=(not wait)) File "/usr/lib/python2.4/site-packages/virtinst/Guest.py", line 541, in start_install return self._do_install(consolecb, meter, removeOld, wait) File "/usr/lib/python2.4/site-packages/virtinst/Guest.py", line 633, in _do_install self.domain = self.conn.createLinux(install_xml, 0) File "/usr/lib/python2.4/site-packages/libvirt.py", line 974, in createLinux if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self) libvirtError: POST operation failed: xend_post: error from xen daemon: (xend.err "Error creating domain: (12, 'Cannot allocate memory')") [root@hp-merlion-01 ~]# rpm -qa | egrep "xen|libvirt" xen-libs-3.0.3-86.el5 rh-tests-virt-xen-migrate-2.0-15 xen-ia64-guest-firmware-1.0.13-1 libvirt-0.6.3-4.el5 xen-3.0.3-86.el5 kernel-xen-devel-2.6.18-151.el5 libvirt-python-0.6.3-4.el5 kernel-xen-2.6.18-151.el5 rh-tests-virt-libvirt-virsh-1.0-29
The patch for this BZ and bug 250762 were collapsed together at some point. Making this one a dup of the latter. *** This bug has been marked as a duplicate of bug 250762 ***
This bug was closed during 5.5 development and it's being removed from the internal tracking bugs (which are now for 5.6).