Description of problem: Currently Nova does a copy of disk images in /var/lib/nova/instances/_base, but if we fallocate the space first we can improve performance of the copies and also we'll know quicker if the disk has enough space so we can error out sooner.
Performance of the copy is not improved, but of subsequent running of the VM, since it will not need to extend the image bit by bit.
(In reply to comment #1) > Performance of the copy is not improved, but of subsequent running of the VM, > since it will not need to extend the image bit by bit. Why would anyone extend the base image? Aren't _base just the base templates on top of which we perform a qcow2 snapshot to run a VM? I'd try to copy to a raw sparse image.
To test this, ensure that preallocate_images=space in nova.conf and after booting an instance, run du on the instances/$instance/disk file to ensure it's fully allocated
@Yaniv you're right, the merged upstream patch https://review.openstack.org/#/c/22054/ preallocates the $instance/disk as this is what will be extended at run time
Just a note here - the config directive 'preallocate' should be documented in nova.conf #-----------------------# [tuser1@interceptor export(keystone_admin)]$ grep -i preallocate -A4 /usr/lib/python2.6/site-packages/nova/virt/driver.py cfg.StrOpt('preallocate_images', default='none', help='VM image preallocation mode: ' '"none" => no storage provisioning is done up front, ' '"space" => storage is fully allocated at instance start'), [tuser1@interceptor export(keystone_admin)]$ #-----------------------# Search for the string in nova.conf #-----------------------# [tuser1@interceptor export(keystone_admin)]$ sudo grep -i preallocate /etc/nova/* [tuser1@interceptor export(keystone_admin)]$ #-----------------------# Version Info: #-----------------------# [tuser1@interceptor export(keystone_admin)]$ rpm -q openstack-nova openstack-nova-2012.2.3-3.el6ost.noarch [tuser1@interceptor export(keystone_admin)]$ #-----------------------# [tuser1@interceptor export(keystone_admin)]$ rpm -q openstack-nova --changelog | grep 891347 - Support preallocated VM images #891347 [tuser1@interceptor export(keystone_admin)]$ #-----------------------#
I don't see the qcow2 overlay (location, for ex: /var/lib/nova/instances/instance-00000019/disk) being used is preallocated. Am I missing anything trivial here ? Some details below: # Add the directive 'preallocate_images=space' in nova.conf & restart all services. (just nova is sufficient, actually) #-----------------------# [tuser1@interceptor export(keystone_admin)]$ sudo grep preallocate /etc/nova/nova.conf # preallocate images -- 891347 preallocate_images=space [tuser1@interceptor export(keystone_admin)]$ #-----------------------# [root@interceptor ~(keystone_admin)]# for j in `for i in $(ls -1 /etc/init.d/openstack-*) ; do $i status | grep running ; done | awk '{print $1}'` ; do service $j restart ; done #-----------------------# # boot a new instance [tuser1@interceptor ~(keystone_user1)]$ #nova boot --flavor 1 --key_name oskey --image 1e6292f9-82bd-4cdb-969e-c863cb1c6692 fed-t1 # list, and check it's running [tuser1@interceptor ~(keystone_user1)]$ nova list +--------------------------------------+-----------+---------+-------------------+ | ID | Name | Status | Networks | +--------------------------------------+-----------+---------+-------------------+ | 54ed73b5-4628-4dae-87bf-5f59a39f094b | fed-t1 | ACTIVE | net1=10.65.207.51 | | bb494526-e993-4adc-a453-902a5a9f530d | fedora-t8 | SHUTOFF | net1=10.65.207.53 | +--------------------------------------+-----------+---------+-------------------+ [tuser1@interceptor ~(keystone_user1)]$ # do a virsh list [tuser1@interceptor ~(keystone_user1)]$ sudo virsh list Id Name State ---------------------------------------------------- 29 instance-00000019 running [tuser1@interceptor ~(keystone_user1)]$ # find the block device in use [tuser1@interceptor ~(keystone_user1)]$ sudo virsh domblklist instance-00000019 Target Source ------------------------------------------------ vda /var/lib/nova/instances/instance-00000019/disk [tuser1@interceptor ~(keystone_user1)]$ # find the disk image (the qcow2 overlay in use) info [tuser1@interceptor ~(keystone_user1)]$ sudo qemu-img info /var/lib/nova/instances/instance-00000019/disk image: /var/lib/nova/instances/instance-00000019/disk file format: qcow2 virtual size: 9.8G (10486808576 bytes) disk size: 8.8M cluster_size: 65536 backing file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 [tuser1@interceptor ~(keystone_user1)]$ # Also find the info of the backing file [tuser1@interceptor ~(keystone_user1)]$ sudo qemu-img info /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 image: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 file format: raw virtual size: 9.8G (10486808576 bytes) disk size: 738M [tuser1@interceptor ~(keystone_user1)]$
Yes "disk size:" should match the virtual size here. Are there any warnings or errors from the fallocate calls in the logs? If not can you enable debugging to see the fallocate calls being executed? One possibility is that instances/ is on NFS or some other file system that doesn't support fallocate?
Created attachment 704148 [details] debugging info
(In reply to comment #11) > Created attachment 704148 [details] > debugging info After some more investigation with Pádraig, it appears 'fallocate' cmd is being prevented to run. Manually test to fallocate an overlay: - shutoff the nova instance - find the block device of the overlay - and fallocate the overlay to the virtual size (10486808576 bytes) #-----------------------# [tuser1@interceptor ~(keystone_user1)]$ sudo virsh domblklist instance-00000018 Target Source ------------------------------------------------ vda /var/lib/nova/instances/instance-00000018/disk #-----------------------# [tuser1@interceptor ~(keystone_user1)]$ ls -lash /var/lib/nova/instances/instance-00000018/disk 9.8M -rw-r--r--. 1 root root 9.9M Mar 1 13:05 /var/lib/nova/instances/instance-00000018/disk #-----------------------# [tuser1@interceptor ~(keystone_user1)]$ qemu-img info /var/lib/nova/instances/instance-00000018/disk image: /var/lib/nova/instances/instance-00000018/disk file format: qcow2 virtual size: 9.8G (10486808576 bytes) disk size: 9.8M cluster_size: 65536 backing file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 #-----------------------# [tuser1@interceptor ~(keystone_user1)]$ sudo fallocate -n -l 10486808576 /var/lib/nova/instances/instance-00000018/disk fallocate: /var/lib/nova/instances/instance-00000018/disk: fallocate failed: No space left on device [tuser1@interceptor ~(keystone_user1)]$ #-----------------------# [tuser1@interceptor ~(keystone_user1)]$ qemu-img info /var/lib/nova/instances/instance-00000018/disk image: /var/lib/nova/instances/instance-00000018/disk file format: qcow2 virtual size: 9.8G (10486808576 bytes) disk size: 9.1G cluster_size: 65536 backing file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 [tuser1@interceptor ~(keystone_user1)]$ #-----------------------# It can be seen now, the disk has 9.1G allocated blocks: #-----------------------# [tuser1@interceptor ~(keystone_user1)]$ sudo ls -lash /var/lib/nova/instances/instance-00000018/disk 9.1G -rw-r--r--. 1 root root 9.9M Mar 1 13:05 /var/lib/nova/instances/instance-00000018/disk [tuser1@interceptor ~(keystone_user1)]$ #-----------------------# To check the size inside the instance, start the nova instance; ssh into it, and run 'df -hT' to check the file system space: #-----------------------# [tuser1@interceptor ~(keystone_user1)]$ ssh -i oskey.priv root.207.53 reverse mapping checking getaddrinfo for dhcp207-53.lab.eng.pnq.redhat.com [10.65.207.53] failed - POSSIBLE BREAK-IN ATTEMPT! [root@localhost ~]# [root@localhost ~]# df -hT Filesystem Type Size Used Avail Use% Mounted on rootfs rootfs 9.3G 881M 8.4G 10% / devtmpfs devtmpfs 239M 0 239M 0% /dev tmpfs tmpfs 247M 0 247M 0% /dev/shm tmpfs tmpfs 247M 784K 246M 1% /run /dev/vda2 ext4 9.3G 881M 8.4G 10% / tmpfs tmpfs 247M 0 247M 0% /sys/fs/cgroup tmpfs tmpfs 247M 0 247M 0% /media [root@localhost ~]# #-----------------------#
(In reply to comment #12) > After some more investigation with Pádraig, it appears 'fallocate' cmd is > being prevented to run. ... > [tuser1@interceptor ~(keystone_user1)]$ sudo fallocate -n -l 10486808576 > /var/lib/nova/instances/instance-00000018/disk > fallocate: /var/lib/nova/instances/instance-00000018/disk: fallocate failed: > No space left on device What's the conclusion here, you don't have enough space on your machine or is it something else?
(In reply to comment #13) > (In reply to comment #12) > > After some more investigation with Pádraig, it appears 'fallocate' cmd is > > being prevented to run. > ... > > [tuser1@interceptor ~(keystone_user1)]$ sudo fallocate -n -l 10486808576 > > /var/lib/nova/instances/instance-00000018/disk > > fallocate: /var/lib/nova/instances/instance-00000018/disk: fallocate failed: > > No space left on device > > What's the conclusion here, Sorry for not summarizing at the end. Conclusion so far is: 'fallocate' command isn't being invoked OpenStack. This needs further investigation. > you don't have enough space on your machine or > is it something else? It's something else. That 'No space left on device' isn't accurate. If you followed the command after fallocate: #-----------------------# [tuser1@interceptor ~(keystone_user1)]$ sudo ls -lash /var/lib/nova/instances/instance-00000018/disk 9.1G -rw-r--r--. 1 root root 9.9M Mar 1 13:05 /var/lib/nova/instances/instance-00000018/disk [tuser1@interceptor ~(keystone_user1)]$ #-----------------------# The "9.9M" is supposed to be the file size, while the "9.1G" is the allocated space on disk. Essentially, we've allocated beyond the end of the file (as we invoked fallocate using -n)
Worked through this with Kashyap today. I'm fairly sure it's working as expected. That was observed above is that fallocation is not done when using a flavor with a 0 disk size. fallocation was seen to be in place when using a flavor with disk size set.
Thanks Pádraig. Just summarizing it here. Summary: -------- 'fallocate' is invoked successfully only for flavor-2 ('small') or higher, as they have elastic FS (meaning, nova can resize the FS of the image). However, flavor-1 images ('tiny') has 'Disk - 0' which means no scope for elastic FS). That can be noticed below: Test info: --------- [1] Boot flavor 2 ('small') instance [tuser1@interceptor glance(keystone_user1)]$ nova boot --flavor 2 --key_name oskey --image 1e6292f9-82bd-4cdb-969e-c863cb1c6692 test9-fed [2] Run nova list [tuser1@interceptor glance(keystone_user1)]$ nova list +--------------------------------------+------------+---------+-------------------+ | ID | Name | Status | Networks | +--------------------------------------+------------+---------+-------------------+ | 8dc43c0d-519a-4238-84c7-29159f6d26ff | test9-fed | ACTIVE | net1=10.65.207.53 | +--------------------------------------+------------+---------+-------------------+ [tuser1@interceptor glance(keystone_user1)]$ [3] Find out the disk image in use [tuser1@interceptor glance(keystone_user1)]$ sudo virsh domblklist instance-0000002f Target Source ------------------------------------------------ vda /export/nova/instances/instance-0000002f/disk [4] Find qemu-img info of the disk (it can be noted here, that disk size *and* virtual size are both same [tuser1@interceptor glance(keystone_user1)]$ sudo qemu-img info /export/nova/instances/instance-0000002f/disk image: /export/nova/instances/instance-0000002f/disk file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 20G cluster_size: 65536 backing file: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 [5] Search for 'fallocate' in nova compute log: ====== [root@interceptor ~]# grep fallocate /var/log/nova/compute.log 2013-03-06 14:47:30 WARNING nova.virt.libvirt.imagebackend [req-a3d2bbec-560f-49cf-9b3d-a7800daedfb8 320ce46de7e24a75a7ff8906d7355ff7 57ff99aae24b4035b52177a722c4091f] fallocate gate vars: size=21474836480, preallocate=True, can=True ====== Additional info: ---------------- Technical notes from discussion with Kevin Wolf (qemu dev) on the benefit of using 'fallocate' for qcow2 images: "It won't improve the qcow2 internal allocations, but at least you get some kind of preallocation on the file system level. "When you write to an unallocated block you get allocations on multiple levels. qcow2 will allocate a cluster in the file format; it still has to do this after fallocate. Then it writes to the image file, so the file system will allocate a block for it; this is the one that you may save. Possibly LVM has to do another allocation, etc."
Turning to VERIFIED per Comment #16 Version info: #-----------------------# [tuser1@interceptor glance(keystone_user1)]$ rpm -q openstack-nova --changelog | grep 891347 - Support preallocated VM images #891347 [tuser1@interceptor glance(keystone_user1)]$ rpm -q openstack-nova openstack-nova-2012.2.3-4.el6ost.noarch [tuser1@interceptor glance(keystone_user1)]$ cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.4 (Santiago) [tuser1@interceptor glance(keystone_user1)]$ #-----------------------#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0657.html