When converting non raw to raw, libvirt leaves the original non raw image on disk
For details on how to test this see bug 837627
Test info: 1/ Version info: ------------ [tuser1@interceptor glance(keystone_user1)]$ rpm -q openstack-nova --changelog | grep 911103 - Remove intermediate libvirt downloaded images #911103 [tuser1@interceptor glance(keystone_user1)]$ rpm -q openstack-nova openstack-nova-2012.2.3-4.el6ost.noarch ------------ 2/ Ensure you the below configs settings in nova.conf ------------ [tuser1@interceptor glance(keystone_user1)]$ sudo egrep '^image_cache_manager_interval|^periodic_interval|^remove_unused_original_minimum_age_seconds' /etc/nova/nova.conf periodic_interval=1 image_cache_manager_interval=1 remove_unused_original_minimum_age_seconds=60 ------------ 3/ Restart all nova services ------------ [tuser1@interceptor glance(keystone_user1)]$ sudo -i [root@interceptor ~(keystone_user1)]# for j in `for i in $(ls -1 /etc/init.d/openstack-nova-*) ; do $i status | grep running ; done | awk '{print $1}'` ; do service $j restart ; done ------------ 4a/ List the instances in base directory (before stopping the running instances). (side note: I changed the default instances_path) ------------ [tuser1@interceptor ~(keystone_user1)]$ ls -l /export/nova/instances/_base/ total 10782100 -rw-r--r--. 1 nova nova 10486808576 Mar 7 17:23 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 -rw-r--r--. 1 nova nova 21474836480 Mar 7 17:23 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 [tuser1@interceptor ~(keystone_user1)]$ ------------ 4b/ Stop and Delete all instances (so that the referenced backing files will be rendered 'unused' ------------ [tuser1@interceptor ~(keystone_user1)]$ nova stop fed-t1 && nova delete fed-t2 && nova stop fed-t3 && nova stop fed-t3 ------------ 5/ Grep for 'removing base file' string: ------------ [root@interceptor ~]# grep -i 'removing base file' /var/log/nova/compute.log 2013-03-07 17:24:49 30765 INFO nova.virt.libvirt.imagecache [-] Removing base file: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 [root@interceptor ~]# ------------ 6/ Check the base instances directory again, so it's removed ------------ [tuser1@interceptor ~(keystone_user1)]$ ls -l /export/nova/instances/_base/ total 10026272 -rw-r--r--. 1 nova nova 21474836480 Mar 7 17:23 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 [tuser1@interceptor ~(keystone_user1)]$ ------------ But, still an image -- the resized (to 20G flavor) of the downloaded image remains. Let's see what's going in the logs: [root@interceptor ~]# tail -f /var/log/nova/compute.log | egrep -i -B3 'removable' ------------ 2013-03-07 17:43:49 30765 DEBUG nova.manager [-] Running periodic task ComputeManager._run_image_cache_manager_pass periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:181 2013-03-07 17:43:49 30765 DEBUG nova.virt.libvirt.imagecache [-] Verify base images verify_base_images /usr/lib/python2.6/site-packages/nova/virt/libvirt/imagecache.py:383 2013-03-07 17:43:49 30765 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2013-03-07 17:43:49 30765 INFO nova.virt.libvirt.imagecache [-] Removable base files: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 . . . ------------ It says - base file too young to remove. However, we've set 'remove_unused_original_minimum_age_seconds=60' . So, it should've been removed ideally
To note further, we've also tried enabling "remove_unused_resized_minimum_age_seconds=60" & restart all nova services 7a/ Set the config directive 'remove_unused_resized_minimum_age_seconds, ------------ $ sudo egrep '^image_cache_manager_interval|^periodic_interval|^remove_unused_original_minimum_age_seconds|^remove_unused_resized_minimum_age_second' /etc/nova/nova.confperiodic_interval=1 image_cache_manager_interval=1 remove_unused_resized_minimum_age_seconds=60 remove_unused_original_minimum_age_seconds=60 ------------ 7b/ Restart all nova services ; boot a different instance; stop, delete it, check the instances_path, /export/nova/i 8/ Check in nova compute.log ------------ [tuser1@interceptor ~(keystone_user1)]$ sudo tail -f /var/log/nova/compute.log | grep -i removable -B3 -A22013-03-07 18:21:15 30765 DEBUG nova.manager [-] Running periodic task ComputeManager._run_image_cache_manager_pass periodic_tasks /usr/lib/python2.6/site-packages/nova/manager.py:181 2013-03-07 18:21:15 30765 DEBUG nova.virt.libvirt.imagecache [-] Verify base images verify_base_images /usr/lib/python2.6/site-packages/nova/virt/libvirt/imagecache.py:383 2013-03-07 18:21:15 30765 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2013-03-07 18:21:15 30765 INFO nova.virt.libvirt.imagecache [-] Removable base files: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2013-03-07 18:21:15 30765 INFO nova.virt.libvirt.imagecache [-] Base file too young to remove: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2013-03-07 18:21:15 30765 DEBUG nova.virt.libvirt.imagecache [-] Verification complete verify_base_images /usr/lib/python2.6/site-packages/nova/virt/libvirt/imagecache.py:428 . . ------------ NOTE: The fedora image used above is a qcow2 image (non raw) =============== [tuser1@interceptor ~(keystone_user1)]$ glance image-list +--------------------------------------+-----------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+-----------+-------------+------------------+------------+--------+ | 1e6292f9-82bd-4cdb-969e-c863cb1c6692 | fedora-17 | qcow2 | bare | 251985920 | active | =============== Conclusion: It's possible that, nova forgets the resized image file (_20) origin. Will file a different bug for that. It doesn't affect the current bug verification -- No stale .part images are left behind when a qcow2 image was downloaded. Turning this bugzilla to VERIFIED per this comment and comment #3
Final conclusion about the remaining _20 images (the resized raw images) noted in commment #5 : I had to enable 'remove_unused_resized_minimum_age_seconds' config directive Config settings: ------------ [tuser1@interceptor ~(keystone_user1)]$ sudo egrep '^image_cache_manager_interval|^periodic_interval|^remove_unused_original_minimum_age_seconds|^remove_unused_resized_minimum_age_second|^remove_unused_base_images' /etc/nova/nova.conf periodic_interval=1 image_cache_manager_interval=1 remove_unused_base_images=true remove_unused_resized_minimum_age_seconds=60 remove_unused_original_minimum_age_seconds=60 ------------ With the above configs; boot a flavor 2 nova instance ; stop & delete it; monitor nova compute logs to find both the base images are being deleted from the instances/_base directory: ------------ . . 2013-03-07 19:56:22 31591 INFO nova.virt.libvirt.imagecache [-] Removable base files: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2013-03-07 19:56:24 31591 INFO nova.virt.libvirt.imagecache [-] Removable base files: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 2013-03-07 19:56:24 31591 INFO nova.virt.libvirt.imagecache [-] Removing base file: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 2013-03-07 19:56:26 31591 INFO nova.virt.libvirt.imagecache [-] Removing base file: /export/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20 ------------ For later reference, adding my entire nova.conf here: ------------ [tuser1@interceptor ~(keystone_user1)]$ sudo cat /etc/nova/nova.conf| grep -v ^$ | grep -v ^# [DEFAULT] fixed_range=10.65.207.48/29 floating_range=10.65.207.56/29 force_dhcp_release=false flat_network_bridge=demonetbr0 flat_injected=false novncproxy_host=0.0.0.0 dhcpbridge=/usr/bin/nova-dhcpbridge dhcpbridge_flagfile=/etc/nova/nova.conf dhcp_domain=novalocal novncproxy_port=6080 metadata_listen=0.0.0.0 allowed_direct_url_schemes=[file] debug=true verbose=true sql_connection=mysql://nova:nova@localhost/nova network_manager=nova.network.manager.FlatDHCPManager periodic_interval=1 instances_path=/export/nova/instances image_cache_manager_interval=1 rpc_backend=nova.openstack.common.rpc.impl_qpid auth_strategy = keystone flat_interface = eth2 public_interface = eth2 volume_api_class = nova.volume.cinder.API enabled_apis = ec2,osapi_compute,metadata preallocate_images=space remove_unused_resized_minimum_age_seconds=60 remove_unused_original_minimum_age_seconds=60 [trusted_computing] [keystone_authtoken] [tuser1@interceptor ~(keystone_user1)]$ ------------
Wow, far too many config vars in this area
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0657.html