Description of problem: Volume wipe with trim algorithm does not reclaim space on rbd cow clone Version-Release number of selected component (if applicable): libvirt-1.3.2-1.fc24_v1.3.1_79_g63e15ad.x86_64 librbd1-9.2.0-2.fc24.x86_64 How reproducible: 100% Steps to Reproduce: 1. create rbd volume in rbd pool # virsh vol-create-as rbd vol1 1G Vol vol1 created 2. add the rbd volume into guest and write a file into it [root@fedora_yy ~]# virsh edit yy Domain yy XML configuration edited. [root@fedora_yy ~]# virsh start yy Domain yy started # virsh domblklist yy Target Source ------------------------------------------------ hda /var/lib/libvirt/images/generic.qcow2 vda yy/vol1 Write 100M data into rbd volume [root@fedora_yy ~]# virsh destroy yy Domain yy destroyed 3. create cow clone [root@fedora_yy ~]# rbd snap create yy/vol1@sn1 [root@fedora_yy ~]# rbd snap protect yy/vol1@sn1 [root@fedora_yy ~]# rbd clone yy/vol1@sn1 yy/vol1.clone [root@fedora_yy ~]# rbd info yy/vol1.clone rbd image 'vol1.clone': size 1024 MB in 256 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.b0256136682b format: 2 features: layering flags: parent: yy/vol1@sn1 overlap: 1024 MB [root@fedora_yy ~]# rados -p yy ls | grep rbd_data.b0256136682b | wc -l 0 # virsh pool-refresh rbd 4. add the cow clone into guest and write 200M data into it [root@fedora_yy ~]# virsh edit yy Domain yy XML configuration edited. [root@fedora_yy ~]# virsh start yy Domain yy started # virsh domblklist yy Target Source ------------------------------------------------ hda /var/lib/libvirt/images/generic.qcow2 vda yy/vol1.clone write 200M data into vol1.clone [root@fedora_yy ~]# virsh destroy yy Domain yy destroyed 5. check object counts [root@fedora_yy ~]# rados -p yy ls | grep rbd_data.b0256136682b | wc -l 54 6. wipe the cow clone with trim algorithm # virsh vol-wipe vol1.clone rbd --algorithm trim Vol vol1.clone wiped 7. check object counts [root@fedora_yy ~]# rados -p yy ls | grep rbd_data.b0256136682b |wc -l 54 Actual results: In step 7, allocated space is not reclaimed. object counts do not change Expected results: Allocated space is reclaimed in cow clone Additional info:
Created attachment 1120368 [details] libvirtd.log
The RADOS objects will not be removed, but they will be trimmed to zero bytes inside Ceph. Can you run: $ ceph df Now trim the volume and wait 30 seconds $ ceph df The pool should now use less space.
Wido I ran the steps as you said, the pool indeed use less space after wipe. What confused me is that, the RADOS objects will be removed on general rbd image, however, they will not be removed on cow clone rbd image. Why do they behave differently 1. tested on cow clone rbd image, the pool use less space as expected, but rados objects are not removed # rbd info yy/vol1.clone rbd image 'vol1.clone': size 1024 MB in 256 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.fac412bfc3f4 format: 2 features: layering flags: parent: yy/vol1@sn1 overlap: 1024 MB [root@fedora_yy ~]# rados -p yy ls | grep rbd_data.fac412bfc3f4 | wc -l 128 # ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 142G 119G 23206M 15.92 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 40647M 0 yy 1 924M 0.63 40647M 240 # virsh vol-wipe vol1.clone rbd --algorithm trim Vol vol1.clone wiped [root@fedora_yy ~]# rados -p yy ls | grep rbd_data.fac412bfc3f4 | wc -l 128 [root@fedora_yy ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 142G 121G 21525M 14.77 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 41223M 0 yy 1 414M 0.28 41223M 240 2. tested on general rbd image, the pool used less space as expected, and rados objects are removed [root@fedora_yy ~]# virsh vol-create-as rbd vol2 1G Vol vol2 created [root@fedora_yy ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 142G 122G 20779M 14.26 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 41511M 0 yy 1 414M 0.28 41511M 241 Write 400M data into the image # ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 142G 119G 23057M 15.82 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 40748M 0 yy 1 828M 0.57 40748M 348 [root@fedora_yy ~]# rbd info yy/vol2 rbd image 'vol2': size 1024 MB in 256 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.b06a35aaa966 format: 2 features: layering, striping flags: stripe unit: 4096 kB stripe count: 1 [root@fedora_yy ~]# rados -p yy ls | grep rbd_data.b06a35aaa966 | wc -l 106 [root@fedora_yy ~]# virsh vol-wipe vol2 rbd --algorithm trim Vol vol2 wiped # rados -p yy ls | grep rbd_data.b06a35aaa966 | wc -l 0 [root@fedora_yy ~]# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 142G 121G 20839M 14.30 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 41449M 0 yy 1 414M 0.28 41449M 242
That is a internal thing inside Ceph. The object do not occupy any space anymore and thus the result is as we want it to be. I think we can close this as it is not a bug. Trimming results in all data being trimmed from the RBD volume.