Description of problem:

Here's the issue from customer:
Hi everyone, we've found an OpenStack+LVM+SSDs corner case with low I/O performance. I'd like to draw your attention to it; maybe someone knows of a workaround/fix for it.

We're currently running Grizzly with nova.conf libvirt_images_type=default, which has resulted in instances in qcow2 files in ext4 filesystems on SSDs.
We've found in the past that with this configuration, IO to/from the qcow2 (from either guest or host) is much slower than the SSD's theoretical performance.

Consequently we're investigating libvirt_images_type=lvm, with libvirt_images_volume_group set to a LV backed by the same SSDs.
We've found that this yields much better iops than the original libvirt_images_type=default. It seems eliminating the qcow layer significantly increases performance.

However, we've found that with libvirt_images_type=lvm, we have a different problem: deleting instances is slow.
It looks like this is a known problem:
https://blueprints.launchpad.net/nova/+spec/lvm-clear-option

Sure enough, we do see a lot of 'dd' processes during a parallel 'nova delete' operation:
# ps aux | grep /bin/dd
root      62982  1.8  0.0 106212  1680 ?        D    19:01 0:01 /bin/dd bs=1048576 if=/dev/zero of=/dev/novainstances/instance-000000de_disk seek=0 count=20480 oflag=direct
root      62991  1.9  0.0 106212  1680 ?        D    19:01 0:01 /bin/dd bs=1048576 if=/dev/zero of=/dev/novainstances/instance-000000e1_disk seek=0 count=20480 oflag=direct
root      62998  1.8  0.0 106212  1680 ?        D    19:01 0:01 /bin/dd bs=1048576 if=/dev/zero of=/dev/novainstances/instance-000000d2_disk seek=0 count=20480 oflag=direct
root      63013  1.9  0.0 106212  1684 ?        D    19:01 0:01 /bin/dd bs=1048576 if=/dev/zero of=/dev/novainstances/instance-000000cf_disk seek=0 count=20480 oflag=direct
root      63289  1.7  0.0 106212  1680 ?        D    19:02 0:01 /bin/dd bs=1048576 if=/dev/zero of=/dev/novainstances/instance-000000d5_disk seek=0 count=20480 oflag=direct
root      63292  1.7  0.0 106212  1680 ?        D    19:02 0:01 /bin/dd bs=1048576 if=/dev/zero of=/dev/novainstances/instance-000000d8_disk seek=0 count=20480 oflag=direct
root      63295  1.6  0.0 106212  1684 ?        D    19:02 0:01 /bin/dd bs=1048576 if=/dev/zero of=/dev/novainstances/instance-000000db_disk seek=0 count=20480 oflag=direct
root      63322  0.0  0.0 103248   860 pts/1    S+   19:03 0:00 grep /bin/dd
#

And it looks like this is due to Grizzly's /usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py which does an unconditional 'dd if=/dev/zero' of LVs upon deletion:

def clear_logical_volume(path):
     """Obfuscate the logical volume.

     :param path: logical volume path
     """
     # TODO(p-draigbrady): We currently overwrite with zeros
     # but we may want to make this configurable in future
     # for more or less security conscious setups.

I perceive multiple less-than-ideal aspects of this, most of them outside OpenStack itself, but impacting on it. Here's my list:

a. Implementing the TODO, ie adding a boolean option to nova.conf to disable LV erasure on delete, would solve the immediate performance problem. However it would introduce potential inter-tenant information disclosure when enabled;
b. With knowledge of the partitions/filesystems in use on the LV, a more efficient but still reasonably secure method could be used. For example overwrite all copies of the superblock and the journal with random garbage;
c. Best of all would be if the LV layer itself knew how to securely destroy its LVs (think "lvdestroy --secure"), not a userland 'dd' process running as root that somehow 'knows' the block device corresponding to the LV that it wants to securely erase;
d. Separately, when destroying LVs backed by SSDs, instead of zeroing blocks, the vendor-specific 'secure delete' facility should be used (most SSD firmwares offer one), or mass discard/trim/unmap commands.
'dd' shouldn't know directly about the block layer, so this seems to me to belong in the LVM layer too.

Questions:
- Can you advise us a way to avoid both the original qcow2 overhead and the new dd overhead?
- Which (if any) of the above ideas are worth advancing? How should we best do that?
- What've I totally missed here?


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Using LVM for the instance: libvirt_images_type=lvm
2. Deleting instance and check with ps | grep dd
3.

Actual results:
Because of dd operations for any deletions, it causes of slowness.


Expected results:
Faster deletion than 'dd'-ing

Additional info: