Bug 1198375

We're investigating the feasibility of using the BlockDeviceDriver, and any issues it may present, to do this for Hadoop workloads. This blog describes the rough process:

    http://cloudgeekz.com/71/how-to-setup-openstack-to-use-local-disks-for-instances.html

It would be helpful if you could also take a look at this and vet it for any potential gotchas/downsides from your point of view.

I've been trying this option out the past couple of days..and it seems to be working quite well, though there is a performance drop (might need some tuning?) on a per device basis. Bare-metal can handle 100+MB/s throughput per SATA drive, we're seeing average around 70-80MB/s per device.

I will run more performance testing and update.

Also, encountered another issue that might be related (or not) - during deletion of instances, the instance does get deleted, but stays in "deleting" state for a long time (hours) and eventually get's into an Error with NoState status. This can then be deleted by the admin user.

(In reply to Dwai from comment #4)
> I've been trying this option out the past couple of days..and it seems to be
> working quite well, though there is a performance drop (might need some
> tuning?) on a per device basis. Bare-metal can handle 100+MB/s throughput
> per SATA drive, we're seeing average around 70-80MB/s per device.
> 
> I will run more performance testing and update.
> 
> Also, encountered another issue that might be related (or not) - during
> deletion of instances, the instance does get deleted, but stays in
> "deleting" state for a long time (hours) and eventually get's into an Error
> with NoState status. This can then be deleted by the admin user.

Can you do a virsh dumpxml on the virtual machine once it's operational (you run this command on the hypervisor)? I'm curious as to exactly what options are/aren't enabled so we can confirm what tuning might be possible either today or in future (might require work in nova).

Here's one dumpxml --

<domain type='kvm' id='4'>
  <name>instance-00000003</name>
  <uuid>dbdea1dd-1d30-451d-9b7f-417e5aae9c90</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2014.2.1-1.el7.centos"/>
      <nova:name>scale-2</nova:name>
      <nova:creationTime>2015-06-23 11:47:30</nova:creationTime>
      <nova:flavor name="scale.xl">
        <nova:memory>96000</nova:memory>
        <nova:disk>200</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>20</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="6cb7387eac1d4224ae61a2995a409958">cloud</nova:user>
        <nova:project uuid="1e43af6e89ff400aaaed6ccba8feeaa0">scale</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="668118f8-2207-4655-9030-4857ef8c20c7"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>98304000</memory>
  <currentMemory unit='KiB'>98304000</currentMemory>
  <vcpu placement='static'>20</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Fedora Project</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>2014.2.1-1.el7.centos</entry>
      <entry name='serial'>2632b6fa-6b69-4129-af11-a6283fd70316</entry>
      <entry name='uuid'>dbdea1dd-1d30-451d-9b7f-417e5aae9c90</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='20' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/virtpool/dbdea1dd-1d30-451d-9b7f-417e5aae9c90_disk'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-eec0b301-8f32-449e-a929-9bde41737f2b-lun-0'/>
      <target dev='vdb' bus='virtio'/>
      <serial>eec0b301-8f32-449e-a929-9bde41737f2b</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-303a8ef6-91a2-46f8-a23d-187d8b3906c8-lun-0'/>
      <target dev='vdc' bus='virtio'/>
      <serial>303a8ef6-91a2-46f8-a23d-187d8b3906c8</serial>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-c8e32540-2887-41aa-946a-6199c366dac0-lun-0'/>
      <target dev='vdd' bus='virtio'/>
      <serial>c8e32540-2887-41aa-946a-6199c366dac0</serial>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-5d6d6117-0159-486f-94f6-2ebf3b24f599-lun-0'/>
      <target dev='vde' bus='virtio'/>
      <serial>5d6d6117-0159-486f-94f6-2ebf3b24f599</serial>
      <alias name='virtio-disk4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-525c3101-029b-460d-be0c-a45b2097f076-lun-0'/>
      <target dev='vdf' bus='virtio'/>
      <serial>525c3101-029b-460d-be0c-a45b2097f076</serial>
      <alias name='virtio-disk5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-af817360-23c6-4d2a-99cd-6eb98f0861e0-lun-0'/>
      <target dev='vdg' bus='virtio'/>
      <serial>af817360-23c6-4d2a-99cd-6eb98f0861e0</serial>
      <alias name='virtio-disk6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-37c284b3-fa48-4778-a2ff-2343973a1c56-lun-0'/>
      <target dev='vdh' bus='virtio'/>
      <serial>37c284b3-fa48-4778-a2ff-2343973a1c56</serial>
      <alias name='virtio-disk7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-a4272723-2029-416d-8499-18d3239b40bc-lun-0'/>
      <target dev='vdi' bus='virtio'/>
      <serial>a4272723-2029-416d-8499-18d3239b40bc</serial>
      <alias name='virtio-disk8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-5d9d253a-69f0-49bb-b682-4883e98945fb-lun-0'/>
      <target dev='vdj' bus='virtio'/>
      <serial>5d9d253a-69f0-49bb-b682-4883e98945fb</serial>
      <alias name='virtio-disk9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-63acb05b-0f78-42c3-abaa-c8fe5708fb0c-lun-0'/>
      <target dev='vdk' bus='virtio'/>
      <serial>63acb05b-0f78-42c3-abaa-c8fe5708fb0c</serial>
      <alias name='virtio-disk10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:97:a7:df'/>
      <source bridge='qbr89842f8d-80'/>
      <target dev='tap89842f8d-80'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/dbdea1dd-1d30-451d-9b7f-417e5aae9c90/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/dbdea1dd-1d30-451d-9b7f-417e5aae9c90/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      <stats period='10'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
</domain>

While reviewing nova-specs I came across this proposal which seems like it *might* assist with this if implemented:

    https://review.openstack.org/#/c/241066/

I need to do some further investigation with engineering to determine if this approach would work.

(In reply to Stephen Gordon from comment #8)
> While reviewing nova-specs I came across this proposal which seems like it
> *might* assist with this if implemented:
> 
>     https://review.openstack.org/#/c/241066/
> 
> I need to do some further investigation with engineering to determine if
> this approach would work.

Spec was not approved for Mitaka, will need to be re-proposed for Newton.