Description of problem: This is an RFE, to be able to do direct disk passthrough for Openstack instances to leverage. This is the most performant configuration for Hadoop clusters, with access to local disks. Version-Release number of selected component (if applicable): OSP 6 How reproducible: N/A Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
We're investigating the feasibility of using the BlockDeviceDriver, and any issues it may present, to do this for Hadoop workloads. This blog describes the rough process: http://cloudgeekz.com/71/how-to-setup-openstack-to-use-local-disks-for-instances.html It would be helpful if you could also take a look at this and vet it for any potential gotchas/downsides from your point of view.
I've been trying this option out the past couple of days..and it seems to be working quite well, though there is a performance drop (might need some tuning?) on a per device basis. Bare-metal can handle 100+MB/s throughput per SATA drive, we're seeing average around 70-80MB/s per device. I will run more performance testing and update. Also, encountered another issue that might be related (or not) - during deletion of instances, the instance does get deleted, but stays in "deleting" state for a long time (hours) and eventually get's into an Error with NoState status. This can then be deleted by the admin user.
(In reply to Dwai from comment #4) > I've been trying this option out the past couple of days..and it seems to be > working quite well, though there is a performance drop (might need some > tuning?) on a per device basis. Bare-metal can handle 100+MB/s throughput > per SATA drive, we're seeing average around 70-80MB/s per device. > > I will run more performance testing and update. > > Also, encountered another issue that might be related (or not) - during > deletion of instances, the instance does get deleted, but stays in > "deleting" state for a long time (hours) and eventually get's into an Error > with NoState status. This can then be deleted by the admin user. Can you do a virsh dumpxml on the virtual machine once it's operational (you run this command on the hypervisor)? I'm curious as to exactly what options are/aren't enabled so we can confirm what tuning might be possible either today or in future (might require work in nova).
Here's one dumpxml -- <domain type='kvm' id='4'> <name>instance-00000003</name> <uuid>dbdea1dd-1d30-451d-9b7f-417e5aae9c90</uuid> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0"> <nova:package version="2014.2.1-1.el7.centos"/> <nova:name>scale-2</nova:name> <nova:creationTime>2015-06-23 11:47:30</nova:creationTime> <nova:flavor name="scale.xl"> <nova:memory>96000</nova:memory> <nova:disk>200</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>20</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="6cb7387eac1d4224ae61a2995a409958">cloud</nova:user> <nova:project uuid="1e43af6e89ff400aaaed6ccba8feeaa0">scale</nova:project> </nova:owner> <nova:root type="image" uuid="668118f8-2207-4655-9030-4857ef8c20c7"/> </nova:instance> </metadata> <memory unit='KiB'>98304000</memory> <currentMemory unit='KiB'>98304000</currentMemory> <vcpu placement='static'>20</vcpu> <resource> <partition>/machine</partition> </resource> <sysinfo type='smbios'> <system> <entry name='manufacturer'>Fedora Project</entry> <entry name='product'>OpenStack Nova</entry> <entry name='version'>2014.2.1-1.el7.centos</entry> <entry name='serial'>2632b6fa-6b69-4129-af11-a6283fd70316</entry> <entry name='uuid'>dbdea1dd-1d30-451d-9b7f-417e5aae9c90</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='20' cores='1' threads='1'/> </cpu> <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='catchup'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/virtpool/dbdea1dd-1d30-451d-9b7f-417e5aae9c90_disk'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-eec0b301-8f32-449e-a929-9bde41737f2b-lun-0'/> <target dev='vdb' bus='virtio'/> <serial>eec0b301-8f32-449e-a929-9bde41737f2b</serial> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-303a8ef6-91a2-46f8-a23d-187d8b3906c8-lun-0'/> <target dev='vdc' bus='virtio'/> <serial>303a8ef6-91a2-46f8-a23d-187d8b3906c8</serial> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-c8e32540-2887-41aa-946a-6199c366dac0-lun-0'/> <target dev='vdd' bus='virtio'/> <serial>c8e32540-2887-41aa-946a-6199c366dac0</serial> <alias name='virtio-disk3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-5d6d6117-0159-486f-94f6-2ebf3b24f599-lun-0'/> <target dev='vde' bus='virtio'/> <serial>5d6d6117-0159-486f-94f6-2ebf3b24f599</serial> <alias name='virtio-disk4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-525c3101-029b-460d-be0c-a45b2097f076-lun-0'/> <target dev='vdf' bus='virtio'/> <serial>525c3101-029b-460d-be0c-a45b2097f076</serial> <alias name='virtio-disk5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-af817360-23c6-4d2a-99cd-6eb98f0861e0-lun-0'/> <target dev='vdg' bus='virtio'/> <serial>af817360-23c6-4d2a-99cd-6eb98f0861e0</serial> <alias name='virtio-disk6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-37c284b3-fa48-4778-a2ff-2343973a1c56-lun-0'/> <target dev='vdh' bus='virtio'/> <serial>37c284b3-fa48-4778-a2ff-2343973a1c56</serial> <alias name='virtio-disk7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-a4272723-2029-416d-8499-18d3239b40bc-lun-0'/> <target dev='vdi' bus='virtio'/> <serial>a4272723-2029-416d-8499-18d3239b40bc</serial> <alias name='virtio-disk8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-5d9d253a-69f0-49bb-b682-4883e98945fb-lun-0'/> <target dev='vdj' bus='virtio'/> <serial>5d9d253a-69f0-49bb-b682-4883e98945fb</serial> <alias name='virtio-disk9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-path/ip-10.17.194.23:3260-iscsi-iqn.2010-10.org.openstack:volume-63acb05b-0f78-42c3-abaa-c8fe5708fb0c-lun-0'/> <target dev='vdk' bus='virtio'/> <serial>63acb05b-0f78-42c3-abaa-c8fe5708fb0c</serial> <alias name='virtio-disk10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <interface type='bridge'> <mac address='fa:16:3e:97:a7:df'/> <source bridge='qbr89842f8d-80'/> <target dev='tap89842f8d-80'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='file'> <source path='/var/lib/nova/instances/dbdea1dd-1d30-451d-9b7f-417e5aae9c90/console.log'/> <target port='0'/> <alias name='serial0'/> </serial> <serial type='pty'> <source path='/dev/pts/1'/> <target port='1'/> <alias name='serial1'/> </serial> <console type='file'> <source path='/var/lib/nova/instances/dbdea1dd-1d30-451d-9b7f-417e5aae9c90/console.log'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='tablet' bus='usb'> <alias name='input0'/> </input> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> <stats period='10'/> </memballoon> </devices> <seclabel type='none'/> </domain>
While reviewing nova-specs I came across this proposal which seems like it *might* assist with this if implemented: https://review.openstack.org/#/c/241066/ I need to do some further investigation with engineering to determine if this approach would work.
(In reply to Stephen Gordon from comment #8) > While reviewing nova-specs I came across this proposal which seems like it > *might* assist with this if implemented: > > https://review.openstack.org/#/c/241066/ > > I need to do some further investigation with engineering to determine if > this approach would work. Spec was not approved for Mitaka, will need to be re-proposed for Newton.