Bug 1470449

Description of problem:

Our application require a number of Cinder volumes to be attached to the Nova instance. They always need to be attached in the same order, the order matter to the software defined storage application. How they are presented in our application determines what "disk" the Cinder volume becomes in the application (our software defined storage VM has boot, root, coredump, data disks, etc.)

We use the OpenStack API to create the resources (Cinder, Neutron, Nova, etc.) and attach them with a Nova server create call.

Most of the time the volumes are attached in the correct order, but about 1 out of 10 times, the order of the volumes as they are presented in the nova API call (Python list) is not preserved. This causes our SDS VM to fail booting because it does not get the disks it expects in the correct order.

Most VMs do not care about the order in which the cinder volumes are presented in the VM, in our case it is significant.



Version-Release number of selected component (if applicable):

This is OpenStack RDO Liberty

openstack-nova-compute-12.0.4-1.el7.noarch


How reproducible:


Steps to Reproduce:

This has been done using the OpenStack API, which is the best way to programmatically reproduce the problem, but could likely be done with OS CLI as well.

1. Create a number of Cinder volumes in a way which they can be uniquely identified in the VM instance (different sizes, etc.
2. Attach volumes to Nova instance and boot.
3. repeat steps 1 & 2 enough times, and the cinder volumes will be attached to the nova instance in a different order than what was specified. This can be verified by checking the libvirt XML file that is generated by nova (virsh dumpxml <domain name>).


Actual results:

Most times the expected result is true, about 1 out of 10 times, the order the volumes are attached to the nova instance is not what is expected.


Expected results:

Given an ordered list of Cinder volumes to be attached to a nova instance, the expected result is that they are attached in the specified order every time.


Additional info:

There is a Canonical launchpad bug filed for this as well:

https://bugs.launchpad.net/nova/+bug/1697580

Additional info from this bug is:

All volumes are attached using block device mapping.

I am not passing in any device name, just letting it do auto-assignment. In the software defined storage application, it's not expecting specific device names, rather it relies on the order in which they show up on first boot, which is one of the reasons that we can't boot and then attach the volumes after the fact.

>> Are you waiting for each volume to show up as "in-use" before attaching another one?

The volumes are not explicitly attached to the instance, they are created, we wait until they are in the active state, and they are put in order into a Python list.

The call to boot the server looks like this:

 server = self.nova.servers.create(name=self.name,
                                          image=self.boot_image_base,
                                          flavor=self.flavor,
                                          config_drive=self.config_drive,
       userdata=self.userdata_fp,
                                          block_device_mapping_v2=self.block_device_list,
                                   nics=self.nic_device_list)

self.block_device_list is the list of volumes that we want mapped to the instance, in a very specific order. As stated in the original message, something on the order of 9 times out of 10 that order is preserved.

BTW, we do have a HEAT template - we don't use it for our internal deployments, but it is something that we could give out to customers. I'm doing a bunch of deployments with the HEAT templates to see if I can get it to fail in the same way.

Let me know what other information should be provided.