Just wanted to add a comment on our suggested approach (from Chet)...


h2. Problem Statement
If you use the nova Boot From Volume (BFV) workflow you can not specify a cinder volume-type on the request. As a result the volume is *always* created in the default volume-type as defined by the default_volume_type option in cinder.conf (if default_volume_type is not set then one of the backends is randomly selected by the cinder scheduler service). In a tiered storage deployment this means users have to first pre-create the volume, wait for the volume to be created, and then boot onto it if they want anything other then the default.

Ideally the solution to this problem is to extend the nova API to accept volume-type as an optional field on the BFV call. After 2 cycles of discussion we have been unable to get agreement from the nova core team to pursue this route. nova has taken that stance that the proper way to address this is to simply create a volume using cinder and boot from that volume. The main problem with this is that you have to create the volume *and* wait for it to become "available" before you issue the boot command. If the status of a volume is not "available" the nova API will reject the request as invalid. Today this is done in the check_attach function of the compute API, however nova core plans on merging https://review.openstack.org/#/c/335358/ which will remove the check_attach code and will instead rely on cinder's reserve volume call (https://github.com/openstack/cinder/blob/master/cinder/volume/api.py#L614). The reserve volume call uses similar logic to the nova API in that it won't allow you "reserve" a volume if its isn't in the "available" or "in-use" (for multiattach only) state. Additionally the reserve volume call changes the status of the volume to "attaching" which isn't accurate at all because its not actually "attaching" rather someone has expressed that they intend to attach to the volume at some future time.

h2. Proposed Solution
We propose to change the reserve volume call such that it will allow for the reserving of a volume that is in the "creating" or "downloading" state. Since we now want to "reserve" a volume that is still in a transitory state ("creating" or "downloading") we wish to change the way the reserve volume call works such that it does not need to change the status to "attaching". We propose that we extend the existing attach_status field to indicate a reserved volume. The attach_status field today have two values "attached" and "detached". We propose the addition of a 3rd value called "pending". When a reserve call is made the status and attach_status of the volume will be checked and assuming the attach could proceed the attach_status field will be updated to "pending".

The code for the reserve volume call would look something like the following.

NOTE: This code has *not* been tested and may not be complete.

{code}
diff --git a/cinder/volume/api.py b/cinder/volume/api.py
index d964eeb..5202cfc 100644
--- a/cinder/volume/api.py
+++ b/cinder/volume/api.py
@@ -613,10 +613,14 @@ class API(base.Base):
     @wrap_check_policy
     def reserve_volume(self, context, volume):
         expected = {'multiattach': volume.multiattach,
-                    'status': (('available', 'in-use') if volume.multiattach
-                               else 'available')}
-
-        result = volume.conditional_update({'status': 'attaching'}, expected)
+                    'attach_status': ('detatched',),
+                    'status': ('available', 'creating', 'downloading')}
+        if volume.multiattach:
+            expected['attach_status'] += ('attached',)
+            expected['status'] += ('in-use',)
+
+        result = volume.conditional_update({'attach_status': 'pending'},
+                                           expected)

         if not result:
             expected_status = utils.build_or_str(expected['status'])
{code}

In addition to the changes to the reserve volume call the unreserve volume call would also need to be updated to account for this change. Additionally there are handful of locations in the code base today where the attach_status of the volume is checked. Those locations would have to be updated to account for the new status of "pending".

Once the cinder side of things is completed, nova needs to be updated to utilization this new functionality. Depending on how the functionality is implemented in cinder (micro versions etc) work may have to be done to allow nova to use the new reserve volume functionality. Additionally the logic on the nova-compute node that waits for a volume to become available before booting the VM will also need to be updated to wait when a request is made to boot from an existing volume (see the changes to nova/virt/block_device.py in https://review.openstack.org/#/c/391444/ for info).

h2. Next Steps
We need to write-up a cinder spec with the above, propose it, and see if we can get some agreement from the cinder cores on the spec.

Cinder change https://review.openstack.org/#/c/387712/ will provided the mechanics to reserve a volume while it is in creating or downloading state. This can be leveraged by the nova boot from volume procedure so that BFV will not have to wait for the volume to be completely available. Hence, the user can program the API call flow without worrying about waiting volume creation to complete. The user operation flow will be:
1. Create volume from image which is yet to complete (job running on the background)
2. Nova bfv reserves the volume returns with dict that instance is waiting for volume to be available.  The BFV call is not blocked but running on the background
3. Volume creation job on background completes after downloading etc
4. Instance creation job on background complete after volume creation completes

This will require changes in Nova to:
1. Call cinder API to reserver volume while it is not available yet
2. Unblock bfv call while waiting for volume to be available.
3. Update instance status per reserved volume and BFV background job status

This blueprint registered, please review/revise: https://blueprints.launchpad.net/nova/+spec/bfv-with-reserved-volume

Hey Eoghan
What is the next step we need to do to help our teams move this along in Ocata?
Thanks
Charles

(In reply to Charles Crouch from comment #4)
> Hey Eoghan
> What is the next step we need to do to help our teams move this along in
> Ocata?
> Thanks
> Charles

We need to actually write a specification, and unfortunately that hasn't been possible for Ocata (there is also quite a long backlog of things that did have specifications but did not land due to the shortened cycle).

To progress this in Pike we will need to identify someone to own writing the specification. Normally this would be the blueprint owner but it's not clear Yuming Ma was taking that here? A review was requested but I don't see a specification with the actual detail linked from:

https://blueprints.launchpad.net/nova/+spec/bfv-with-reserved-volume

There was a specification on the older blueprint this superseded but it of course was using the proxy APIs approach which upstream have rejected.

Stephen, I can take up on writing the spec and get the ball rolling. Any comments on the blueprint so far?

spec review at : https://review.openstack.org/#/c/417541

Hi, Stephen, I have committed the review (see above URL), please comment/review. Thanks

(In reply to Yuming Ma from comment #9)
> Hi, Stephen, I have committed the review (see above URL), please
> comment/review. Thanks

Thanks, we're working on preparation for the Pike and the associated PTG at the moment so I will try and get an assignee from our side. Leaving the NEEDINFO open for now.

Thanks Steve, I am planning to go to the PTG as well and please let me know who on RH side are going and let's sync up before and during the meetings. My email: yumima

(In reply to Yuming Ma from comment #11)
> Thanks Steve, I am planning to go to the PTG as well and please let me know
> who on RH side are going and let's sync up before and during the meetings.
> My email: yumima

Hopefully will be confirming who will be following this effort from our side early next week.

Based on the discussions at the PTG this work is merging into the spec for using cinder's new attachment API:

https://review.openstack.org/#/c/373203/

Closing out as WONTFIX for openstack-nova based on [1] and various other PTG/Forum discussions during previous cycles. Users will either need to create their volumes first _or_ extend the openstackclient to orchestrate the creation of volumes using a specific type before calling the Nova API.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117242.html