Bug 1019401 - nova: failing to boot instance from image (create new volume) because of time out
nova: failing to boot instance from image (create new volume) because of time...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
4.0
x86_64 Linux
medium Severity high
: ---
: 6.0 (Juno)
Assigned To: Lee Yarwood
yeylon@redhat.com
storage
: Triaged, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-15 11:58 EDT by Dafna Ron
Modified: 2016-04-26 12:54 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-02 09:42:51 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (10.85 MB, application/x-gzip)
2013-10-15 11:58 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1213953 None None None Never

  None (edit)
Description Dafna Ron 2013-10-15 11:58:22 EDT
Created attachment 812619 [details]
logs

Description of problem:

I fail to boot an instance from an image (create new volume) option on a time out. 

there is a downstream bug opened for making the time out configurable and not hard coded since some images will take longer than other to download. 

opening an upstream bug as well since I am failing 99.99% of the time to boot the instance 

upstream bug: 

https://bugs.launchpad.net/nova/+bug/1213953

Version-Release number of selected component (if applicable):

openstack-nova-scheduler-2013.2-0.24.rc1.el6ost.noarch
openstack-nova-compute-2013.2-0.24.rc1.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. install openstack using packstack on rhel6.5 and configure cinder and glance to work with gluster
2. create an image and try to boot an instance using the "boot from image (create new volume) option
3. we fail on timeout which is hard coded instead of configurable 

Actual results:

I wanted to change the time out to a different value but the time out is hard coded in the code and not configurable. 

Expected results:


Additional info:

eharney sent me the place in the code: 

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L897

as you can see, time out is hard coded and we should be able to change this value in case we have large images 


/var/log/nova/scheduler.log:2013-10-15 18:00:40.417 3123 ERROR nova.scheduler.filter_scheduler [req-199f8c34-f57e-4e6b-a226-60eca124257b a663c5728f3a4221b7d5ae9e9a50beab 162fc167f17d464f9832412245fb7283] [instance: a5d75758-3350-4754-8cef-043d4306dfcd] Error from last host: nott-vdsa.qa.lab.tlv.redhat.com (node nott-vdsa.qa.lab.tlv.redhat.com): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1028, in _build_instance\n    context, instance, bdms)\n', u'  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1393, in _prep_block_device\n    instance=instance)\n', u'  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1376, in _prep_block_device\n    self._await_block_device_map_created))\n', u'  File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 283, in attach_block_devices\n    block_device_mapping)\n', u'  File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 238, in attach\n    wait_func(context, vol[\'id\'])\n', u'  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 901, in _await_block_device_map_created\n    attempts=attempts)\n', u'VolumeNotCreated: Volume d3e1c197-5e54-4c15-a500-1f847d7372cf did not finish being created even after we waited 69 seconds or 60 attempts.\n']
Comment 1 Solly Ross 2013-10-16 16:57:59 EDT
I'm adding two config options: poll_blk_dev_tries (default is 60) and poll_blk_dev_interval (default is 1).  Currently running tests, but I thought I'd give an update.  Sound good?
Comment 2 Dafna Ron 2013-10-17 05:18:00 EDT
yes :) you're awesome!
Comment 3 Solly Ross 2013-10-17 13:55:20 EDT
It looks like upstream wants a much more complicated fix to this issue.  We may have to do a downstream patch on this one for the moment and wait for rhos-5.0 for the more complicated fix (changing the Cinder API to actually provide status information).
Comment 4 Nikola Dipanov 2013-10-29 08:26:46 EDT
Adding the link to the upstream discussion when the patch for the config options was first proposed https://review.openstack.org/#/c/42876/6/ for completeness. Special attention should be paid to inline comments on the linked review.
Comment 5 Solly Ross 2013-11-04 13:39:44 EST
I've conferred with some others on the downstream list, and it looks like it's been decided to just do the upstream fix and backport that.  I'm beginning working on a patch to the upstream Cinder and Nova APIs.
Comment 7 shilpa 2014-02-03 02:19:40 EST
This bug also affects booting instance from volume snapshot, because a bootable volume is created here and nova times out on it.

openstack-cinder-2013.2.1-5.el6ost.noarch
openstack-nova-2013.2.1-2.el6ost.noarch


2014-02-03 12:37:26.602 25255 ERROR nova.compute.manager [req-a1a3c7fe-6166-476f-bc3c-86fa816187b4 a74de68faf7a47ac8559c8460121da6e 0db2318a153e4a9486796627e35c5e1e] [i
nstance: 5691dc36-983e-403b-a46f-90ad81e36fa5] Instance failed block device setup
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5] Traceback (most recent call last):
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.
py", line 1381, in _prep_block_device
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5]     self._await_block_device_map_created) +
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5]   File "/usr/lib/python2.6/site-packages/nova/virt/block_devic
e.py", line 283, in attach_block_devices
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5]     block_device_mapping)
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5]   File "/usr/lib/python2.6/site-packages/nova/virt/block_devic
e.py", line 215, in attach
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5]     wait_func(context, vol['id'])
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.
py", line 901, in _await_block_device_map_created
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5]     attempts=attempts)
2014-02-03 12:37:26.602 25255 TRACE nova.compute.manager [instance: 5691dc36-983e-403b-a46f-90ad81e36fa5] VolumeNotCreated: Volume 72128f54-3099-4cd3-80d5-80e7146e8a1e 
did not finish being created even after we waited 66 seconds or 60 attempts.
Comment 8 Solly Ross 2014-03-18 17:43:50 EDT
I realized that I haven't actually written down what went on in the upstream discussion here.

Basically, there were some people in favor of having a variable timeout, and some that were in favor of revamping the API to automatically customize the timeout, and so on and so forth.  However, it was mentioned that technically, this is just a convenience operation that wraps two separate operations: creating a volume from an image or snapshot, and then booting an instance from the resulting volume.

For the moment, I think we should add a release note that under certain configurations users should avoid using the convenience method, and just perform the two methods separately.

Hopefully, eventually we'll be able to reach a consensus upstream about how to fix this.
Comment 9 Dafna Ron 2014-03-31 09:38:58 EDT
Following what Solly wrote, I think that if this option currently fails on small volumes (10GB), and there is no fix decided on for the near future, a release note is not enough :) 
we need to make it a tech preview or/and set timeout to a larger value on red hat installations + add a workaround on how to increase the timeout if needed.
Comment 12 Andres Toomsalu 2015-03-05 06:29:40 EST
Just for feedback: this issue is still very much alive and causing problems in production deployments with volume (SAN) backends - where volume sizes are larger than in development environments. Affects heavily backup/snaphot restoration process - which easily run into timeout limits.
Comment 15 jwang 2015-08-29 09:00:27 EDT
I hit this issue again on RHELOSP6.

1.
Cinder backend is LVM

2.
Glance image virtual size is 110G
Comment 16 Lee Yarwood 2015-10-16 12:47:24 EDT
(In reply to Andres Toomsalu from comment #12)
> Just for feedback: this issue is still very much alive and causing problems
> in production deployments with volume (SAN) backends - where volume sizes
> are larger than in development environments. Affects heavily backup/snaphot
> restoration process - which easily run into timeout limits.

(In reply to jwang from comment #15)
> I hit this issue again on RHELOSP6.
> 
> 1.
> Cinder backend is LVM
> 
> 2.
> Glance image virtual size is 110G

Hello Dafna, Andres, jwang, Jack, can you confirm which version of nova you are using in your environments?

I believe the following change introduced configurables in Juno / RHEL OSP 6 and then Icehouse / RHEL OSP 5 (via 2014.1.4) that can be used here :

[juno] Make the block device mapping retries configurable
https://review.openstack.org/#/c/102891/

[stable/icehouse] Make the block device mapping retries configurable
https://review.openstack.org/#/c/129276/

~~~
Make the block device mapping retries configurable

When booting instances passing in block-device and increasing the
volume size, instances can go in to error state if the volume takes
longer to create than the hard code value (max_tries(180)/wait_between(1))
set in nova/compute/manager.py
def _await_block_device_map_created(self,
                                    context,
                                    vol_id,
                                    max_tries=180,
                                    wait_between=1):
To fix this, max_retries/wait_between should be made configurable.
Looking through the different releases, Grizzly was 30, Havana was
60 , IceHouse is 180.
This change adds two configuration options:
a)  `block_device_allocate_retries` which can be set in nova.conf
by the user to configure the number of block device mapping retries.
It defaults to 60 and replaces the max_tries argument in the above method.
b) `block_device_allocate_retries_interval` which allows the user
to specify the time interval between consecutive retries. It defaults to 3
and replaces wait_between argument in the above method.
~~~
Comment 17 Lee Yarwood 2015-11-02 09:42:51 EST
Closing this out with CURRENTRELEASE given c#16, please reopen if this is still a problem.
Comment 18 Andres Toomsalu 2015-11-30 07:24:11 EST
We were using Icehouse.

Note You need to log in before you can comment on or make changes to this bug.