Bug 1266246 - Cinder volumes created from images are marked as available even though they were not successfully transferred and were unbootable
Cinder volumes created from images are marked as available even though they w...
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
5.0 (RHEL 6)
Unspecified Unspecified
urgent Severity urgent
: ---
: 5.0 (RHEL 6)
Assigned To: Flavio Percoco
: ZStream
Depends On:
  Show dependency treegraph
Reported: 2015-09-24 17:04 EDT by John Fulton
Modified: 2018-02-08 05:57 EST (History)
7 users (show)

See Also:
Fixed In Version: openstack-cinder-2014.1.5-2.el6ost, openstack-cinder-2014.1.5-2.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-12-22 09:26:37 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description John Fulton 2015-09-24 17:04:47 EDT
A. Description of problem:
Cinder volumes created from images are marked as available even though they were not successfully transferred and were unbootable.

B. Version-Release number of selected component (if applicable):

C. How reproducible:
Only occurs during a load test 25% of the time. 

D Steps to Reproduce:
1. Boot 60 instances from the same qcow2 image
2. Observe that approximately 15 will not boot
3. Inspect each of the 15 and see that their cinder volume is not an accurate bootable copy of the image

E. Actual results:
- 100% of volumes report that they are available
- 25% of volumes are unbootable and should not have been marked as available

F. Expected results:
If volumes were unable to boot, then they should not have been marked as available. 

E. Additional info:
Using 3par cinder driver https://blueprints.launchpad.net/cinder/+spec/hp3par-volume-driver
Comment 5 Flavio Percoco 2015-09-25 09:13:59 EDT
From the logs, it seems that this is not a Glance issue.

The image is downloaded correctly and the checksum verification doesn't fail. Furthermore, the image is successfully copied into the volume. Then the image metadata is also copied to the volume metadata.

Would it me possible to have compute's logs as well?

According to the logs, the bootable flag is being set to the volume as well.
Comment 6 John Fulton 2015-09-25 09:17:31 EDT
Yes, I will get the logs from the compute nodes.
Comment 8 John Fulton 2015-09-29 12:23:34 EDT
An additional test was conducted. 
- Create 128 VMs, 4 at once, then delete them and do this 32 times
- Of the 128 VMs created, 34 of them had the symptoms of this bug
Comment 9 John Fulton 2015-10-02 13:22:07 EDT
An additional test was conducted after setting volume_clear=zero in cinder.conf. 
- Create 512 VMs, 8 at once, then delete them and do this 64 times
- Of 512 VMs created, 444 passed the test but 68 had the symptoms of this bug
Comment 11 Sergey Gotliv 2015-10-20 07:21:48 EDT
We are working to provide an official hotfix. I will update this bug with the list of the relevant fixes once we complete review process.
Comment 12 Sergey Gotliv 2015-11-01 22:26:18 EST
Fixed here https://bugzilla.redhat.com/show_bug.cgi?id=1255523#c20
Comment 15 errata-xmlrpc 2015-12-22 09:26:37 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.