Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1623347

Summary: Can't launch instance from the image if create volume for the image at the same time on openstack
Product: Red Hat OpenStack Reporter: mxie <mxie>
Component: openstack-cinderAssignee: Cinder Bugs List <cinder-bugs>
Status: CLOSED NOTABUG QA Contact: Avi Avraham <aavraham>
Severity: high Docs Contact: Kim Nylander <knylande>
Priority: high    
Version: 13.0 (Queens)CC: abishop, berrange, cschwede, dasmith, eglynn, eharney, jhakimra, juzhou, kchamart, mbooth, mxie, mzhan, pgrist, sbauza, sgordon, srevivo, tzheng, vromanso, xiaodwan
Target Milestone: ---Keywords: TestBlocker, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-10 06:42:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
nova-compute.log none

Description mxie@redhat.com 2018-08-29 07:31:20 UTC
Created attachment 1479396 [details]
nova-compute.log

Description of problem:
Can't launch instance from the image which has operating system on openstack

Version-Release number of selected component (if applicable):
openstack-glance-16.0.1-2.el7ost.noarch


How reproducible:
100%

Steps to Reproduce:
1.Convert a guest to openstack by virt-v2v
# virt-v2v rhel7.6-lvm -o glance -of qcow2
[   2.0] Opening the source -i libvirt rhel7.6-lvm
[   2.0] Creating an overlay to protect the source from being modified
[   2.2] Initializing the target -o glance
[   2.2] Opening the overlay
[  22.6] Inspecting the overlay
virt-v2v: warning: mount: mount exited with status 32: mount: 
/dev/mapper/rhel_bootp--73--75--10-home is write-protected, mounting 
read-only
mount: unknown filesystem type '(null)' (ignored)
[  49.9] Checking for sufficient free disk space in the guest
[  49.9] Estimating space required on target for each disk
[  49.9] Converting Red Hat Enterprise Linux Server 7.6 Beta (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 160.9] Mapping filesystem data to avoid copying unused and blank areas
[ 161.1] Closing the overlay
[ 161.6] Checking if the guest needs BIOS or UEFI to boot
[ 161.6] Assigning disks to buses
[ 161.6] Copying disk 1/1 to /var/tmp/glance.l1dr1R/sda (qcow2)
    (100.00/100%)
[ 355.6] Creating output metadata
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| architecture     | x86_64                               |
| checksum         | 7b986c4ef9d758f051c2ef7e5ebeb0b7     |
| container_format | bare                                 |
| created_at       | 2018-08-29T06:13:15Z                 |
| disk_format      | qcow2                                |
| hw_cpu_cores     | 2                                    |
| hw_cpu_sockets   | 1                                    |
| hw_disk_bus      | virtio                               |
| hw_rng_model     | virtio                               |
| hw_video_model   | qxl                                  |
| hw_vif_model     | virtio                               |
| hypervisor_type  | kvm                                  |
| id               | 1093239d-8289-4136-8f53-12d45952fa25 |
| min_disk         | 0                                    |
| min_ram          | 2048                                 |
| name             | rhel7.6-lvm                          |
| os_distro        | rhel                                 |
| os_type          | linux                                |
| os_version       | 7.6                                  |
| owner            | 32f578286ede446faa1ebed0ea772abe     |
| protected        | False                                |
| size             | 12619743232                          |
| status           | active                               |
| tags             | []                                   |
| updated_at       | 2018-08-29T06:15:47Z                 |
| virtual_size     | None                                 |
| visibility       | shared                               |
| vm_mode          | hvm                                  |
+------------------+--------------------------------------+
[ 512.9] Finishing off


2.Input instance name,flavor and network info to launch the instance from image on openstack after conversion

3.But the instance is failed to launched with below error on openstack

Message: Build of instance ad703618-6c71-429b-a47d-b3492a766f88 aborted: Volume 50380aac-4033-4193-a03b-83e5dd51f763 did not finish being created even after we waited 190 seconds or 61 attempts. And its status is downloading.
Code    : 500
Details : File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1840, in _do_build_and_run_instance filter_properties, request_spec) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2062, in _build_and_run_instance bdms=block_device_mapping) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2014, in _build_and_run_instance block_device_mapping) as resources: File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2221, in _build_resources reason=e.format_message()) 


Actual results:
As above description

Expected results:
Can launch instance from the image which is converted by virt-v2v on openstack successfully

Additional info:
1.If create image and select guest image which has OS at file option from local server on openstack, also failed to launch instance from this image 
2.Can launch instance from image which has no OS on openstack

Comment 1 Matthew Booth 2018-08-30 15:05:50 UTC
A quick look at the report suggests cinder failed to create the volume. Just a reminder that whoever looks at this is going to want sosreports.

Comment 2 mxie@redhat.com 2018-09-06 02:52:12 UTC
Hi Matthew,

   My colleague HuChenLi (chhu@) found two workarounds for this problem,could you please help to confirm if the bug needs to be fixed? Thanks 

(1) Don't create volume when launch instance from the image directly which has OS on openstack

(2)Change the block_device_allocate_retries to large number in nova.cfg and restart nova-compute service, then can launch instance from the image directly on openstack 

# Number of times to retry block device allocation on failures. Starting with
# Liberty, Cinder can use image volume cache. This may help with block device
# allocation performance. Look at the cinder image_volume_cache_enabled
# configuration option.
#
# Possible values:
#
# * 60 (default)
# * If value is 0, then one attempt is made.
# * Any negative value is treated as 0.
# * For any value > 0, total attempts are (value + 1)
#  (integer value)
#block_device_allocate_retries=60
block_device_allocate_retries=600

Comment 4 Christian Schwede (cschwede) 2018-09-07 13:28:17 UTC
Can you please provide full sosreports to further debug this? Thx!

Comment 5 mxie@redhat.com 2018-09-10 04:00:28 UTC
Hi Christian,

   There is so many logs in openstack server, could you please list which logs you want?

Thanks

Comment 6 Matthew Booth 2018-09-10 17:11:35 UTC
(In reply to mxie from comment #5)
> Hi Christian,
> 
>    There is so many logs in openstack server, could you please list which
> logs you want?

If you provide reports with the sosreport tool it will fetch all of them.

Matt