Bug 1121591

Summary: Viewing console while launching a VDI instances causes error status
Product: Red Hat OpenStack Reporter: Tzach Shefi <tshefi>
Component: openstack-novaAssignee: Russell Bryant <rbryant>
Status: CLOSED NOTABUG QA Contact: Ami Jeain <ajeain>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 5.0 (RHEL 6)CC: ndipanov, tshefi, yeylon
Target Milestone: ---   
Target Release: 5.0 (RHEL 7)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-13 15:05:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Compute log
none
new compute log
none
Cinder volume log none

Description Tzach Shefi 2014-07-21 09:58:03 UTC 
Created attachment 919576 [details]
Compute log

Description of problem: If you access an instance's console while launching an a VDI image, instance status: error. Ran this a few times if however you wait until instance has completed booting before accessing console it boots up fine. This doesn't happen with Cirros QCOW.  

VDI image used, must be uncompressed:
http://downloads.sourceforge.net/virtualboximage/dsl-4.2.5-x86.7z

Version-Release number of selected component (if applicable):
RHEL 6.5 
openstack-nova-compute-2014.1.1-2.el6ost.noarch

How reproducible:
Every time, tested three times.

Steps to Reproduce:
1. In my case Glance uses a Gluster share, don't think it's related.
2. Upload VDI image to Glance.
3. Launch an instance from VDI image.
4. Before instances completes boot process, go to instance's console.
5. Instance's failed to boot, status error. 
6. Repeat above steps, this time wait for instance to complete boot process then go to console, status is active. 

Actual results:

Failed to boot VDI instance, when accessing instance's console before instance is active.  

On compute.log instance ID  13b23f83-9587-4068-a6d1-d2b8034d0e80 status error, while instance ID 542bfd5e-866e-4af4-ab59-f1ecb86bfa2f booted up fine. 

Expected results:

VDI instance should boot up even if you look at console during boot process, just like Cirros qcow.
Comment 1 Russell Bryant 2014-07-30 15:17:02 UTC 
There's several other errors in the compute log that show that the instance would have failed to boot for another reason.  Primarily:

2014-07-21 12:10:21.655 7692 TRACE nova.compute.manager [instance: 13b23f83-9587-4068-a6d1-d2b8034d0e80] FlavorDiskTooSmall: Flavor's disk is too small for requested image.

I want to make sure the boot failed for the reason you think it did.  Can you try testing again that shows this failure with a clean log with only failures related to the console?
Comment 2 Tzach Shefi 2014-08-04 08:50:10 UTC 
Created attachment 923803 [details]
new compute log
Comment 3 Tzach Shefi 2014-08-04 08:51:46 UTC 
See above compute.log, while trying to recreate this I may have found a new bug. 
Using same VDI image source, can't boot instance:                                                                                                                                        
2014-08-04 11:42:25.318 16648 ERROR nova.compute.manager [req-9b5a8aa8-3bad-47bc-b1d4-4e732166810d b643e376b9f746c89744b218b3b3288e 12f4debd3b314201bbab93341f94d67e] [instance: 75694195-1568-4234-810e-b60d4e798486] Error: Unexpected error while running command.                                                                     
Command: env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/759fd7c54b5726bb3e060daeb676e15f44be97f2.part                                               
Exit code: 1                                                                                                                                                         
Stdout: ''                                                                                                                                                           
Stderr: "Could not open '/var/lib/nova/instances/_base/759fd7c54b5726bb3e060daeb676e15f44be97f2.part': Operation not permitted\n"                                    
2014-08-04 11:42:25.318 16648 TRACE nova.compute.manager [instance: 75694195-1568-4234-810e-b60d4e798486] Traceback (most recent call last):                         
2014-08-04 11:42:25.318 16648 TRACE nova.compute.manager [instance: 75694195-1568-4234-810e-b60d4e798486]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1305, in _build_instance                                                                 

Any tips would be welcomed :)
Comment 4 Tzach Shefi 2014-08-04 08:53:21 UTC 
Forgot to mention, Horizon gives the usual:

Error: Failed to launch instance "test": Please try again later [Error: No valid host was found.
Comment 5 Russell Bryant 2014-08-06 15:21:23 UTC 
(In reply to Tzach Shefi from comment #3)
> See above compute.log, while trying to recreate this I may have found a new
> bug. 
> Using same VDI image source, can't boot instance:                           
> 
> 2014-08-04 11:42:25.318 16648 ERROR nova.compute.manager
> [req-9b5a8aa8-3bad-47bc-b1d4-4e732166810d b643e376b9f746c89744b218b3b3288e
> 12f4debd3b314201bbab93341f94d67e] [instance:
> 75694195-1568-4234-810e-b60d4e798486] Error: Unexpected error while running
> command.                                                                     
> Command: env LC_ALL=C LANG=C qemu-img info
> /var/lib/nova/instances/_base/759fd7c54b5726bb3e060daeb676e15f44be97f2.part 
> 
> Exit code: 1                                                                
> 
> Stdout: ''                                                                  
> 
> Stderr: "Could not open
> '/var/lib/nova/instances/_base/759fd7c54b5726bb3e060daeb676e15f44be97f2.
> part': Operation not permitted\n"                                    
> 2014-08-04 11:42:25.318 16648 TRACE nova.compute.manager [instance:
> 75694195-1568-4234-810e-b60d4e798486] Traceback (most recent call last):    
> 
> 2014-08-04 11:42:25.318 16648 TRACE nova.compute.manager [instance:
> 75694195-1568-4234-810e-b60d4e798486]   File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1305, in
> _build_instance                                                             
> 
> 
> Any tips would be welcomed :)

Have you checked for any SELinux errors?

Also, how was this deployment done?
Comment 6 Tzach Shefi 2014-08-07 06:49:30 UTC 
Packstack deployment, since then I tired to run various other VDI images downloaded from virtualbox site (guess they should be ok if they are posted). 

No avc found on audit.log, but just to be sure setenforce 0 for testing, didn't help.  

Adding insult to injury it even failed to create a volume from VDI image, status error.  

Cinder log (also attached)

2014-08-07 09:38:37.867 28906 TRACE oslo.messaging.rpc.dispatcher     raise exception.ImageCopyFailure(reason=ex.stderr)
2014-08-07 09:38:37.867 28906 TRACE oslo.messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: Could not open '/var/lib/cinder/conversion/tmpgbL3B0': Operation not permitted


Should we keep working on this here, or open a new bug Nova VDI image problem?
Comment 7 Tzach Shefi 2014-08-07 06:50:49 UTC 
Created attachment 924721 [details]
Cinder volume log

should open another bug about cinder volume from VDI image?
Comment 8 Russell Bryant 2014-08-13 15:05:12 UTC 
Actually, looking at this again, VDI images aren't something supported at all.  Also, an image made for use with virtualbox may not work at all, either.  So, I don't think this is a bug, and you just need to get a valid image.