Bug 1003639 - openstack-nova: Unable to boot an instance on Folsom all-in-one with local storage (qemu-img info Could not open '/var/lib/nova/instances/).
openstack-nova: Unable to boot an instance on Folsom all-in-one with local st...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
2.0 (Folsom)
x86_64 Linux
high Severity high
: ---
: 4.0
Assigned To: Vladan Popovic
Ami Jeain
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-02 10:29 EDT by Daniel Korn
Modified: 2016-04-26 19:31 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-03 05:18:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
compute.log (79.68 KB, text/plain)
2013-09-02 10:31 EDT, Daniel Korn
no flags Details
packstack answer file (7.38 KB, text/plain)
2013-09-02 10:56 EDT, Daniel Korn
no flags Details

  None (edit)
Description Daniel Korn 2013-09-02 10:29:32 EDT
openstack-nova: Unable to boot an instance on Folsom all-in-one with local storage (qemu-img info Could not open '/var/lib/nova/instances/).

Environment: 
-------------
qemu-kvm-rhev-0.12.1.2-2.355.el6_4.4.x86_64
qemu-img-rhev-0.12.1.2-2.355.el6_4.4.x86_64
libvirt-0.10.2-18.el6_4.9.x86_64
openstack-nova-2012.2.4-5.el6ost.noarch
openstack-packstack-2012.2.3-0.12.dev495.el6ost.noarch

Description of problem:
------------------------
After deploying latest Folsom (puddle: 2013-07-09.2) using packstack and uploading an image, when trying to launch an Instance, it immediately turns into ERROR state in the Status field (seen on the nova list). from compute.log it seems that qemu-img info failed to access the instances folder ('/var/lib/nova/instances/') . 

looking at the instances folder permission it looks like this (after installation):

[root@cougar10 tmp(keystone_admin)]$ ls -lZ /var/lib/nova/
drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 buckets
drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 CA
drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 images
drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 instances

[root@cougar10 tmp(keystone_admin)]$ ls -lZ /var/lib/nova/instances/
drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 _base
drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 instance-00000001

Note: In order to isolate the problem - I've disabled the SELinux on the machine and the problem still occurred. 

nova list:
-----------
[root@cougar10 tmp(keystone_admin)]$ nova list
+--------------------------------------+-------+--------+----------+
| ID                                   | Name  | Status | Networks |
+--------------------------------------+-------+--------+----------+
| ec11d9c9-2b14-4ede-9df4-9e756770af63 | rhl01 | ERROR  |          |
+--------------------------------------+-------+--------+----------+

nova show output:
--------------------
| fault | {u'message': u'NoValidHost', u'code': 500, u'created': u'2013-09-02T12:16:55Z'} |
| OS-EXT-STS:power_state | 0                                                                      


Steps to Reproduce:
--------------------
1. Install openstack all-in-one with local-storage (folsom latest puddle) 
2. boot instance.


Results:
---------
Instance switch into 'Error' state

compute.log:
--------------
2013-09-02 15:16:46 AUDIT nova.compute.manager [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Starting instance...
2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Attempting claim: memory 2048 MB, disk 20 GB, VCPUs 1
2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Total memory: 32100 MB, used: 512 MB, free: 31588 MB
2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Memory limit: 48150 MB, free: 47638 MB
2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Total disk: 442 GB, used: 0 GB, free: 442 GB
2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Disk limit not specified, defaulting to unlimited
2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Total VCPUs: 6, used: 0
2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] CPU limit: 96
2013-09-02 15:16:53 INFO nova.virt.libvirt.driver [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Creating image
2013-09-02 15:16:53 ERROR nova.compute.manager [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Instance failed to spawn
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Traceback (most recent call last):
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 756, in _spawn
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     block_device_info)
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     temp_level, payload)
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     self.gen.next()
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     return f(*args, **kw)
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1112, in spawn
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     admin_pass=admin_password)
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1378, in _create_image
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     project_id=instance['project_id'])
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 132, in cache
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     *args, **kwargs)
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 206, in create_image
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     if size and size < disk.get_disk_size(base):
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 115, in get_disk_size
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     size = images.qemu_img_info(path)['virtual size']
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 50, in qemu_img_info
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     'qemu-img', 'info', path)
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 210, in execute
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63]     cmd=' '.join(cmd))
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] ProcessExecutionError: Unexpected error while running command.
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Command: env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/64773a31d97a746a69369038342735cc2058f9bd
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Exit code: 1
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Stdout: ''
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Stderr: "qemu-img: Could not open '/var/lib/nova/instances/_base/64773a31d97a746a69369038342735cc2058f9bd'\n"
2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] 
2013-09-02 15:16:53 INFO nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Aborting claim: [Claim ec11d9c9-2b14-4ede-9df4-9e756770af63: 2048 MB memory, 20 GB disk, 1 VCPUS]
2013-09-02 15:16:55 ERROR nova.compute.manager [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Build error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 503, in _run_instance\n    injected_files, admin_password)\n', '  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 756, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped\n    temp_level, payload)\n', '  File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped\n    return f(*args, **kw)\n', '  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1112, in spawn\n    admin_pass=admin_password)\n', '  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1378, in _create_image\n    project_id=instance[\'project_id\'])\n', '  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 132, in cache\n    *args, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 206, in create_image\n    if size and size < disk.get_disk_size(base):\n', '  File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 115, in get_disk_size\n    size = images.qemu_img_info(path)[\'virtual size\']\n', '  File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 50, in qemu_img_info\n    \'qemu-img\', \'info\', path)\n', '  File "/usr/lib/python2.6/site-packages/nova/utils.py", line 210, in execute\n    cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/64773a31d97a746a69369038342735cc2058f9bd\nExit code: 1\nStdout: \'\'\nStderr: "qemu-img: Could not open \'/var/lib/nova/instances/_base/64773a31d97a746a69369038342735cc2058f9bd\'\\n"\n']
2013-09-02 15:16:57 5216 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 29540
2013-09-02 15:16:57 5216 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 422
2013-09-02 15:16:57 5216 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 5
2013-09-02 15:16:57 5216 INFO nova.compute.resource_tracker [-] Compute_service record updated for cougar10.scl.lab.tlv.redhat.com 
2013-09-02 15:16:57 5216 INFO nova.compute.manager [-] Updating host status
2013-09-02 15:17:57 5216 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 29540
2013-09-02 15:17:57 5216 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 422
2013-09-02 15:17:57 5216 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 5
Comment 1 Daniel Korn 2013-09-02 10:31:44 EDT
Created attachment 792882 [details]
compute.log
Comment 2 Perry Myers 2013-09-02 10:38:58 EDT
Can you please try with RHOS 3.0 / Grizzly?

RHOS 2.1 / Folsom went officially unsupported as soon as RHOS 3.0 was released because it was never formally released as as supported product.  If you can reproduce on Grizzly, we'd definitely want to dig into this more deeply.
Comment 3 Daniel Korn 2013-09-02 10:56:22 EDT
Created attachment 792891 [details]
packstack answer file
Comment 5 Vladan Popovic 2013-10-01 09:11:22 EDT
(In reply to daniel korn from comment #3)
> Created attachment 792891 [details]
> packstack answer file

Hi Daniel,
is this answer file for Grizzly or Folsom?
I can't run it on Grizzly, the following errors occur:
 - ERROR : Failed handling answer file: 'Parser cannot find option CONFIG_QUANTUM_INSTALL in answer file.'

   ---   (after adding it i get)

 - ERROR : Failed handling answer file: 'Parser cannot find option CONFIG_NOVA_CONDUCTOR_HOST in answer file.'

 - ERROR : Failed handling answer file: 'Parser cannot find option CONFIG_NOVA_NETWORK_DEFAULTFLOATINGPOOL in answer file.'

... and so on.
It looks to me like this answer file if for Folsom. I reproduced the answer file for Grizzly and installed everything 

Can't reproduce it in Grizzly with the same image and instance.
The instances permissions I have are these:

[root@localhost ~(keystone_admin)]# ls -lZ /var/lib/nova/instances/
drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 b491fc3d-cef1-40e8-a4c2-6047912a5aa9
drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 _base
drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 locks

[root@localhost ~(keystone_admin)]# nova list
+--------------------------------------+--------------------------+--------+--------------------------+
| ID                                   | Name                     | Status | Networks                 |
+--------------------------------------+--------------------------+--------+--------------------------+
| b491fc3d-cef1-40e8-a4c2-6047912a5aa9 | fedora-prebuilt-instance | ACTIVE | novanetwork=192.168.32.2 |
+--------------------------------------+--------------------------+--------+--------------------------+



In Folsom i get the following:

[root@localhost ~(keystone_admin)]$ nova list
+--------------------------------------+--------------------------+--------+----------+
| ID                                   | Name                     | Status | Networks |
+--------------------------------------+--------------------------+--------+----------+
| a4926c8d-757a-433c-be6c-36906482563f | fedora-prebuilt-instance | ERROR  |          |
+--------------------------------------+--------------------------+--------+----------+

[root@localhost ~(keystone_admin)]$ ls -lZ /var/lib/nova/instances/
drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 _base
drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 instance-00000001


should I close this as current release?

Note You need to log in before you can comment on or make changes to this bug.