openstack-nova: Unable to boot an instance on Folsom all-in-one with local storage (qemu-img info Could not open '/var/lib/nova/instances/). Environment: ------------- qemu-kvm-rhev-0.12.1.2-2.355.el6_4.4.x86_64 qemu-img-rhev-0.12.1.2-2.355.el6_4.4.x86_64 libvirt-0.10.2-18.el6_4.9.x86_64 openstack-nova-2012.2.4-5.el6ost.noarch openstack-packstack-2012.2.3-0.12.dev495.el6ost.noarch Description of problem: ------------------------ After deploying latest Folsom (puddle: 2013-07-09.2) using packstack and uploading an image, when trying to launch an Instance, it immediately turns into ERROR state in the Status field (seen on the nova list). from compute.log it seems that qemu-img info failed to access the instances folder ('/var/lib/nova/instances/') . looking at the instances folder permission it looks like this (after installation): [root@cougar10 tmp(keystone_admin)]$ ls -lZ /var/lib/nova/ drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 buckets drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 CA drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 images drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 instances [root@cougar10 tmp(keystone_admin)]$ ls -lZ /var/lib/nova/instances/ drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 _base drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 instance-00000001 Note: In order to isolate the problem - I've disabled the SELinux on the machine and the problem still occurred. nova list: ----------- [root@cougar10 tmp(keystone_admin)]$ nova list +--------------------------------------+-------+--------+----------+ | ID | Name | Status | Networks | +--------------------------------------+-------+--------+----------+ | ec11d9c9-2b14-4ede-9df4-9e756770af63 | rhl01 | ERROR | | +--------------------------------------+-------+--------+----------+ nova show output: -------------------- | fault | {u'message': u'NoValidHost', u'code': 500, u'created': u'2013-09-02T12:16:55Z'} | | OS-EXT-STS:power_state | 0 Steps to Reproduce: -------------------- 1. Install openstack all-in-one with local-storage (folsom latest puddle) 2. boot instance. Results: --------- Instance switch into 'Error' state compute.log: -------------- 2013-09-02 15:16:46 AUDIT nova.compute.manager [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Starting instance... 2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Attempting claim: memory 2048 MB, disk 20 GB, VCPUs 1 2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Total memory: 32100 MB, used: 512 MB, free: 31588 MB 2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Memory limit: 48150 MB, free: 47638 MB 2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Total disk: 442 GB, used: 0 GB, free: 442 GB 2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Disk limit not specified, defaulting to unlimited 2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Total VCPUs: 6, used: 0 2013-09-02 15:16:46 AUDIT nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] CPU limit: 96 2013-09-02 15:16:53 INFO nova.virt.libvirt.driver [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Creating image 2013-09-02 15:16:53 ERROR nova.compute.manager [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Instance failed to spawn 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Traceback (most recent call last): 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 756, in _spawn 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] block_device_info) 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] temp_level, payload) 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] self.gen.next() 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] return f(*args, **kw) 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1112, in spawn 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] admin_pass=admin_password) 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1378, in _create_image 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] project_id=instance['project_id']) 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 132, in cache 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] *args, **kwargs) 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 206, in create_image 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] if size and size < disk.get_disk_size(base): 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 115, in get_disk_size 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] size = images.qemu_img_info(path)['virtual size'] 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 50, in qemu_img_info 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] 'qemu-img', 'info', path) 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] File "/usr/lib/python2.6/site-packages/nova/utils.py", line 210, in execute 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] cmd=' '.join(cmd)) 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] ProcessExecutionError: Unexpected error while running command. 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Command: env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/64773a31d97a746a69369038342735cc2058f9bd 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Exit code: 1 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Stdout: '' 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Stderr: "qemu-img: Could not open '/var/lib/nova/instances/_base/64773a31d97a746a69369038342735cc2058f9bd'\n" 2013-09-02 15:16:53 5216 TRACE nova.compute.manager [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] 2013-09-02 15:16:53 INFO nova.compute.resource_tracker [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] Aborting claim: [Claim ec11d9c9-2b14-4ede-9df4-9e756770af63: 2048 MB memory, 20 GB disk, 1 VCPUS] 2013-09-02 15:16:55 ERROR nova.compute.manager [req-72d8160c-9e20-4925-8007-ab58c20f01b8 f5d18390920e4fc98f05e86d34a66bdd 48b3eead870b4bba8ae1db5cea0c6a4d] [instance: ec11d9c9-2b14-4ede-9df4-9e756770af63] Build error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 503, in _run_instance\n injected_files, admin_password)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 756, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped\n temp_level, payload)\n', ' File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped\n return f(*args, **kw)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1112, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1378, in _create_image\n project_id=instance[\'project_id\'])\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 132, in cache\n *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 206, in create_image\n if size and size < disk.get_disk_size(base):\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 115, in get_disk_size\n size = images.qemu_img_info(path)[\'virtual size\']\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 50, in qemu_img_info\n \'qemu-img\', \'info\', path)\n', ' File "/usr/lib/python2.6/site-packages/nova/utils.py", line 210, in execute\n cmd=\' \'.join(cmd))\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/64773a31d97a746a69369038342735cc2058f9bd\nExit code: 1\nStdout: \'\'\nStderr: "qemu-img: Could not open \'/var/lib/nova/instances/_base/64773a31d97a746a69369038342735cc2058f9bd\'\\n"\n'] 2013-09-02 15:16:57 5216 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 29540 2013-09-02 15:16:57 5216 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 422 2013-09-02 15:16:57 5216 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 5 2013-09-02 15:16:57 5216 INFO nova.compute.resource_tracker [-] Compute_service record updated for cougar10.scl.lab.tlv.redhat.com 2013-09-02 15:16:57 5216 INFO nova.compute.manager [-] Updating host status 2013-09-02 15:17:57 5216 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 29540 2013-09-02 15:17:57 5216 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 422 2013-09-02 15:17:57 5216 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 5
Created attachment 792882 [details] compute.log
Can you please try with RHOS 3.0 / Grizzly? RHOS 2.1 / Folsom went officially unsupported as soon as RHOS 3.0 was released because it was never formally released as as supported product. If you can reproduce on Grizzly, we'd definitely want to dig into this more deeply.
Created attachment 792891 [details] packstack answer file
(In reply to daniel korn from comment #3) > Created attachment 792891 [details] > packstack answer file Hi Daniel, is this answer file for Grizzly or Folsom? I can't run it on Grizzly, the following errors occur: - ERROR : Failed handling answer file: 'Parser cannot find option CONFIG_QUANTUM_INSTALL in answer file.' --- (after adding it i get) - ERROR : Failed handling answer file: 'Parser cannot find option CONFIG_NOVA_CONDUCTOR_HOST in answer file.' - ERROR : Failed handling answer file: 'Parser cannot find option CONFIG_NOVA_NETWORK_DEFAULTFLOATINGPOOL in answer file.' ... and so on. It looks to me like this answer file if for Folsom. I reproduced the answer file for Grizzly and installed everything Can't reproduce it in Grizzly with the same image and instance. The instances permissions I have are these: [root@localhost ~(keystone_admin)]# ls -lZ /var/lib/nova/instances/ drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 b491fc3d-cef1-40e8-a4c2-6047912a5aa9 drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 _base drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 locks [root@localhost ~(keystone_admin)]# nova list +--------------------------------------+--------------------------+--------+--------------------------+ | ID | Name | Status | Networks | +--------------------------------------+--------------------------+--------+--------------------------+ | b491fc3d-cef1-40e8-a4c2-6047912a5aa9 | fedora-prebuilt-instance | ACTIVE | novanetwork=192.168.32.2 | +--------------------------------------+--------------------------+--------+--------------------------+ In Folsom i get the following: [root@localhost ~(keystone_admin)]$ nova list +--------------------------------------+--------------------------+--------+----------+ | ID | Name | Status | Networks | +--------------------------------------+--------------------------+--------+----------+ | a4926c8d-757a-433c-be6c-36906482563f | fedora-prebuilt-instance | ERROR | | +--------------------------------------+--------------------------+--------+----------+ [root@localhost ~(keystone_admin)]$ ls -lZ /var/lib/nova/instances/ drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 _base drwxr-xr-x. nova nova unconfined_u:object_r:nova_var_lib_t:s0 instance-00000001 should I close this as current release?