Bug 1192862 - Glance fails to start after RHEL-OSP6 install with Ceph backend (missing known_stores)
Summary: Glance fails to start after RHEL-OSP6 install with Ceph backend (missing know...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-foreman-installer
Version: 6.0 (Juno)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z2
: Installer
Assignee: Jason Guiditta
QA Contact: Yogev Rabl
URL:
Whiteboard:
: 1183946 1196921 (view as bug list)
Depends On:
Blocks: 1163445
TreeView+ depends on / blocked
 
Reported: 2015-02-16 00:04 UTC by Alessandro Vozza
Modified: 2023-02-22 23:02 UTC (History)
26 users (show)

Fixed In Version: openstack-foreman-installer-3.0.17-1.el7ost
Doc Type: Bug Fix
Doc Text:
The known_stores option was incorrectly set for Glance, which caused Ceph backend for Glance to fail to start. This fix configures known_stores correctly and Glance now starts successfully.
Clone Of:
Environment:
Last Closed: 2015-04-07 15:08:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1196921 1 None None None 2021-01-20 06:05:38 UTC
Red Hat Product Errata RHSA-2015:0791 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux OpenStack Platform Installer update 2015-04-07 19:07:29 UTC

Internal Links: 1196921

Description Alessandro Vozza 2015-02-16 00:04:15 UTC
Description of problem:

After deployment of RHEL-OSP6 and manual installation of Ceph backend, set "Include Glance" manually to true (in Advanced Configuration/Controller) but glance fails to start.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Deploy RHEL-OSP6 in HA on 3 controllers
2. Manually setup Ceph (HEALTH_OK)
3. Set the flag "Install glance" to true
4. Watch puppet fails

Actual results:


Expected results:
glance-api starting regularly. 

Additional info:

After successful deployment (RHEL-OSP6 in HA, on hardware), I proceed to install Ceph manually. After successful Ceph cluster creation, I went back to Foreman and set "Install Glance" to true manually, run puppet on the controllers and observed glance failing. I then run glance manually:

sudo -u glance -H glance-api

and got the error

ERROR: Store for scheme rbd not found

in glance-api.conf, the line for stores is commented out; however, setting it manually to:

stores=glance.store.rbd.Store

gets reverted by puppet:

Notice: /Stage[main]/Glance::Api/Glance_api_config[glance_store/stores]/ensure: removed
Info: /Stage[main]/Glance::Api/Glance_api_config[glance_store/stores]: Scheduling refresh of Exec[glance-manage db_sync]
Info: /Stage[main]/Glance::Api/Glance_api_config[glance_store/stores]: Scheduling refresh of Service[glance-api]
Notice: /Stage[main]/Glance::Registry/Exec[glance-manage db_sync]: Triggered 'refresh' from 1 events
Info: /Stage[main]/Glance::Registry/Exec[glance-manage db_sync]: Scheduling refresh of Service[glance-api]
Info: /Stage[main]/Glance::Registry/Exec[glance-manage db_sync]: Scheduling refresh of Service[glance-registry]
Notice: /Stage[main]/Glance::Registry/Service[glance-registry]: Triggered 'refresh' from 1 events
Error: Could not start Service[glance-api]: Execution of '/usr/bin/systemctl start openstack-glance-api' returned 1: Job for openstack-glance-api.service failed. See 'systemctl status openstack-glance-api.service' and 'journalctl -xn' for details.
Wrapped exception:
Execution of '/usr/bin/systemctl start openstack-glance-api' returned 1: Job for openstack-glance-api.service failed. See 'systemctl status openstack-glance-api.service' and 'journalctl -xn' for details.
Error: /Stage[main]/Glance::Api/Service[glance-api]/ensure: change from stopped to running failed: Could not start Service[glance-api]: Execution of '/usr/bin/systemctl start openstack-glance-api' returned 1: Job for openstack-glance-api.service failed. See 'systemctl status openstack-glance-api.service' and 'journalctl -xn' for details.

The YAML file of the controller doesn't contain a reference to $known_stores in glance::api (see https://review.openstack.org/#/c/82699/).

Related to https://bugzilla.redhat.com/show_bug.cgi?id=1183946

Comment 5 Alex Stellwag 2015-02-17 15:38:47 UTC
I did the following to work around the issue:

I changed /etc/puppet/environments/production/modules/glance/manifests/api.pp to include the rbd store:

root@idospinstaller manifests]# diff -Nur api.pp.orig api.pp
--- api.pp.orig	2015-02-17 16:28:06.837029944 +0100
+++ api.pp	2015-02-17 16:28:27.555821296 +0100
@@ -197,7 +197,7 @@
   $cert_file                = false,
   $key_file                 = false,
   $ca_file                  = false,
-  $known_stores             = false,
+  $known_stores             = ['glance.store.rbd.Store'],
   $database_connection      = 'sqlite:///var/lib/glance/glance.sqlite',
   $database_idle_timeout    = 3600,
   $image_cache_dir          = '/var/lib/glance/image-cache',

This allowed me to work through the installation w/o further issues (so far ;)

Cheers, Alex

Comment 6 Rajini Karthik 2015-02-27 18:43:49 UTC
This is a bug in foreman-quickstart-puppet-module ( astapor). The openstack-puppet-module class 
glance::api is expecting the known_stores to be passed by foreman-quickstart-puppet-module.
Basically the real fix is to check the backend and set the $known_stores variable in the quickstart glance module.

File to be fixed(quickstart):
openstack-foreman-installer/puppet/modules/quickstack/manifests/glance.pp

Comment 7 Steve Hand 2015-02-27 20:07:09 UTC
*** Bug 1196921 has been marked as a duplicate of this bug. ***

Comment 8 Jason Guiditta 2015-03-03 19:38:41 UTC
(In reply to Alex Stellwag from comment #5)
> I did the following to work around the issue:
> 
> I changed
> /etc/puppet/environments/production/modules/glance/manifests/api.pp to
> include the rbd store:
> 
> root@idospinstaller manifests]# diff -Nur api.pp.orig api.pp
> --- api.pp.orig	2015-02-17 16:28:06.837029944 +0100
> +++ api.pp	2015-02-17 16:28:27.555821296 +0100
> @@ -197,7 +197,7 @@
>    $cert_file                = false,
>    $key_file                 = false,
>    $ca_file                  = false,
> -  $known_stores             = false,
> +  $known_stores             = ['glance.store.rbd.Store'],
>    $database_connection      = 'sqlite:///var/lib/glance/glance.sqlite',
>    $database_idle_timeout    = 3600,
>    $image_cache_dir          = '/var/lib/glance/image-cache',
> 
> This allowed me to work through the installation w/o further issues (so far
> ;)
> 
> Cheers, Alex

I am currently testing a patch that does exactly this, though it will substitute in the value of $backend parameter from the glance manifest that Rajini mentioned (still using the long name for the store though).  the only possible issue I see in this is that the short name file seems to have the long name with 'filesystem', so I may have to munge the value for that case.

Comment 9 Jason Guiditta 2015-03-03 20:44:28 UTC
Patch posted:
https://github.com/redhat-openstack/astapor/pull/483

Comment 10 Gaël Chamoulaud 2015-03-04 16:32:21 UTC
*** Bug 1183946 has been marked as a duplicate of this bug. ***

Comment 11 Jason Guiditta 2015-03-10 18:38:32 UTC
Merged by Crag

Comment 12 arkady kanevsky 2015-03-13 13:14:21 UTC
what is the workaround till A2?
What is the way to remove glance-api.conf from puppet control after initial install?

Comment 13 Jason Guiditta 2015-03-13 14:14:35 UTC
The only 2 workarounds I can think of are to add the puppet change to an existing deployment, or to not rerun puppet after configuration, manually setting the config file value as desired.

Comment 15 Yogev Rabl 2015-03-30 12:32:26 UTC
the Staypuft didn't installed Glance: Bug 1207158

Comment 16 Mike Burns 2015-03-30 18:53:55 UTC
A new bug is filed for that issue which is not a bug.  moving back to ON_QA

Comment 17 Yogev Rabl 2015-04-05 13:28:06 UTC
The parameter 'stores' is been set: 
stores=glance.store.rbd.Store,glance.store.http.Store

in staypuft version:
ruby193-rubygem-staypuft-0.5.22-1.el7ost.noarch

Comment 19 errata-xmlrpc 2015-04-07 15:08:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0791.html


Note You need to log in before you can comment on or make changes to this bug.