Bug 1090648 - glance-manage db_sync silently fails to prepare the database
Summary: glance-manage db_sync silently fails to prepare the database
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RDO
Classification: Community
Component: openstack-utils
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: Icehouse
Assignee: Pádraig Brady
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-04-23 20:46 UTC by Federico Simoncelli
Modified: 2016-01-04 14:45 UTC (History)
6 users (show)

Fixed In Version: openstack-utils-2014.1-3
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1094675 (view as bug list)
Environment:
Last Closed: 2015-04-29 15:43:40 UTC


Attachments (Terms of Use)

Description Federico Simoncelli 2014-04-23 20:46:38 UTC
Description of problem:
glance-manage db_sync silently fails to prepare the database:

# glance-manage db_sync && echo ok
ok

# mysql glance -e "show tables"
+------------------+
| Tables_in_glance |
+------------------+
| migrate_version  |
+------------------+

Version-Release number of selected component (if applicable):
openstack-glance-2014.1-1.el6.noarch

How reproducible:
100%

Steps to Reproduce:
1. connection = mysql://glance:glance@localhost/glance
2. openstack-db --init --service glance

Actual results:
# openstack-db --init --service glance && echo ok
Verified connectivity to MySQL.
Creating 'glance' database.
Initializing the glance database, please wait...
Complete!
ok

# mysql glance -e "show tables"
+------------------+
| Tables_in_glance |
+------------------+
| migrate_version  |
+------------------+

Expected results:
The glance database should be ready to be used or an error should be clearly reported.

Additional info:
Adding some debugging it's possible to see that the actual error is:

# glance-manage db_sync && echo ok
Tables "migrate_version" have non utf8 collation, please make sure all tables are CHARSET=utf8
ok

If CHARSET=utf8 is another issue that should be fixed separately (and not just something related to my env) feel free to split the bug.

Comment 1 Brad P. Crochet 2014-05-05 14:59:58 UTC
As a workaround, one can set:

db_enforce_mysql_charset=False

in /etc/glance/glance.conf

This will get you past this error.

Comment 2 Brad P. Crochet 2014-05-05 15:00:21 UTC
Sorry... /etc/glance/glance-api.conf

Comment 3 Federico Simoncelli 2014-05-05 15:55:42 UTC
(In reply to Brad P. Crochet from comment #1)
> As a workaround, one can set:
> 
> db_enforce_mysql_charset=False
> 
> in /etc/glance/glance.conf
> 
> This will get you past this error.

The biggest problem is that the error is not printed anywhere (I had to manually add debugging to the code) so no one can tell if this is the issue they're having.

Comment 4 Flavio Percoco 2014-05-06 09:44:23 UTC
I agree a proper error is necessary.

Comment 5 Flavio Percoco 2014-05-06 12:03:59 UTC
I can't replicate this issue :/

Running `glance-manage` without setting db_enforce_mysql_charset=False` runs and fails with an error code when `ValueError` is raised.

Is there a way I can test this issue in the same environment where you hit it?

Comment 6 Federico Simoncelli 2014-05-06 15:30:52 UTC
To find the issue I had to modify DbCommands.sync() in glance/cmd/manage.py adding a try/except and a print:

        try:
            migration.db_sync(db_api.get_engine(),
                              db_migration.MIGRATE_REPO_PATH,
                              version,
                              sanity_check=self._need_sanity_check())
        except Exception as e:
            print(e)

These are some of the packages I am using:

python-sqlalchemy-0.5.5-3.el6_2.noarch
openstack-swift-1.13.1-1.el6.noarch
openstack-swift-plugin-swift3-1.7-1.el6.noarch
python-sqlalchemy0.7-0.7.8-1.el6ost.x86_64
openstack-keystone-2014.1-2.el6.noarch
openstack-glance-2014.1-1.el6.noarch
openstack-utils-2013.2-3.el6.noarch

Comment 7 Flavio Percoco 2014-05-07 12:47:57 UTC
Ahhh, that clear things up.

Do you have the glance-api logs? I *think* the whole exception (pre- try/except) may have been logged to the glance log file. `glance-manage` reads /etc/glance/glance-api.conf and it will use the logging configurations found there.

If you have the log files, I'll check everything. Otherwise, I'll try to replicate it.

Comment 8 Federico Simoncelli 2014-05-07 21:20:39 UTC
/var/log/glance/api.log:

2014-05-07 17:18:57.679 16348 CRITICAL glance [-] ValueError: Tables "migrate_version" have non utf8 collation, please make sure all tables are CHARSET=utf8
2014-05-07 17:18:57.679 16348 TRACE glance Traceback (most recent call last):
2014-05-07 17:18:57.679 16348 TRACE glance   File "/usr/bin/glance-manage", line 10, in <module>
2014-05-07 17:18:57.679 16348 TRACE glance     sys.exit(main())
2014-05-07 17:18:57.679 16348 TRACE glance   File "/usr/lib/python2.6/site-packages/glance/cmd/manage.py", line 259, in main
2014-05-07 17:18:57.679 16348 TRACE glance     return CONF.command.action_fn()
2014-05-07 17:18:57.679 16348 TRACE glance   File "/usr/lib/python2.6/site-packages/glance/cmd/manage.py", line 160, in sync
2014-05-07 17:18:57.679 16348 TRACE glance     CONF.command.current_version)
2014-05-07 17:18:57.679 16348 TRACE glance   File "/usr/lib/python2.6/site-packages/glance/cmd/manage.py", line 137, in sync
2014-05-07 17:18:57.679 16348 TRACE glance     sanity_check=self._need_sanity_check())
2014-05-07 17:18:57.679 16348 TRACE glance   File "/usr/lib/python2.6/site-packages/glance/openstack/common/db/sqlalchemy/migration.py", line 195, in db_sync
2014-05-07 17:18:57.679 16348 TRACE glance     _db_schema_sanity_check(engine)
2014-05-07 17:18:57.679 16348 TRACE glance   File "/usr/lib/python2.6/site-packages/glance/openstack/common/db/sqlalchemy/migration.py", line 221, in _db_schema_sanity_check
2014-05-07 17:18:57.679 16348 TRACE glance     ) % ','.join(table_names))
2014-05-07 17:18:57.679 16348 TRACE glance ValueError: Tables "migrate_version" have non utf8 collation, please make sure all tables are CHARSET=utf8

As a side note: installing keystone on the same mysql server wasn't a problem.

Comment 9 Flavio Percoco 2014-05-08 07:56:41 UTC
The UTF8 issue is known. That is, Glance was the only service still using non-UTF8 tables and those need to be migrated. With that said, I think this bug could be fixed by making `openstack-db` use other logging values on the CLI. I'll move this bug under `openstack-utils` and close Glance's upstream bug since there's no bug here from a Glance stand point.

Thanks Fede.

Comment 10 Pádraig Brady 2014-05-20 11:15:30 UTC
As per comment #5, glance-manage prints out the ValueError?
If not it should. There have been long term issues with the
early startup issues in glance not being logged.

openstack-db doesn't suppress any error messages from glance-manage,
and glance-manage should log appropriately given the current
logging setting of the system given not only openstack-db may
be calling it.

Comment 11 Flavio Percoco 2014-05-20 11:37:38 UTC
It prints the ValueError but it logs it in the log file. This happens because glance-manage is reading the global config which defaults to logging to the log file.

There are 2 ways to fix this that I can think OTOH:

1. Changing Glance's default (use_stderr = True)
2. Make openstack-db explicitly specify std(out|err) as the preferred logging output, hence the move to openstack-utils.

I'd prefer changing this in openstack-utils rather than making `use_stderr=True` the default value. This also applies to other projects.

openstack-db could do something like:

    $ glance-manage -d -v --log-file= db_sync

The above unsets the log file and forces glance-manage to use stdout.

Comment 12 Pádraig Brady 2014-05-20 15:03:17 UTC
Ah sorry I thought Federico needed to patch glance to get the log, when in fact it was logged to the standard location.

So as for changing glance-manage to log to stderr rather than to logs,
I'm a bit reluctant, as there are often superfluous things logged to openstack logs, and having everything logged to a single location can have its advantages too.

I suppose the best thing would be to bolster the sanity check done in openstack-db, or otherwise honor/diagnose the exit status of the glance-manage command.
I presume it does exit with failure code as you said in comment 5.
Hopefully this patch addresses things appropriately:
https://github.com/redhat-openstack/openstack-utils/commit/db3ca160


Note You need to log in before you can comment on or make changes to this bug.