Bug 1290236 - Overcloud deployment fails on step5 with ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory
Overcloud deployment fails on step5 with ImportError: libmysqlclient.so.18: c...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
urgent Severity high
: ---
: 8.0 (Liberty)
Assigned To: chris alfonso
yeylon@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-09 17:15 EST by Steve Linabery
Modified: 2016-04-18 02:54 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-09 17:41:46 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Steve Linabery 2015-12-09 17:15:32 EST
Description of problem:
When deploying a 'vanilla' (i.e. 1 controller, 1 compute, no custom configuration files, flat networking) overcloud, step5 of deployment fails. neutron-server and nova-scheduler fail to start.

In /var/log/nova on controller:
[root@overcloud-controller-0 nova]# grep ERR *
nova-manage.log:2015-12-09 16:17:29.106 15422 DEBUG oslo_db.sqlalchemy.session [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py:513
[root@overcloud-controller-0 nova]# tail -500 nova-scheduler.log
2015-12-09 16:19:25.343 23204 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/site-packages/nova/servicegroup/api.py:68
2015-12-09 16:19:25.435 23204 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-12-09 16:19:25.527 23204 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/site-packages/nova/servicegroup/api.py:68
2015-12-09 16:19:25.535 23204 DEBUG oslo_db.api [req-ae60367d-6664-4446-a78b-e8170cb9d090 - - - - -] Loading backend 'sqlalchemy' from 'nova.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:214
2015-12-09 16:19:25.537 23204 CRITICAL nova [req-ae60367d-6664-4446-a78b-e8170cb9d090 - - - - -] ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory
2015-12-09 16:19:25.537 23204 TRACE nova Traceback (most recent call last):
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/bin/nova-scheduler", line 10, in <module>
2015-12-09 16:19:25.537 23204 TRACE nova     sys.exit(main())
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/cmd/scheduler.py", line 44, in main
2015-12-09 16:19:25.537 23204 TRACE nova     topic=CONF.scheduler_topic)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/service.py", line 277, in create
2015-12-09 16:19:25.537 23204 TRACE nova     db_allowed=db_allowed)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/service.py", line 148, in __init__
2015-12-09 16:19:25.537 23204 TRACE nova     self.manager = manager_class(host=self.host, *args, **kwargs)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 63, in __init__
2015-12-09 16:19:25.537 23204 TRACE nova     self.driver = importutils.import_object(scheduler_driver)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 38, in import_object
2015-12-09 16:19:25.537 23204 TRACE nova     return import_class(import_str)(*args, **kwargs)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 56, in __init__
2015-12-09 16:19:25.537 23204 TRACE nova     super(FilterScheduler, self).__init__(*args, **kwargs)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/scheduler/driver.py", line 44, in __init__
2015-12-09 16:19:25.537 23204 TRACE nova     CONF.scheduler_host_manager)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 38, in import_object
2015-12-09 16:19:25.537 23204 TRACE nova     return import_class(import_str)(*args, **kwargs)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 328, in __init__
2015-12-09 16:19:25.537 23204 TRACE nova     self._init_aggregates()
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 340, in _init_aggregates
2015-12-09 16:19:25.537 23204 TRACE nova     aggs = objects.AggregateList.get_all(elevated)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
2015-12-09 16:19:25.537 23204 TRACE nova     result = fn(cls, context, *args, **kwargs)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/objects/aggregate.py", line 184, in get_all
2015-12-09 16:19:25.537 23204 TRACE nova     db_aggregates = db.aggregate_get_all(context)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1779, in aggregate_get_all
2015-12-09 16:19:25.537 23204 TRACE nova     return IMPL.aggregate_get_all(context)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5521, in aggregate_get_all
2015-12-09 16:19:25.537 23204 TRACE nova     return _aggregate_get_query(context, models.Aggregate).all()
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5346, in _aggregate_get_query
2015-12-09 16:19:25.537 23204 TRACE nova     read_deleted=read_deleted)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 310, in model_query
2015-12-09 16:19:25.537 23204 TRACE nova     session = get_session(use_slave=use_slave)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 185, in get_session
2015-12-09 16:19:25.537 23204 TRACE nova     facade = _create_facade_lazily(_MAIN_FACADE, conf_group)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 167, in _create_facade_lazily
2015-12-09 16:19:25.537 23204 TRACE nova     _ENGINE_FACADE[facade] = _create_facade(conf_group)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 159, in _create_facade
2015-12-09 16:19:25.537 23204 TRACE nova     retry_interval=conf_group.retry_interval)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py", line 796, in __init__
2015-12-09 16:19:25.537 23204 TRACE nova     **engine_kwargs)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py", line 395, in create_engine
2015-12-09 16:19:25.537 23204 TRACE nova     engine = sqlalchemy.create_engine(url, **engine_args)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 386, in create_engine
2015-12-09 16:19:25.537 23204 TRACE nova     return strategy.create(*args, **kwargs)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 75, in create
2015-12-09 16:19:25.537 23204 TRACE nova     dbapi = dialect_cls.dbapi(**dbapi_args)
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 92, in dbapi
2015-12-09 16:19:25.537 23204 TRACE nova     return __import__('MySQLdb')
2015-12-09 16:19:25.537 23204 TRACE nova   File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 19, in <module>
2015-12-09 16:19:25.537 23204 TRACE nova     import _mysql
2015-12-09 16:19:25.537 23204 TRACE nova ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory
2015-12-09 16:19:25.537 23204 TRACE nova
[root@overcloud-controller-0 nova]#
[root@overcloud-controller-0 nova]# grep CRIT *
nova-scheduler.log:2015-12-09 16:19:25.537 23204 CRITICAL nova [req-ae60367d-6664-4446-a78b-e8170cb9d090 - - - - -] ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory
Comment 2 Steve Linabery 2015-12-09 17:19:44 EST
Some additional info. We used the stock overcloud image and used libguestfs to add the device-mapper-multipath to the image.
Comment 3 Dan Sneddon 2015-12-09 17:21:33 EST
In this particular case, after the error occurred, the controller was investigated. The file: /usr/lib64/mysql/libmysqlclient.so.18 did not exist, however the file /usr/lib64/mysql/libmysqlclient.so.18.0.0 did.

This was a custom image, made by taking the original image and making the following steps:

  enable root password
  multipath RPMs:
    lvm2
    kpartx
    device-multipath

openstack puppet modules were upgraded on the undercloud to most recent build attached to an unshipped advisory, then downgraded to the original OPM from the official image.
Comment 4 Dan Sneddon 2015-12-09 17:25:13 EST
(In reply to Dan Sneddon from comment #3)
> In this particular case, after the error occurred, the controller was
> investigated. The file: /usr/lib64/mysql/libmysqlclient.so.18 did not exist,
> however the file /usr/lib64/mysql/libmysqlclient.so.18.0.0 did.

Correction. It is not known whether the symlink from libmysqlclient.so.18 to libmysqlclient.so.18.0.0 was there or not. Only matching files were searched for, not symlinks.
Comment 5 Steve Linabery 2015-12-09 17:41:15 EST
Reverting to stock overcloud image resulted in CREATE_COMPLETE.

Note You need to log in before you can comment on or make changes to this bug.