Bug 1290236
| Summary: | Overcloud deployment fails on step5 with ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Steve Linabery <slinaber> |
| Component: | rhosp-director | Assignee: | chris alfonso <calfonso> |
| Status: | CLOSED NOTABUG | QA Contact: | yeylon <yeylon> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 7.0 (Kilo) | CC: | dsneddon, hbrock, mburns, rhel-osp-director-maint, srevivo |
| Target Milestone: | --- | ||
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-12-09 22:41:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Some additional info. We used the stock overcloud image and used libguestfs to add the device-mapper-multipath to the image. In this particular case, after the error occurred, the controller was investigated. The file: /usr/lib64/mysql/libmysqlclient.so.18 did not exist, however the file /usr/lib64/mysql/libmysqlclient.so.18.0.0 did.
This was a custom image, made by taking the original image and making the following steps:
enable root password
multipath RPMs:
lvm2
kpartx
device-multipath
openstack puppet modules were upgraded on the undercloud to most recent build attached to an unshipped advisory, then downgraded to the original OPM from the official image.
(In reply to Dan Sneddon from comment #3) > In this particular case, after the error occurred, the controller was > investigated. The file: /usr/lib64/mysql/libmysqlclient.so.18 did not exist, > however the file /usr/lib64/mysql/libmysqlclient.so.18.0.0 did. Correction. It is not known whether the symlink from libmysqlclient.so.18 to libmysqlclient.so.18.0.0 was there or not. Only matching files were searched for, not symlinks. Reverting to stock overcloud image resulted in CREATE_COMPLETE. |
Description of problem: When deploying a 'vanilla' (i.e. 1 controller, 1 compute, no custom configuration files, flat networking) overcloud, step5 of deployment fails. neutron-server and nova-scheduler fail to start. In /var/log/nova on controller: [root@overcloud-controller-0 nova]# grep ERR * nova-manage.log:2015-12-09 16:17:29.106 15422 DEBUG oslo_db.sqlalchemy.session [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py:513 [root@overcloud-controller-0 nova]# tail -500 nova-scheduler.log 2015-12-09 16:19:25.343 23204 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/site-packages/nova/servicegroup/api.py:68 2015-12-09 16:19:25.435 23204 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative 2015-12-09 16:19:25.527 23204 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __init__ /usr/lib/python2.7/site-packages/nova/servicegroup/api.py:68 2015-12-09 16:19:25.535 23204 DEBUG oslo_db.api [req-ae60367d-6664-4446-a78b-e8170cb9d090 - - - - -] Loading backend 'sqlalchemy' from 'nova.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:214 2015-12-09 16:19:25.537 23204 CRITICAL nova [req-ae60367d-6664-4446-a78b-e8170cb9d090 - - - - -] ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory 2015-12-09 16:19:25.537 23204 TRACE nova Traceback (most recent call last): 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/bin/nova-scheduler", line 10, in <module> 2015-12-09 16:19:25.537 23204 TRACE nova sys.exit(main()) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/cmd/scheduler.py", line 44, in main 2015-12-09 16:19:25.537 23204 TRACE nova topic=CONF.scheduler_topic) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 277, in create 2015-12-09 16:19:25.537 23204 TRACE nova db_allowed=db_allowed) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 148, in __init__ 2015-12-09 16:19:25.537 23204 TRACE nova self.manager = manager_class(host=self.host, *args, **kwargs) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 63, in __init__ 2015-12-09 16:19:25.537 23204 TRACE nova self.driver = importutils.import_object(scheduler_driver) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 38, in import_object 2015-12-09 16:19:25.537 23204 TRACE nova return import_class(import_str)(*args, **kwargs) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 56, in __init__ 2015-12-09 16:19:25.537 23204 TRACE nova super(FilterScheduler, self).__init__(*args, **kwargs) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/scheduler/driver.py", line 44, in __init__ 2015-12-09 16:19:25.537 23204 TRACE nova CONF.scheduler_host_manager) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 38, in import_object 2015-12-09 16:19:25.537 23204 TRACE nova return import_class(import_str)(*args, **kwargs) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 328, in __init__ 2015-12-09 16:19:25.537 23204 TRACE nova self._init_aggregates() 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 340, in _init_aggregates 2015-12-09 16:19:25.537 23204 TRACE nova aggs = objects.AggregateList.get_all(elevated) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper 2015-12-09 16:19:25.537 23204 TRACE nova result = fn(cls, context, *args, **kwargs) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/objects/aggregate.py", line 184, in get_all 2015-12-09 16:19:25.537 23204 TRACE nova db_aggregates = db.aggregate_get_all(context) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1779, in aggregate_get_all 2015-12-09 16:19:25.537 23204 TRACE nova return IMPL.aggregate_get_all(context) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5521, in aggregate_get_all 2015-12-09 16:19:25.537 23204 TRACE nova return _aggregate_get_query(context, models.Aggregate).all() 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5346, in _aggregate_get_query 2015-12-09 16:19:25.537 23204 TRACE nova read_deleted=read_deleted) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 310, in model_query 2015-12-09 16:19:25.537 23204 TRACE nova session = get_session(use_slave=use_slave) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 185, in get_session 2015-12-09 16:19:25.537 23204 TRACE nova facade = _create_facade_lazily(_MAIN_FACADE, conf_group) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 167, in _create_facade_lazily 2015-12-09 16:19:25.537 23204 TRACE nova _ENGINE_FACADE[facade] = _create_facade(conf_group) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 159, in _create_facade 2015-12-09 16:19:25.537 23204 TRACE nova retry_interval=conf_group.retry_interval) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py", line 796, in __init__ 2015-12-09 16:19:25.537 23204 TRACE nova **engine_kwargs) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/session.py", line 395, in create_engine 2015-12-09 16:19:25.537 23204 TRACE nova engine = sqlalchemy.create_engine(url, **engine_args) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 386, in create_engine 2015-12-09 16:19:25.537 23204 TRACE nova return strategy.create(*args, **kwargs) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 75, in create 2015-12-09 16:19:25.537 23204 TRACE nova dbapi = dialect_cls.dbapi(**dbapi_args) 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 92, in dbapi 2015-12-09 16:19:25.537 23204 TRACE nova return __import__('MySQLdb') 2015-12-09 16:19:25.537 23204 TRACE nova File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 19, in <module> 2015-12-09 16:19:25.537 23204 TRACE nova import _mysql 2015-12-09 16:19:25.537 23204 TRACE nova ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory 2015-12-09 16:19:25.537 23204 TRACE nova [root@overcloud-controller-0 nova]# [root@overcloud-controller-0 nova]# grep CRIT * nova-scheduler.log:2015-12-09 16:19:25.537 23204 CRITICAL nova [req-ae60367d-6664-4446-a78b-e8170cb9d090 - - - - -] ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory