Description of problem: When connection string is using in query `?ssl_verify_cert=true`, services error out exception. This option has been implemented in PyMySQL 1.0.0 with following commit: https://github.com/PyMySQL/PyMySQL/commit/aefbdbe1dc6dc022f2b02d2f4c4564d4ec929175 At the same time, according to upper-constraints, Xena should have used PyMySQL 1.0.2: https://opendev.org/openstack/requirements/src/branch/stable/xena/upper-constraints.txt#L409 However RDO relies on CentOS provided package that is shipped as 0.10.1 for both CentOS 8 Stream and 9 Stream. Which basically makes impossible to use encryption for MySQL connection. Version-Release number of selected component (if applicable): python3-PyMySQL 0.10.1 How reproducible: Define following in keystone.conf: [database] connection = mysql+pymysql://glance:password.0.1/glance?charset=utf8&ssl_verify_cert=true Actual results: 2022-04-12 12:47:56.775 1429 CRITICAL keystone [-] Unhandled error: TypeError: __init__() got an unexpected keyword argument 'ssl_verify_cert' 2022-04-12 12:47:56.775 1429 ERROR keystone Traceback (most recent call last): 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/bin/keystone-manage", line 10, in <module> 2022-04-12 12:47:56.775 1429 ERROR keystone sys.exit(main()) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/keystone/cmd/manage.py", line 40, in main 2022-04-12 12:47:56.775 1429 ERROR keystone cli.main(argv=sys.argv, developer_config_file=developer_config) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/keystone/cmd/cli.py", line 1376, in main 2022-04-12 12:47:56.775 1429 ERROR keystone CONF.command.cmd_class.main() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/keystone/cmd/cli.py", line 333, in main 2022-04-12 12:47:56.775 1429 ERROR keystone sys.exit(DbSync.check_db_sync_status()) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/keystone/cmd/cli.py", line 275, in check_db_sync_status 2022-04-12 12:47:56.775 1429 ERROR keystone expand_version = upgrades.get_db_version(repo='expand_repo') 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/keystone/common/sql/upgrades.py", line 232, in get_db_version 2022-04-12 12:47:56.775 1429 ERROR keystone with sql.session_for_read() as session: 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2022-04-12 12:47:56.775 1429 ERROR keystone return next(self.gen) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1059, in _transaction_scope 2022-04-12 12:47:56.775 1429 ERROR keystone context=context) as resource: 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ 2022-04-12 12:47:56.775 1429 ERROR keystone return next(self.gen) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 654, in _session 2022-04-12 12:47:56.775 1429 ERROR keystone bind=self.connection, mode=self.mode) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 414, in _create_session 2022-04-12 12:47:56.775 1429 ERROR keystone self._start() 2022-04-12 12:47:56.775 1429 ERROR keystone bind=self.connection, mode=self.mode) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 414, in _create_session 2022-04-12 12:47:56.775 1429 ERROR keystone self._start() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 506, in _start 2022-04-12 12:47:56.775 1429 ERROR keystone engine_args, maker_args) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 530, in _setup_for_connection 2022-04-12 12:47:56.775 1429 ERROR keystone sql_connection=sql_connection, **engine_kwargs) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator 2022-04-12 12:47:56.775 1429 ERROR keystone return wrapped(*args, **kwargs) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line 211, in create_engine 2022-04-12 12:47:56.775 1429 ERROR keystone test_conn = _test_connection(engine, max_retries, retry_interval) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line 386, in _test_connection 2022-04-12 12:47:56.775 1429 ERROR keystone return engine.connect() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 3165, in connect 2022-04-12 12:47:56.775 1429 ERROR keystone return self._connection_cls(self, close_with_result=close_with_result) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 96, in __init__ 2022-04-12 12:47:56.775 1429 ERROR keystone else engine.raw_connection() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 3244, in raw_connection 2022-04-12 12:47:56.775 1429 ERROR keystone return self._wrap_pool_connect(self.pool.connect, _connection) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 3211, in _wrap_pool_connect 2022-04-12 12:47:56.775 1429 ERROR keystone return fn() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 307, in connect 2022-04-12 12:47:56.775 1429 ERROR keystone return _ConnectionFairy._checkout(self) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout 2022-04-12 12:47:56.775 1429 ERROR keystone fairy = _ConnectionRecord.checkout(pool) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 425, in checkout 2022-04-12 12:47:56.775 1429 ERROR keystone rec = pool._do_get() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get 2022-04-12 12:47:56.775 1429 ERROR keystone self._dec_overflow() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__ 2022-04-12 12:47:56.775 1429 ERROR keystone with_traceback=exc_tb, 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2022-04-12 12:47:56.775 1429 ERROR keystone raise exception 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get 2022-04-12 12:47:56.775 1429 ERROR keystone return self._create_connection() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection 2022-04-12 12:47:56.775 1429 ERROR keystone return _ConnectionRecord(self) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 368, in __init__ 2022-04-12 12:47:56.775 1429 ERROR keystone self.__connect() 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 611, in __connect 2022-04-12 12:47:56.775 1429 ERROR keystone pool.logger.debug("Error on connect(): %s", e) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__ 2022-04-12 12:47:56.775 1429 ERROR keystone with_traceback=exc_tb, 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_ 2022-04-12 12:47:56.775 1429 ERROR keystone raise exception 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 605, in __connect 2022-04-12 12:47:56.775 1429 ERROR keystone connection = pool._invoke_creator(self) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/create.py", line 578, in connect 2022-04-12 12:47:56.775 1429 ERROR keystone return dialect.connect(*cargs, **cparams) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 584, in connect 2022-04-12 12:47:56.775 1429 ERROR keystone return self.dbapi.connect(*cargs, **cparams) 2022-04-12 12:47:56.775 1429 ERROR keystone File "/usr/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect 2022-04-12 12:47:56.775 1429 ERROR keystone return Connection(*args, **kwargs) 2022-04-12 12:47:56.775 1429 ERROR keystone TypeError: __init__() got an unexpected keyword argument 'ssl_verify_cert' 2022-04-12 12:47:56.775 1429 ERROR keystone Expected results: Service running properly and encrypting connection to MySQL.
In TripleO, we do allow TLS connection to mysql, however we don't rely on ssl_verify_cert=true for doing so. Instead we use ssl configurations flags read from a dedicated file: connection=mysql+pymysql://keystone:password.localdomain/keystone?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo In /etc/my.cnf.d/tripleo.cnf, we rely on a CA to provide cert verification: [tripleo] bind-address=<VIP> ssl=1 ssl-ca=/etc/ipa/ca.crt [client] ssl=1 ssl-ca=/etc/ipa/ca.crt Mysql uses its own certificate that has been signed with the CA above: [mysqld] ... ssl ssl-cert = /etc/pki/tls/certs/mysql.crt ssl-cipher = !SSLv2:kEECDH:kRSA:kEDH:kPSK:+3DES:!aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES:!SSLv3:!TLSv1 ssl-key = /etc/pki/tls/private/mysql.key ...