Bug 2074555

Summary: Old PyMySQL version errors out when enabling encryption
Product: [Community] RDO Reporter: Dmitriy Rabotjagov <noonedeadpunk>
Component: python-keystoneclientAssignee: Lance Bragstad <lbragsta>
Status: CLOSED UPSTREAM QA Contact: Jeremy Agee <jagee>
Severity: medium Docs Contact:
Priority: unspecified    
Version: trunkCC: dciabrin, nkinder
Target Milestone: ---   
Target Release: trunk   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-02-10 04:01:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dmitriy Rabotjagov 2022-04-12 13:34:14 UTC 
Description of problem:

When connection string is using in query `?ssl_verify_cert=true`, services error out exception.

This option has been implemented in PyMySQL 1.0.0 with following commit:
https://github.com/PyMySQL/PyMySQL/commit/aefbdbe1dc6dc022f2b02d2f4c4564d4ec929175

At the same time, according to upper-constraints, Xena should have used PyMySQL 1.0.2:
https://opendev.org/openstack/requirements/src/branch/stable/xena/upper-constraints.txt#L409

However RDO relies on CentOS provided package that is shipped as 0.10.1 for both CentOS 8 Stream and 9 Stream.

Which basically makes impossible to use encryption for MySQL connection.

Version-Release number of selected component (if applicable):
python3-PyMySQL 0.10.1

How reproducible:

Define following in keystone.conf:
[database]
connection = mysql+pymysql://glance:password.0.1/glance?charset=utf8&ssl_verify_cert=true

Actual results:
2022-04-12 12:47:56.775 1429 CRITICAL keystone [-] Unhandled error: TypeError: __init__() got an unexpected keyword argument 'ssl_verify_cert'
2022-04-12 12:47:56.775 1429 ERROR keystone Traceback (most recent call last):
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/bin/keystone-manage", line 10, in <module>
2022-04-12 12:47:56.775 1429 ERROR keystone     sys.exit(main())
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/keystone/cmd/manage.py", line 40, in main
2022-04-12 12:47:56.775 1429 ERROR keystone     cli.main(argv=sys.argv, developer_config_file=developer_config)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/keystone/cmd/cli.py", line 1376, in main
2022-04-12 12:47:56.775 1429 ERROR keystone     CONF.command.cmd_class.main()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/keystone/cmd/cli.py", line 333, in main
2022-04-12 12:47:56.775 1429 ERROR keystone     sys.exit(DbSync.check_db_sync_status())
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/keystone/cmd/cli.py", line 275, in check_db_sync_status
2022-04-12 12:47:56.775 1429 ERROR keystone     expand_version = upgrades.get_db_version(repo='expand_repo')
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/keystone/common/sql/upgrades.py", line 232, in get_db_version
2022-04-12 12:47:56.775 1429 ERROR keystone     with sql.session_for_read() as session:
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
2022-04-12 12:47:56.775 1429 ERROR keystone     return next(self.gen)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1059, in _transaction_scope
2022-04-12 12:47:56.775 1429 ERROR keystone     context=context) as resource:
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
2022-04-12 12:47:56.775 1429 ERROR keystone     return next(self.gen)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 654, in _session
2022-04-12 12:47:56.775 1429 ERROR keystone     bind=self.connection, mode=self.mode)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 414, in _create_session
2022-04-12 12:47:56.775 1429 ERROR keystone     self._start()
2022-04-12 12:47:56.775 1429 ERROR keystone     bind=self.connection, mode=self.mode)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 414, in _create_session
2022-04-12 12:47:56.775 1429 ERROR keystone     self._start()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 506, in _start
2022-04-12 12:47:56.775 1429 ERROR keystone     engine_args, maker_args)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 530, in _setup_for_connection
2022-04-12 12:47:56.775 1429 ERROR keystone     sql_connection=sql_connection, **engine_kwargs)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
2022-04-12 12:47:56.775 1429 ERROR keystone     return wrapped(*args, **kwargs)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line 211, in create_engine
2022-04-12 12:47:56.775 1429 ERROR keystone     test_conn = _test_connection(engine, max_retries, retry_interval)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line 386, in _test_connection
2022-04-12 12:47:56.775 1429 ERROR keystone     return engine.connect()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 3165, in connect
2022-04-12 12:47:56.775 1429 ERROR keystone     return self._connection_cls(self, close_with_result=close_with_result)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
2022-04-12 12:47:56.775 1429 ERROR keystone     else engine.raw_connection()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 3244, in raw_connection
2022-04-12 12:47:56.775 1429 ERROR keystone     return self._wrap_pool_connect(self.pool.connect, _connection)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 3211, in _wrap_pool_connect
2022-04-12 12:47:56.775 1429 ERROR keystone     return fn()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 307, in connect
2022-04-12 12:47:56.775 1429 ERROR keystone     return _ConnectionFairy._checkout(self)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
2022-04-12 12:47:56.775 1429 ERROR keystone     fairy = _ConnectionRecord.checkout(pool)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
2022-04-12 12:47:56.775 1429 ERROR keystone     rec = pool._do_get()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
2022-04-12 12:47:56.775 1429 ERROR keystone     self._dec_overflow()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
2022-04-12 12:47:56.775 1429 ERROR keystone     with_traceback=exc_tb,
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
2022-04-12 12:47:56.775 1429 ERROR keystone     raise exception
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
2022-04-12 12:47:56.775 1429 ERROR keystone     return self._create_connection()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
2022-04-12 12:47:56.775 1429 ERROR keystone     return _ConnectionRecord(self)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
2022-04-12 12:47:56.775 1429 ERROR keystone     self.__connect()
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
2022-04-12 12:47:56.775 1429 ERROR keystone     pool.logger.debug("Error on connect(): %s", e)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
2022-04-12 12:47:56.775 1429 ERROR keystone     with_traceback=exc_tb,
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
2022-04-12 12:47:56.775 1429 ERROR keystone     raise exception
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
2022-04-12 12:47:56.775 1429 ERROR keystone     connection = pool._invoke_creator(self)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/create.py", line 578, in connect
2022-04-12 12:47:56.775 1429 ERROR keystone     return dialect.connect(*cargs, **cparams)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 584, in connect
2022-04-12 12:47:56.775 1429 ERROR keystone     return self.dbapi.connect(*cargs, **cparams)
2022-04-12 12:47:56.775 1429 ERROR keystone   File "/usr/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect
2022-04-12 12:47:56.775 1429 ERROR keystone     return Connection(*args, **kwargs)
2022-04-12 12:47:56.775 1429 ERROR keystone TypeError: __init__() got an unexpected keyword argument 'ssl_verify_cert'
2022-04-12 12:47:56.775 1429 ERROR keystone 

Expected results:

Service running properly and encrypting connection to MySQL.
Comment 1 Damien Ciabrini 2022-04-12 13:50:23 UTC 
In TripleO, we do allow TLS connection to mysql, however we don't rely on ssl_verify_cert=true for doing so.

Instead we use ssl configurations flags read from a dedicated file:

connection=mysql+pymysql://keystone:password.localdomain/keystone?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo

In /etc/my.cnf.d/tripleo.cnf, we rely on a CA to provide cert verification:

[tripleo]
bind-address=<VIP>
ssl=1
ssl-ca=/etc/ipa/ca.crt
[client]
ssl=1
ssl-ca=/etc/ipa/ca.crt

Mysql uses its own certificate that has been signed with the CA above:

[mysqld]
...
ssl
ssl-cert = /etc/pki/tls/certs/mysql.crt
ssl-cipher = !SSLv2:kEECDH:kRSA:kEDH:kPSK:+3DES:!aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES:!SSLv3:!TLSv1
ssl-key = /etc/pki/tls/private/mysql.key
...
Comment 2 Red Hat Bugzilla 2025-02-10 04:01:10 UTC 
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.