Bug 1690225

Summary:	[RHOSP13][bmaas] ironic won't deploy with TLS Everywhere
Product:	Red Hat OpenStack	Reporter:	Robin Cernin <rcernin>
Component:	openstack-tripleo-heat-templates	Assignee:	Dmitry Tantsur <dtantsur>
Status:	CLOSED ERRATA	QA Contact:	Alexander Chuzhoy <sasha>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	13.0 (Queens)	CC:	asimonel, bfournie, chris.smart, dbecker, dtantsur, ealcaniz, fiezzi, josorior, mburns, morazi, nsatsia, pkomarov, ramishra, rpittau, rtweed, sasha, sputhenp
Target Milestone:	async	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-8.2.0-14.el7ost puppet-tripleo-8.3.6-17.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1693898 (view as bug list)		Environment:
Last Closed:	2019-04-30 17:27:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1693898

Description Robin Cernin 2019-03-19 05:03:34 UTC

During deployment of RHOSP13 with BMaaS, ironic-dbsync fails with Access denied to Mysql DB:

# cat ironic-dbsync.log
2019-03-18 19:28:38.853 13 CRITICAL ironic [-] Unhandled error: OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'ironic'@'192.168.204.16' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)
2019-03-18 19:28:38.853 13 ERROR ironic Traceback (most recent call last):
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/bin/ironic-dbsync", line 10, in <module>
2019-03-18 19:28:38.853 13 ERROR ironic     sys.exit(main())
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/ironic/cmd/dbsync.py", line 326, in main
2019-03-18 19:28:38.853 13 ERROR ironic     CONF.command.func()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/ironic/cmd/dbsync.py", line 104, in upgrade
2019-03-18 19:28:38.853 13 ERROR ironic     self._check_versions()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/ironic/cmd/dbsync.py", line 89, in _check_versions
2019-03-18 19:28:38.853 13 ERROR ironic     if migration.version() is None:
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/ironic/db/migration.py", line 40, in version
2019-03-18 19:28:38.853 13 ERROR ironic     return get_backend().version()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/migration.py", line 41, in version
2019-03-18 19:28:38.853 13 ERROR ironic     engine = enginefacade.writer.get_engine()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 809, in get_engine
2019-03-18 19:28:38.853 13 ERROR ironic     return self._factory.get_writer_engine()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 358, in get_writer_engine
2019-03-18 19:28:38.853 13 ERROR ironic     self._start()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 491, in _start
2019-03-18 19:28:38.853 13 ERROR ironic     engine_args, maker_args)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 515, in _setup_for_connection
2019-03-18 19:28:38.853 13 ERROR ironic     sql_connection=sql_connection, **engine_kwargs)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
2019-03-18 19:28:38.853 13 ERROR ironic     return wrapped(*args, **kwargs)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine
2019-03-18 19:28:38.853 13 ERROR ironic     test_conn = _test_connection(engine, max_retries, retry_interval)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 362, in _test_connection
2019-03-18 19:28:38.853 13 ERROR ironic     return engine.connect()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2102, in connect
2019-03-18 19:28:38.853 13 ERROR ironic     return self._connection_cls(self, **kwargs)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
2019-03-18 19:28:38.853 13 ERROR ironic     if connection is not None else engine.raw_connection()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection
2019-03-18 19:28:38.853 13 ERROR ironic     self.pool.unique_connection, _connection)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2019-03-18 19:28:38.853 13 ERROR ironic     e, dialect, self)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2019-03-18 19:28:38.853 13 ERROR ironic     util.raise_from_cause(newraise, exc_info)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-03-18 19:28:38.853 13 ERROR ironic     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2019-03-18 19:28:38.853 13 ERROR ironic     return fn()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection
2019-03-18 19:28:38.853 13 ERROR ironic     return _ConnectionFairy._checkout(self)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout
2019-03-18 19:28:38.853 13 ERROR ironic     fairy = _ConnectionRecord.checkout(pool)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
2019-03-18 19:28:38.853 13 ERROR ironic     rec = pool._do_get()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get
2019-03-18 19:28:38.853 13 ERROR ironic     self._dec_overflow()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2019-03-18 19:28:38.853 13 ERROR ironic     compat.reraise(exc_type, exc_value, exc_tb)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get
2019-03-18 19:28:38.853 13 ERROR ironic     return self._create_connection()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
2019-03-18 19:28:38.853 13 ERROR ironic     return _ConnectionRecord(self)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
2019-03-18 19:28:38.853 13 ERROR ironic     self.__connect(first_connect_check=True)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect
2019-03-18 19:28:38.853 13 ERROR ironic     connection = pool._invoke_creator(self)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2019-03-18 19:28:38.853 13 ERROR ironic     return dialect.connect(*cargs, **cparams)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2019-03-18 19:28:38.853 13 ERROR ironic     return self.dbapi.connect(*cargs, **cparams)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2019-03-18 19:28:38.853 13 ERROR ironic     return Connection(*args, **kwargs)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__
2019-03-18 19:28:38.853 13 ERROR ironic     self.connect()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 932, in connect
2019-03-18 19:28:38.853 13 ERROR ironic     self._request_authentication()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1152, in _request_authentication
2019-03-18 19:28:38.853 13 ERROR ironic     auth_packet = self._read_packet()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2019-03-18 19:28:38.853 13 ERROR ironic     packet.check_error()
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2019-03-18 19:28:38.853 13 ERROR ironic     err.raise_mysql_exception(self._data)
2019-03-18 19:28:38.853 13 ERROR ironic   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2019-03-18 19:28:38.853 13 ERROR ironic     raise errorclass(errno, errval)
2019-03-18 19:28:38.853 13 ERROR ironic OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'ironic'@'192.168.204.16' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)
2019-03-18 19:28:38.853 13 ERROR ironic



Ansible log:


            "INFO:nova_statedir:Checking uid: 162 gid: 162 path: /var/lib/nova/.ssh/config",
            "INFO:nova_statedir:Checking uid: 162 gid: 162 path: /var/lib/nova/instances/",
            "INFO:nova_statedir:Changing ownership of /var/lib/nova/instances from 162:162 to 42436:42436",
            "INFO:nova_statedir:Checking uid: 162 gid: 162 path: /var/lib/nova/networks/",
            "INFO:nova_statedir:Changing ownership of /var/lib/nova/networks from 162:162 to 42436:42436",
            "INFO:nova_statedir:Nova statedir ownership complete",
            "stdout: 2019-03-18 13:10:50.425 12 INFO barbican.model.sync [-] Syncing the secret_stores table with barbican.conf",
            "Error running ['docker', 'run', '--name', 'ironic_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=ironic_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 1, \"image\": \"192.168.1.1:8787/rhosp13/openstack-ironic-api:13.0-67\", \"command\": \"/usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c \\'ironic-dbsync --config-file /etc/ironic/ironic.conf\\'\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/ironic_api/etc/ironic:/etc/ironic:ro\", \"/var/log/containers/ironic:/var/log/ironic\", \"/var/log/containers/httpd/ironic-api:/var/log/httpd\"], \"net\": \"host\", \"detach\": false, \"privileged\": false}', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/ironic_api/etc/ironic:/etc/ironic:ro', '--volume=/var/log/containers/ironic:/var/log/ironic', '--volume=/var/log/containers/httpd/ironic-api:/var/log/httpd', '192.168.1.1:8787/rhosp13/openstack-ironic-api:13.0-67', '/usr/bin/bootstrap_host_exec', 'ironic_api', 'su', 'ironic', '-s', '/bin/bash', '-c', \"'ironic-dbsync\", '--config-file', \"/etc/ironic/ironic.conf'\"]. [1]",
            "stdout: (cellv2) Creating default cell_v2 cell",
            "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported",
            "stdout: fbb11a410e8276558bb098bd8643f73e5dcebe2ff6136624e26e9b0a6e11b60f",
            "stdout: 8832a9978bbd01579952dd05cc4b1810c1855c6d4ce03f0a0900de36a9aaef72",
            "/usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (1831, u'Duplicate index `block_device_mapping_instance_uuid_virtual_name_device_name_idx`. This is deprecated and will be disallowed in a future release.')",
            "/usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (1831, u'Duplicate index `uniq_instances0uuid`. This is deprecated and will be disallowed in a future release.')",
            "stdout: 14137b3137cca16938271971a79d5842cd4ca94e94a6b355ad8bf27fb2a8a033",
            "stdout: 5dcc6dac3e982e25c8031e13f9d3a936ca36bbe98a39015385a719644fb6af19",
            "stdout: 1d0fcf726225ed7c7fd4e208eb557a594b82161e9f2e8c21e856a92f861e3131",
            "stdout: 1c68e68d2e69a633f2048a61c49e5f6ff6a7b38323f49847a1c0d1bc9ca92526"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/ec83bfc6-4016-438e-8d56-f4338aecbb14_playbook.retry

    PLAY RECAP *********************************************************************
    localhost                  : ok=5    changed=2    unreachable=0    failed=1

  deploy_stderr: |

Comment 6 Dmitry Tantsur 2019-03-19 08:50:38 UTC

Hi Robin,

Could you check if https://review.openstack.org/#/c/644512/ fixes this problem?

Also note that internal TLS with Ironic may need special treatment of IPA, since there is no way to pass certificates there.

Comment 17 Juan Antonio Osorio 2019-03-20 13:23:27 UTC

Dmitry; the tasks you put up look about right.

Comment 31 Alexander Chuzhoy 2019-03-25 20:22:17 UTC

Reproduced the issue with openstack-tripleo-heat-templates-8.2.0-6.2.el7ost.noarch
Waiting for the fix to land.

Comment 38 Alexander Chuzhoy 2019-04-12 02:41:23 UTC

Verified:

Environment:
openstack-tripleo-heat-templates-8.3.1-5.el7ost.noarch
puppet-tripleo-8.4.1-2.el7ost.noarch


The reported issue doesn't reproduce - successfully deployed OC with ironic BMaaS and TLS everywhere.

Comment 41 errata-xmlrpc 2019-04-30 17:27:39 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0939