Bug 1723306

Summary: [UPDATE] controller update fails with error: " "CantStartEngineError: No sql_connection parameter is established"
Product: Red Hat OpenStack Reporter: Raviv Bar-Tal <rbartal>
Component: openstack-tripleo-heat-templatesAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED ERRATA QA Contact: Raviv Bar-Tal <rbartal>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: dbecker, ekuris, lbezdick, mbracho, mburns, morazi, slinaber, tvignaud, yobshans
Target Milestone: z7Keywords: Regression, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.3.1-54.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-10 13:05:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1627497, 1719673    
Attachments:
Description Flags
controller update log none

Description Raviv Bar-Tal 2019-06-24 08:55:52 UTC
Created attachment 1583942 [details]
controller update log

Description of problem:

overcloud update fails with this error:
2019-06-24 03:39:45 | fatal: [controller-2]: FAILED! => {"changed": true, "cmd": "nova-manage cell_v2 list_cells --verbose | grep 'mysql+pymysql://nova:UCRUMjFVXPNQuGgNeUbRAwuTU.1.29/nova_cell0?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf' || nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --transport-url 'none:///' --database_connection='mysql+pymysql://nova:UCRUMjFVXPNQuGgNeUbRAwuTU.1.29/nova_cell0?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf'", "delta": "0:00:08.022272", "end": "2019-06-24 07:39:43.993213", "msg": "non-zero return code", "rc": 1, "start": "2019-06-24 07:39:35.970941", "stderr": "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported\n  exception.NotSupportedWarning\n/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported\n  exception.NotSupportedWarning", "stderr_lines": ["/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported", "  exception.NotSupportedWarning", "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported", "  exception.NotSupportedWarning"], "stdout": "An error has occurred:\nTraceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/nova/cmd/manage.py\", line 1654, in main\n    ret = fn(*fn_args, **fn_kwargs)\n  File \"/usr/lib/python2.7/site-packages/nova/cmd/manage.py\", line 1499, in update_cell\n    cell_mapping = objects.CellMapping.get_by_uuid(ctxt, cell_uuid)\n  File \"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py\", line 184, in wrapper\n    result = fn(cls, context, *args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/nova/objects/cell_mapping.py\", line 65, in get_by_uuid\n    db_mapping = cls._get_by_uuid_from_db(context, uuid)\n  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 987, in wrapper\n    with self._transaction_scope(context):\n  File \"/usr/lib64/python2.7/contextlib.py\", line 17, in __enter__\n    return self.gen.next()\n  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 1037, in _transaction_scope\n    context=context) as resource:\n  File \"/usr/lib64/python2.7/contextlib.py\", line 17, in __enter__\n    return self.gen.next()\n  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 640, in _session\n    bind=self.connection, mode=self.mode)\n  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 404, in _create_session\n    self._start()\n  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 491, in _start\n    engine_args, maker_args)\n  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 513, in _setup_for_connection\n    \"No sql_connection parameter is established\")\nCantStartEngineError: No sql_connection parameter is established", "stdout_lines": ["An error has occurred:", "Traceback (most recent call last):", "  File \"/usr/lib/python2.7/site-packages/nova/cmd/manage.py\", line 1654, in main", "    ret = fn(*fn_args, **fn_kwargs)", "  File \"/usr/lUpdate failed with: Ansible failed, check log at /var/lib/mistral/8bd5ee37-71f5-4ed3-b84c-ac854c0e6235/ansible.log.
2019-06-24 03:39:45 | ib/python2.7/site-packages/nova/cmd/manage.py\", line 1499, in update_cell", "    cell_mapping = objects.CellMapping.get_by_uuid(ctxt, cell_uuid)", "  File \"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py\", line 184, in wrapper", "    result = fn(cls, context, *args, **kwargs)", "  File \"/usr/lib/python2.7/site-packages/nova/objects/cell_mapping.py\", line 65, in get_by_uuid", "    db_mapping = cls._get_by_uuid_from_db(context, uuid)", "  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 987, in wrapper", "    with self._transaction_scope(context):", "  File \"/usr/lib64/python2.7/contextlib.py\", line 17, in __enter__", "    return self.gen.next()", "  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 1037, in _transaction_scope", "    context=context) as resource:", "  File \"/usr/lib64/python2.7/contextlib.py\", line 17, in __enter__", "    return self.gen.next()", "  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 640, in _session", "    bind=self.connection, mode=self.mode)", "  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 404, in _create_session", "    self._start()", "  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 491, in _start", "    engine_args, maker_args)", "  File \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 513, in _setup_for_connection", "    \"No sql_connection parameter is established\")", "CantStartEngineError: No sql_connection parameter is established"]}

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. install osp13 z5
2. update to passed_phase 2 - currently 2019-06-20.1
3.

Actual results:


Expected results:


Additional info:
controller update log is attached

Comment 1 Raviv Bar-Tal 2019-06-24 11:01:32 UTC
more logs can be found in this url:
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/OSPD-Customized-Update-virt/43/artifact/

Comment 3 Yuri Obshansky 2019-06-24 18:16:13 UTC
The same issue happened when running minor update from z4 up to passed_phase2 / 2019-06-20.1
The job https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-edge-day2-13-rhel-virthost-3cont-spine-2leaf-2comp-update/1/

Comment 12 errata-xmlrpc 2019-07-10 13:05:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1738