Bug 1624254
| Summary: | innodb_log_file_size is low for Openstack HCI with high CephAnsiblePlaybookVerbosity | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Anil Dhingra <adhingra> |
| Component: | openstack-tripleo-common | Assignee: | Giulio Fidente <gfidente> |
| Status: | CLOSED DUPLICATE | QA Contact: | Yogev Rabl <yrabl> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 13.0 (Queens) | CC: | adhingra, gfidente, johfulto, mburns, slinaber |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-10-10 12:27:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
workaround ~~~ The default size is; [root@gti-rhosp-director ~]# mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 131782 Server version: 10.1.20-MariaDB MariaDB Server Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> SHOW VARIABLES WHERE Variable_Name LIKE "innodb_log_file_size"; +----------------------+----------+ | Variable_name | Value | +----------------------+----------+ | innodb_log_file_size | 50331648 | +----------------------+----------+ 1 row in set (0.00 sec) Increased this value to 1G as below. [root@gti-rhosp-director ~]# cat /etc/my.cnf.d/mistral-fix.cnf # To fix mistral hang when storing ansible logs. [mysqld] innodb_log_file_size = 1G and restarted the mariadb.service [root@gti-rhosp-director ~]# mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 58 Server version: 10.1.20-MariaDB MariaDB Server Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> SHOW VARIABLES WHERE Variable_Name LIKE "innodb_log_file_size"; +----------------------+------------+ | Variable_name | Value | +----------------------+------------+ | innodb_log_file_size | 1073741824 | +----------------------+------------+ 1 row in set (0.00 sec) I am tempted to close this as duplicate of BZ#1602840 ; not sure if you are able to test the fix at https://review.openstack.org/588083 to make sure it fixes the issue? haven't tried fix as above workaround work for me ,yes it looks similar to BZ#1602840 *** This bug has been marked as a duplicate of bug 1602840 *** |
Description of problem: Looks like the ceph-ansible logs been pushed to store in the Undercloud's Mariadb and deployment has failed due to low innodb_log_file_size ,if we need to increase CephAnsiblePlaybookVerbosity to troubleshoot it generate more logs & default size is not enough Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: analysis on /var/log/mistral/engine.log showed me below error just after the ceph-ansible play completion. INSTALLER STATUS *************************************************************** Install Ceph Monitor : Complete (0:02:41) Install Ceph Manager : Complete (0:00:31) Install Ceph OSD : Complete (0:11:56) Install Ceph Client : Complete (0:01:22) Wednesday 29 August 2018 23:04:44 +0700 (0:00:00.106) 0:17:18.758 ****** =============================================================================== ..., error=None, cancel=False]] 2018-08-29 23:04:47.551 1938 INFO workflow_trace [req-e4e56005-f8bd-4d00-b69d-a21402bc4f22 b0aa3fc44f2744c687bd42fae519a986 6a5995b0cd574b28a0187ed274fb738b - default default] Action 'tripleo.ansible-playbook' ( 3b01560d-022a-461a-9a42-60f179eb76b9)(task=ceph_install) [RUNNING -> SUCCESS, result = {log_path: /tmp/ansible-mistral-action3TSIh6/ansible.log, stderr: Using /usr/share/ceph-ansible/a...] 2018-08-29 23:04:47.853 1938 DEBUG mistral.services.scheduler [req-43b5ed1c-cd94-4a1b-81ff-75e3ec6e468f b0aa3fc44f2744c687bd42fae519a986 6a5995b0cd574b28a0187ed274fb738b - default default] Scheduler deleted 6 de layed calls. delete_calls /usr/lib/python2.7/site-packages/mistral/services/scheduler.py:310 2018-08-29 23:04:48.039 1938 ERROR oslo_db.sqlalchemy.exc_filters [req-e4e56005-f8bd-4d00-b69d-a21402bc4f22 b0aa3fc44f2744c687bd42fae519a986 6a5995b0cd574b28a0187ed274fb738b - default default] DBAPIError excepti on wrapped from (pymysql.err.InternalError) (1118, u'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.') [SQL: u' UPDATE action_executions_v2 SET updated_at=%(updated_at)s, state=%(state)s, accepted=%(accepted)s, output=%(output)s WHERE action_executions_v2.id = %(action_executions_v2_id)s'] [parameters: {'output': '{"resul t": {"log_path": "/tmp/ansible-mistral-action3TSIh6/ansible.log", "stderr": "Using /usr/share/ceph-ansible/ansible.cfg as config file\\n\\nPLA ... (13992284 characters truncated) ... statements should not includ e jinja2 templating delimiters\\nsuch as {{ }} or {% %}. Found: {{ groups.get(mgr_group_name, []) | length > 0\\n}}\\n"}}', 'state': 'SUCCESS', 'accepted': 1, 'updated_at': datetime.datetime(2018, 8, 29, 16, 4, 47), 'action_executions_v2_id': u'3b01560d-022a-461a-9a42-60f179eb76b9'}] (Background on this error at: http://sqlalche.me/e/2j85): InternalError: (1118, u'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.') ~~ couldn't find this value been set in any of the *.cnf files in /etc/my.cnf.d of the Undercloud.