Bug 1085282
| Summary: | mysql 5.5 stops working with "InnoDB: Fatal error: cannot initialize AIO sub-system" | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Andre Dietisheim <adietish> | ||||
| Component: | Image | Assignee: | Michal Fojtik <mfojtik> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 2.x | CC: | admiller, bparees, chunchen, jhadvig, wzheng, xtian | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-05-15 15:28:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 883943 [details]
mysql log
Ben, this bug is hard to reproduce in devenv, but I saw this before. The problem is that MySQL 5.5 has an internal issue[1] how it detects the AIO. In some cases (especially in PROD), we hit this issue in some gears... We have several options here: 1) Disable innodb_use_native_aio in config file. This will hit performance of MySQL, but the MySQL should work... 2) Make it configurable (rhc env), so in case users hit this issue they can start the MySQL, backup data, etc... 3) Work on getting this fixed in upstream or investigate if this is not already fixed there (then just bump the mysql version in SCL)> [1] http://help.directadmin.com/item.php?id=529 Also worked on this issue and prepared an PR, after which I realized that disabling the innodb_use_native_aio setting, will slow the performance of the MySQL. Closed PR: https://github.com/openshift/origin-server/pull/5189 From documentation: aio-nr & aio-max-nr: aio-nr is the running total of the number of events specified on the io_setup system call for all currently active aio contexts. If aio-nr reaches aio-max-nr then io_setup will fail with EAGAIN. Note that raising aio-max-nr does not result in the pre-allocation or re-sizing of any kernel data structures. https://www.kernel.org/doc/Documentation/sysctl/fs.txt OPS: Do you think we can tune this to fix this bug? PR: https://github.com/openshift/origin-server/pull/5204 There is nothing more we can do from a cartridge point of view here except provide users a way to disable Native AIO in MySQL when this issue occurs. This will make the MySQL performance a bit worse, but MySQL will still operate without any problems and so users can take a snapshot of MySQL database and remove/add the MySQL cartridge. To QA: To test this, just set the OPENSHIFT_MYSQL_AIO=0 and restart the mysql cartridge in application. Then rhc ssh into mysql gear and verify that the 'innodb_use_native_aio' is disabled (set to 0) Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/8c0633c07d90d4b27f450d653ecc574a5fc877ff Bug 1085282 - Added OPENSHIFT_MYSQL_AIO variable to allow users to disable mysql AIO support It's fixed, verified on devenv_4655, please refer to the following results: 1. Create an app with mysql-5.5 rhc app create cpy27 python-2.7 mysql-5.5 --no-git 2. Set the OPENSHIFT_MYSQL_AIO=0 for mysql cart rhc env-set OPENSHIFT_MYSQL_AIO=0 -a cpy27 3. Restart the mysql-5.5 rhc cartridge restart mysql-5 -a cpy27 4. SSH into this app rhc ssh cpy27 mysql mysql> show variables where Variable_name = 'innodb_use_native_aio'; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | innodb_use_native_aio | OFF | +-----------------------+-------+ 1 row in set (0.00 sec) The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: My application (a redmine quickstart with ruby-1.9. mysql-5.5 and phpmyadmin) stopped working today. The error log shows the following: Starting MySQL 5.5 cartridge MySQL server failed to start: 140408 03:31:45 mysqld_safe Logging to '/var/lib/openshift/533079a150044663b400033d/mysql//log/mysql_error.log'. 140408 03:31:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/533079a150044663b400033d/mysql/data/ 140408 03:31:48 mysqld_safe mysqld from pid file /var/lib/openshift/533079a150044663b400033d/mysql/pid/mysql.pid ended 140408 03:31:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/533079a150044663b400033d/mysql/data/ 140408 3:31:45 [Note] Plugin 'FEDERATED' is disabled. 140408 3:31:45 InnoDB: The InnoDB memory heap is disabled 140408 3:31:45 InnoDB: Mutexes and rw_locks use GCC atomic builtins 140408 3:31:45 InnoDB: Compressed tables use zlib 1.2.3 140408 3:31:45 InnoDB: Using Linux native AIO 140408 3:31:45 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up. InnoDB: Warning: io_setup() attempt 1 failed. InnoDB: Warning: io_setup() attempt 2 failed. InnoDB: Warning: io_setup() attempt 3 failed. InnoDB: Warning: io_setup() attempt 4 failed. InnoDB: Warning: io_setup() attempt 5 failed. 140408 3:31:48 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts. InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf 140408 3:31:48 InnoDB: Fatal error: cannot initialize AIO sub-system 140408 3:31:48 [ERROR] Plugin 'InnoDB' init function returned error. 140408 3:31:48 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 140408 3:31:48 [ERROR] Unknown/unsupported storage engine: InnoDB 140408 3:31:48 [ERROR] Aborting 140408 3:31:48 [Note] /opt/rh/mysql55/root/usr/libexec/mysqld: Shutdown complete 140408 03:31:48 mysqld_safe mysqld from pid file /var/lib/openshift/533079a150044663b400033d/mysql/pid/mysql.pid ended Failed to execute: 'control restart' for /var/lib/openshift/533079a150044663b400033d/mysql Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: