Bug 1106450
Summary: | mysqld doesn't start after installation of mysql catridge on systems with high IO wait | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Miheer Salunke <misalunk> | |
Component: | ImageStreams | Assignee: | John W. Lamb <jolamb> | |
Status: | CLOSED ERRATA | QA Contact: | libra bugs <libra-bugs> | |
Severity: | low | Docs Contact: | ||
Priority: | high | |||
Version: | 2.1.0 | CC: | adellape, bleanhar, gpei, jialiu, jkeck, jokerman, jolamb, lmeyer, misalunk, mmccomas | |
Target Milestone: | --- | Keywords: | Upstream | |
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | openshift-origin-cartridge-mysql-1.23.4.5-1.el6op | Doc Type: | Bug Fix | |
Doc Text: |
The control script "start" routine for the MySQL cartridge was configured to check the mysqld service 10 times, wait 1 second between each check, and time out if the service did not become available within that time. During periods of high I/O load, and under certain other conditions, the mysqld service might fail to start before all 10 checks have been performed. This caused the deployment or scale-up operation to fail. This bug fix increases the number of checks to 45 for a minimum timeout duration of 45 seconds. It also introduces the OPENSHIFT_MYSQL_START_TIMEOUT and OPENSHIFT_MYSQL_STOP_TIMEOUT environment variables which users can set using the client tools to specify the number of retries for the control "start" and "stop" routines, respectively. The MySQL cartridge is now more tolerant to high latency system conditions by default, and can be manipulated by the user to successfully deploy under a variety of system load and latency scenarios. After applying this fix, a cartridge upgrade is required.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1147060 (view as bug list) | Environment: | ||
Last Closed: | 2014-10-02 13:59:26 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1147060 |
Description
Miheer Salunke
2014-06-09 11:43:19 UTC
Allow setting of timeout with an env var at app create time? Upstream fix: https://github.com/openshift/origin-server/pull/5785 Sets start timeout to 45 seconds, allows setting of timeout at app-create using user env var OPENSHIFT_MYSQL_START_TIMEOUT. Also stop timeout can be set with OPENSHIFT_MYSQL_STOP_TIMEOUT. Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/afb1d7117522eaf9cb88bd31858031ff28438652 mysql cart: user-settable start timeout Increase startup timeout for MySQL cart to 45 seconds, allow setting `start` timeout via `OPENSHIFT_MYSQL_START_TIMEOUT` user env var and `stop` timeout via `OPENSHIFT_MYSQL_STOP_TIMEOUT`. Enterprise bug 1106450 https://bugzilla.redhat.com/show_bug.cgi?id=1106450 This is marked as ON_QA, but there's no puddle available for it yet. One should be available tomorrow, or if you want to test against the current 2.1.z puddle you can pull the updated openshift-origin-cartridge-mysql from brew: https://brewweb.devel.redhat.com/buildinfo?buildID=386459 Verify this bug with package openshift-origin-cartridge-mysql-1.23.4.5-1.el6op Checked the code, the default start_timeout in /usr/libexec/openshift/cartridges/mysql/bin/control has been set to 45s now. OPENSHIFT_MYSQL_START_TIMEOUT and OPENSHIFT_MYSQL_STOP_TIMEOUT env variable could work. Steps: 1. Create one scalable app, set env variable for this app rhc env-set OPENSHIFT_MYSQL_START_TIMEOUT=5 OPENSHIFT_MYSQL_STOP_TIMEOUT=5 -a app1 2. Log into the app to check the env variable in the app [app1-11.ose21z-manual.com.cn 541b9c99db26c8ae500002ea]\> env|grep TIMEOUT OPENSHIFT_MYSQL_START_TIMEOUT=5 OPENSHIFT_MYSQL_STOP_TIMEOUT=5 3. Try to add mysql cartridge to this app [root@broker ~]# rhc cartridge-add -a app1 -c mysql-5.5 Adding mysql-5.5 to application 'app1' ... Starting MySQL 5.5 cartridge MySQL server failed to start: 140918 23:03:37 mysqld_safe Logging to '/var/lib/openshift/541bc72cdb26c8ae5000041d/mysql//stdout.err'. ... 140918 23:03:41 InnoDB: Log file ./ib_logfile1 did not exist: new to be created InnoDB: Setting log file ./ib_logfile1 size to 153 MB InnoDB: Database physically writes the file full: wait... InnoDB: Progress in MB: 100 Failed to execute: 'control start' for /var/lib/openshift/541bc72cdb26c8ae5000041d/mysql 4. Make OPENSHIFT_MYSQL_START_TIMEOUT to a bigger one, then add mysql cartridge again [root@broker ~]# rhc env-set OPENSHIFT_MYSQL_START_TIMEOUT=40 -a app1 Setting environment variable(s) ... done [root@broker ~]# rhc cartridge-add -a app1 -c mysql-5.5 -g medium Adding mysql-5.5 to application 'app1' ... done mysql-5.5 (MySQL 5.5) ... Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/ Setting version to 2.1.0 to reflect product fix was shipped for, and to distinguish from soon-to-be-created clone of this bug for 2.0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1353.html |