Bug 1147060 - mysqld doesn't start after installation of mysql catridge on systems with high IO wait
Summary: mysqld doesn't start after installation of mysql catridge on systems with hig...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 2.0.0
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: ---
: ---
Assignee: John W. Lamb
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 1106450
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-26 18:22 UTC by John W. Lamb
Modified: 2019-07-11 08:13 UTC (History)
12 users (show)

Fixed In Version: openshift-origin-cartridge-mysql-1.18.4-1.el6op
Doc Type: Bug Fix
Doc Text:
The control script "start" routine for the MySQL cartridge was configured to check the mysqld service 10 times, wait 1 second between each check, and time out if the service did not become available within that time. During periods of high I/O load, and under certain other conditions, the mysqld service might fail to start before all 10 checks have been performed. This caused the deployment or scale-up operation to fail. This bug fix increases the number of checks to 45 for a minimum timeout duration of 45 seconds. It also introduces the OPENSHIFT_MYSQL_START_TIMEOUT and OPENSHIFT_MYSQL_STOP_TIMEOUT environment variables which users can set using the client tools to specify the number of retries for the control "start" and "stop" routines, respectively. The MySQL cartridge is now more tolerant to high latency system conditions by default, and can be manipulated by the user to successfully deploy under a variety of system load and latency scenarios. After applying this fix, a cartridge upgrade is required.
Clone Of: 1106450
Environment:
Last Closed: 2015-08-11 17:37:00 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:1588 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 2.0.9 bug fix update 2015-08-11 21:36:51 UTC

Comment 2 John W. Lamb 2014-10-02 19:19:18 UTC 
Waiting on build system to successfully test and merge https://github.com/openshift/enterprise-server/pull/395
Comment 6 Gaoyun Pei 2015-01-20 09:39:21 UTC 
Verify this bug with openshift-origin-cartridge-mysql-1.18.4-1.el6op.noarch.

Checked the code, the default start_timeout in /usr/libexec/openshift/cartridges/mysql/bin/control has been set to 45s now.
And OPENSHIFT_MYSQL_START_TIMEOUT and OPENSHIFT_MYSQL_STOP_TIMEOUT env variable were added to control timeout during cartridge start and stop.

Steps:
1. Create one scalable app, set env variable for this app
rhc env-set OPENSHIFT_MYSQL_START_TIMEOUT=1 OPENSHIFT_MYSQL_STOP_TIMEOUT=1 -a test1

[root@broker ~]# rhc env list -a test1
OPENSHIFT_MYSQL_START_TIMEOUT=1
OPENSHIFT_MYSQL_STOP_TIMEOUT=1


2. Try to add mysql cartridge to this app
[root@broker ~]# rhc cartridge add -a test1 -c mysql-5.1
Adding mysql-5.1 to application 'test1' ... Starting MySQL cartridge
Unable to complete the requested operation due to: An invalid exit code (157)
was returned from the server broker.ose20z.example.com.  This indicates an
unexpected problem during the execution of your request..
Reference ID: 16c681989b3bc65340a66ed893d76787

3. Make OPENSHIFT_MYSQL_START_TIMEOUT to a bigger one, then add mysql cartridge again
[root@broker ~]# rhc env-set OPENSHIFT_MYSQL_START_TIMEOUT=3 OPENSHIFT_MYSQL_STOP_TIMEOUT=1 -a test1
Setting environment variable(s) ... done
[root@broker ~]# rhc cartridge add -a test1 -c mysql-5.1
Adding mysql-5.1 to application 'test1' ... done

mysql-5.1 (MySQL 5.1)
...

MySQL 5.1 database added.  Please make note of these credentials:

...
Comment 8 errata-xmlrpc 2015-08-11 17:37:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1588.html
Note You need to log in before you can comment on or make changes to this bug.