Bug 1083109

Summary:	MySQL 5.5 never restarts properly, causing repository action_hooks not to fire (leading to data loss in our use case)
Product:	OpenShift Online	Reporter:	steven.merrill
Component:	Image	Assignee:	Ben Parees <bparees>
Status:	CLOSED NOTABUG	QA Contact:	libra bugs <libra-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	2.x	CC:	dmueller2001, steven.merrill, vvitek
Target Milestone:	---	Keywords:	UpcomingRelease
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-04-02 08:11:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description steven.merrill 2014-04-01 13:44:17 UTC

Description of problem:

MySQL 5.5 does not properly restart, causing Quickstart hooks not to fire

Version-Release number of selected component (if applicable):

OpenShift Online

How reproducible:

Make an application with the php-5.3, cron, and mysql-5.5 cartridges. Push new code. Every time, MySQL 5.5 will fail to start.

This unfortunately caused data loss for us, as well, since the repository action_hooks did not fire.

Steps to Reproduce:
1. Make a new LAMP stack app: rhc app-create openatrium php-5.3 mysql-5.5 cron https://cartreflect-claytondev.rhcloud.com/reflect?github=smerrill/openshift-community-pressflow7 --from-code=https://github.com/smerrill/openatrium-2-quickstart.git
2. Install OpenAtrium, add a new commit and push
3. Watch the server error out with this message:
remote: Stopping MySQL 5.5 cartridge
remote: Stopping PHP 5.3 cartridge (Apache+mod_php)
remote: Waiting for stop to finish
remote: Stopping Cron cartridge
remote: Building git ref 'master', commit 756d528
remote: CLIENT_MESSAGE: The deplist.txt file was deprecated in favor of .openshift/pear.txt file.
remote: CLIENT_MESSAGE: Please, consider running `git mv deplist.txt .openshift/pear.txt' command.
remote: Checking deplist.txt for PEAR dependency..
remote: Checking PEAR: pear.drush.org/drush-6.0.0
remote:
remote: drush/drush is already installed and is the same as the released version 6.0.0
remote: upgrade failed
remote:
remote: Preparing build for deployment
remote: Deployment id is de07739e
remote: Activating deployment
remote: Starting MySQL 5.5 cartridge
remote: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.so
ck' (2)

Actual results:

The other cartridges do not start, and important hooks like https://github.com/smerrill/openatrium-2-quickstart/blob/master/.openshift/action_hooks/post_start_php did not fire, leading to data loss as the existing files directory was written over.

Expected results:

MySQL should start and all action_hooks should run.

Additional info:

I'm using MySQL 5.5 because it's the only cartridge where we can enable the innodb_file_per_table flag since it is a global option now: https://github.com/openshift/origin-server/pull/5091 . Without that option, our gear was filling up very quickly with no way to recover the space except by backing up, removing the cartridge, re-adding the cartridge, and restoring the database. If MySQL 5.5 doesn't work and MySQL 5.1 can't add innodb_file_per_table, OpenShift Online isn't a great option for applications that generate a decent amount of MySQL data.

Comment 1 Diane Mueller 2014-04-01 15:43:42 UTC

impacts the Drupal-based Origin community site

Comment 2 Vojtech Vitek 2014-04-01 16:39:20 UTC

> Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'
Did you try connecting to MySQL using explicit $OPENSHIFT_MYSQL_DB_SOCKET env var value instead of 'localhost'? Did you try connecting over 'tcp' protocol ($OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT) instead of 'socket'?

Comment 3 steven.merrill 2014-04-01 16:55:18 UTC

Vojtech,

I don't have a problem when the MySQL cartridge is running - PHP and Drupal can connect just fine. There's a problem when the cartridge stops and starts after a git push - since we switched to MySQL 5.5 on this application, it has never successfully started up again after a git push, and that means a few things:

1) The other cartridges (php, cron, pressflow7) don't start up.
2) The action_hooks in the main repository do not run (and thus the symlink for the files directory is not created)

To get the application up and running again, I have to SSH into the gear and "gear restart" the mysql, cron, and php cartridges since the cron and php cartridges are not running at that point. An invocation of "gear restart" over SSH stops and starts MySQL 5.5 just fine, but the git post-receive hook version does not for some reason.

Comment 4 Andy Goldstein 2014-04-01 18:29:23 UTC

I've tried to reproduce but have not had much luck. If you can make the error happen reliably, please let me know and I'll resume looking into it. My guess at this point is that something weird is going on either in your pressflow cart or in your quickstart's action hooks.

Comment 5 steven.merrill 2014-04-02 08:11:33 UTC

Thanks to the team for the help tracking this down today.

The issue ended up being with my post_start_mysql action in the quickstart. I have moved the MySQL innodb_file_per_table setting to post_deploy in https://github.com/smerrill/openatrium-2-quickstart/commit/a1db47410e5df6c5b20c52d2fd8771cbfa69db0b and it appears to be working well.