Bug 1083109 - MySQL 5.5 never restarts properly, causing repository action_hooks not to fire (leading to data loss in our use case)
Summary: MySQL 5.5 never restarts properly, causing repository action_hooks not to fir...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Image
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Ben Parees
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-04-01 13:44 UTC by steven.merrill
Modified: 2015-05-15 00:40 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-04-02 08:11:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1039849 0 low CLOSED php and zend cartridges are missing mysql default socket 2021-02-22 00:41:40 UTC

Internal Links: 1039849

Description steven.merrill 2014-04-01 13:44:17 UTC
Description of problem:

MySQL 5.5 does not properly restart, causing Quickstart hooks not to fire

Version-Release number of selected component (if applicable):

OpenShift Online

How reproducible:

Make an application with the php-5.3, cron, and mysql-5.5 cartridges. Push new code. Every time, MySQL 5.5 will fail to start.

This unfortunately caused data loss for us, as well, since the repository action_hooks did not fire.

Steps to Reproduce:
1. Make a new LAMP stack app: rhc app-create openatrium php-5.3 mysql-5.5 cron https://cartreflect-claytondev.rhcloud.com/reflect?github=smerrill/openshift-community-pressflow7 --from-code=https://github.com/smerrill/openatrium-2-quickstart.git
2. Install OpenAtrium, add a new commit and push
3. Watch the server error out with this message:
remote: Stopping MySQL 5.5 cartridge
remote: Stopping PHP 5.3 cartridge (Apache+mod_php)
remote: Waiting for stop to finish
remote: Stopping Cron cartridge
remote: Building git ref 'master', commit 756d528
remote: CLIENT_MESSAGE: The deplist.txt file was deprecated in favor of .openshift/pear.txt file.
remote: CLIENT_MESSAGE: Please, consider running `git mv deplist.txt .openshift/pear.txt' command.
remote: Checking deplist.txt for PEAR dependency..
remote: Checking PEAR: pear.drush.org/drush-6.0.0
remote:
remote: drush/drush is already installed and is the same as the released version 6.0.0
remote: upgrade failed
remote:
remote: Preparing build for deployment
remote: Deployment id is de07739e
remote: Activating deployment
remote: Starting MySQL 5.5 cartridge
remote: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.so
ck' (2)

Actual results:

The other cartridges do not start, and important hooks like https://github.com/smerrill/openatrium-2-quickstart/blob/master/.openshift/action_hooks/post_start_php did not fire, leading to data loss as the existing files directory was written over.

Expected results:

MySQL should start and all action_hooks should run.

Additional info:

I'm using MySQL 5.5 because it's the only cartridge where we can enable the innodb_file_per_table flag since it is a global option now: https://github.com/openshift/origin-server/pull/5091 . Without that option, our gear was filling up very quickly with no way to recover the space except by backing up, removing the cartridge, re-adding the cartridge, and restoring the database. If MySQL 5.5 doesn't work and MySQL 5.1 can't add innodb_file_per_table, OpenShift Online isn't a great option for applications that generate a decent amount of MySQL data.

Comment 1 Diane Mueller 2014-04-01 15:43:42 UTC
impacts the Drupal-based Origin community site

Comment 2 Vojtech Vitek 2014-04-01 16:39:20 UTC
> Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'
Did you try connecting to MySQL using explicit $OPENSHIFT_MYSQL_DB_SOCKET env var value instead of 'localhost'? Did you try connecting over 'tcp' protocol ($OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT) instead of 'socket'?

Comment 3 steven.merrill 2014-04-01 16:55:18 UTC
Vojtech,

I don't have a problem when the MySQL cartridge is running - PHP and Drupal can connect just fine. There's a problem when the cartridge stops and starts after a git push - since we switched to MySQL 5.5 on this application, it has never successfully started up again after a git push, and that means a few things:

1) The other cartridges (php, cron, pressflow7) don't start up.
2) The action_hooks in the main repository do not run (and thus the symlink for the files directory is not created)

To get the application up and running again, I have to SSH into the gear and "gear restart" the mysql, cron, and php cartridges since the cron and php cartridges are not running at that point. An invocation of "gear restart" over SSH stops and starts MySQL 5.5 just fine, but the git post-receive hook version does not for some reason.

Comment 4 Andy Goldstein 2014-04-01 18:29:23 UTC
I've tried to reproduce but have not had much luck. If you can make the error happen reliably, please let me know and I'll resume looking into it. My guess at this point is that something weird is going on either in your pressflow cart or in your quickstart's action hooks.

Comment 5 steven.merrill 2014-04-02 08:11:33 UTC
Thanks to the team for the help tracking this down today.

The issue ended up being with my post_start_mysql action in the quickstart. I have moved the MySQL innodb_file_per_table setting to post_deploy in https://github.com/smerrill/openatrium-2-quickstart/commit/a1db47410e5df6c5b20c52d2fd8771cbfa69db0b and it appears to be working well.


Note You need to log in before you can comment on or make changes to this bug.