Bug 1045342

Also meet this error "A gear start did not complete on 1 gear. Please try again and contact support if the issue persists." when restarting a non-scalable app after it hitting limit of quota via rhc & web console.

Create a non-scalable app and add its storage until it hits limit of quota. Then check its page and try to restart via rhc & web console. 
Fail to restart the app with error "Warning: Gear 52b3b1ee03ef64f58f0000be is using 100.0% of disk quota. A gear restart did not complete on 1 gear. Please try again and contact support if the issue persists.". But it can be stopped and then started.

Steps to Reproduce:
1. Create a non-scalable app and add storage until hit limitation of quota

[php-lala.int.rhcloud.com php]\> dd if=/dev/zero of=testfile count=1000M
dm-0: write failed, user block limit reached.

2. Try to restart the app

$ rhc app restart php                                                                                                                                          
Warning: Gear 52b3b1ee03ef64f58f0000be is using 100.0% of disk quota
A gear restart did not complete on 1 gear. Please try again and contact support if the issue persists.

$ rhc app stop php                                                                                                                                 
Warning: Gear 52b3b1ee03ef64f58f0000be is using 100.0% of disk quota

RESULT:
php stopped

$ rhc app start php
Warning: Gear 52b3b1ee03ef64f58f0000be is using 100.0% of disk quota

RESULT:
php started

3. Visit the app in browser, and find it can still be access.

Here is log when try to restart the app

==> php/logs/error_log-20131220-000000-EST <==
piped log program '/usr/sbin/rotatelogs /var/lib/openshift/52b3b1ee03ef64f58f0000be/php//logs/access_log-%Y%m%d-%H%M%S-%Z 86400' failed unexpectedly
Error writing to the file /var/lib/openshift/52b3b1ee03ef64f58f0000be/php//logs/access_log-20131220-000000-EST
piped log program '/usr/sbin/rotatelogs /var/lib/openshift/52b3b1ee03ef64f58f0000be/php//logs/access_log-%Y%m%d-%H%M%S-%Z 86400' failed unexpectedly
[Thu Dec 19 22:15:49 2013] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c4,c138
[Thu Dec 19 22:15:49 2013] [notice] Digest: generating secret for digest authentication ...
[Thu Dec 19 22:15:49 2013] [notice] Digest: done
[Thu Dec 19 22:15:49 2013] [notice] Apache/2.2.22 (Unix) configured -- resuming normal operations
[Thu Dec 19 22:16:06 2013] [notice] Digest: generating secret for digest authentication ...
[Thu Dec 19 22:16:06 2013] [notice] Digest: done
[Thu Dec 19 22:16:06 2013] [notice] Apache/2.2.22 (Unix) configured -- resuming normal operations

The affected application: php (uuid: 52b3b1ee03ef64f58f0000be)

This is also reproduced after upgrade from devenv-stage_619 to devenv_4160. According to the development.log from broker, the mysqld process of got a normal shutdown after it has initialized buffer pools and log sequence.

The mysql cartridge version is openshift-origin-cartridge-mysql-1.19.2-1.git.0.e08a705.el6.noarch

On the broker, find the mysql gear dir and start the mysql gear as root directly:
<-------------------->
[root@ip-10-139-10-138 ~]# /usr/bin/mysqld_safe --defaults-file=/var/lib/openshift/52b3be3d97452eea95000223/mysql/conf/my.cnf
131220 04:04:50 mysqld_safe Logging to '/var/lib/openshift/52b3be3d97452eea95000223/mysql//log//mysql_error.log'.
131220 04:04:51 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/52b3be3d97452eea95000223/mysql//data/
131220 04:04:51 mysqld_safe mysqld from pid file /var/lib/openshift/52b3be3d97452eea95000223/mysql//pid/mysql.pid ended
<-------------------->

On the other hand, 'tailf /var/lib/openshift/52b3be3d97452eea95000223/mysql//log//mysql_error.log' to find out why this fails:
<-------------------------------------->
131220 04:06:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/52b3be3d97452eea95000223/mysql//data/
/usr/libexec/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
131220  4:06:12 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
131220  4:06:12  InnoDB: Initializing buffer pool, size = 32.0M
131220  4:06:12  InnoDB: Completed initialization of buffer pool
131220  4:06:12  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ibdata1
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.
131220 04:06:12 mysqld_safe mysqld from pid file /var/lib/openshift/52b3be3d97452eea95000223/mysql//pid/mysql.pid ended
<-------------------------------------->

I guess the problem is something caused by mysql(or selinux)permission issues.

Also met the error when control the app without mysql embedded on INT(devenv_4169). And the app show 503 in web page.

How reproducible:
Sometimes

#rhc app stop pacman -l yadu+m
A gear stop did not complete on 1 gear. Please try again and contact support if the issue persists.

#rhc app start pacman -l yadu+m
A gear start did not complete on 1 gear. Please try again and contact support if the issue persists.


# rhc app show -a pacman 
pacman @ http://pacman-dyint.int.rhcloud.com/ (uuid: 52aa7dec03ef649bee000333)
------------------------------------------------------------------------------
  Domain:          dyint
  Created:         Dec 13 11:24 AM
  Gears:           1 (defaults to small)
  Git URL:         ssh://52aa7dec03ef649bee000333.rhcloud.com/~/git/pacman.git/
  Initial Git URL: git://github.com/ramr/pacman.git
  SSH:             52aa7dec03ef649bee000333.rhcloud.com
  Deployment:      auto (on git push)

  nodejs-0.10 (Node.js 0.10)
  --------------------------
    Gears: Located with rockmongo-1.1, mongodb-2.2

  rockmongo-1.1 (RockMongo 1.1)
  -----------------------------
    Gears:          Located with nodejs-0.10, mongodb-2.2
    Connection URL: https://pacman-dyint.int.rhcloud.com/rockmongo/

  mongodb-2.2 (MongoDB 2.2)
  -------------------------
    Gears:          Located with nodejs-0.10, rockmongo-1.1
    Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/
    Database Name:  pacman
    Password:       P-WGXCgV7d7K
    Username:       admin

# rhc app show pacman --state
Cartridge nodejs-0.10, rockmongo-1.1, mongodb-2.2 is started

This bug seems to address at least 3 different bugs :-)

https://bugzilla.redhat.com/show_bug.cgi?id=1045342#c3 ->

I saw this error when I tried to use the new mysql55 cartridge with database created using the older mysql51. So I guess the problem here is that the 'mysql' cartridge in your application was somehow upgraded to a mysql55 while the database data were not upgraded.

Or you are trying to use the new mysql55 using database created with mysql51, which is, currently, not supported.

Chunchen, do you see this in the logs as well ^^. In that case, something in migration/upgrade is broken and it is trying to update mysql51 to mysql55 while using the old database.

https://bugzilla.redhat.com/show_bug.cgi?id=1045342#c4 -> I don't see mysql here, just mongodb.

https://bugzilla.redhat.com/show_bug.cgi?id=1045342#c1 -> This seems to be an unrelated to this bug. Perhaps a new PHP bug?

To QA: Guys can you please retest this, as I just found that similar bug (https://bugzilla.redhat.com/show_bug.cgi?id=1045337) seems to be now automagically fixed in INT.

I will verify it after INT migration.

Test on INT(devenv_4205)
Accroding to Comment1, create a non-scalable app and add its storage until it hits limit of quota. Then check its page and try to restart via rhc & web console. The app can be restart successfully via rhc&web console. I can't reproduce it again. So no other problem with it.

INT(devenv_4205)
This issue still was there for me(xiuwang+m), for starting/restarting existed application with embedding mysql will show "A gear start did not complete on 1 gear. Please try again and contact support if the issue persists."

@Michal Fojtik, the issue is reproduced, retested it on INT(devenv_4205), please help to check again, some affected apps as below:

cpython27mngpsq84mms @ http://cpython27mngpsq84mms-cdm.int.rhcloud.com/ (uuid: 52ce0e446cec0e3f0700023a)
cruby19mngpsq84mms @ http://cruby19mngpsq84mms-cdm.int.rhcloud.com/ (uuid: 52ce0ea06cec0eb868000108)
cjbossews20mngpsq84mms @ http://cjbossews20mngpsq84mms-cdm.int.rhcloud.com/ (uuid: 52ce0f0303ef64586c00087d)
cpython27ssmin2gearsjkns @ http://cpython27ssmin2gearsjkns-cdm.int.rhcloud.com/ (uuid: 52ce16e003ef64ed4800012f)
rails @ http://rails-cdm.int.rhcloud.com/ (uuid: 52ce32536cec0ec15a000114)
cruby19ssmin2gearsjkns @ http://cruby19ssmin2gearsjkns-cdm.int.rhcloud.com/ (uuid: 52ce18226cec0ec570000093)

Finally I reproduce this on devenv. The issue is mysql-5.1 where the OPENSHIFT_MYSQL_VERSION variable is not available (and thus mysql_context function does not launch any command). This should fixed in this PR:

https://github.com/openshift/origin-server/pull/4459

I tested mysql-5.1, mysql-5.5 both scaled and not scaled... everything seems to work OK.

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/e6f853b040984917b7e7072c0f4c6f407c67ffd1
Bug 1045342 - Fix the $MYSQL_VERSION env var is missing for mysql-5.1

It's fixed, verified on INT(devenv_4223), please refer to the following results:

$ rhc app show cruby19ssmin2gearsjkns
cruby19ssmin2gearsjkns @ http://cruby19ssmin2gearsjkns-cdm.int.rhcloud.com/ (uuid: 52ce18226cec0ec570000093)
------------------------------------------------------------------------------------------------------------
  Domain:     cdm
  Created:    Jan 09 11:31 AM
  Gears:      4 (defaults to small)
  Git URL:    ssh://52ce18226cec0ec570000093.rhcloud.com/~/git/cruby19ssmin2gearsjkns.git/
  SSH:        52ce18226cec0ec570000093.rhcloud.com
  Deployment: auto (on git push)

  mysql-5.1 (MySQL 5.1)
  ---------------------
    Gears:          1 small
    Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
    Database Name:  cruby19ssmin2gearsjkns
    Password:       uz9IIw5BYp1m
    Username:       adminhMaGeq1

  mongodb-2.2 (MongoDB 2.2)
  -------------------------
    Gears:          1 small
    Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/
    Database Name:  cruby19ssmin2gearsjkns
    Password:       ZSjJ6IRnmz17
    Username:       admin

  ruby-1.9 (Ruby 1.9)
  -------------------
    Scaling: x2 (minimum: 2, maximum: available) on small gears

  haproxy-1.4 (Web Load Balancer)
  -------------------------------
    Gears: Located with ruby-1.9, cron-1.4, jenkins-client-1

  cron-1.4 (Cron 1.4)
  -------------------
    Scaling: x2 (minimum: 2, maximum: available) on small gears

  jenkins-client-1 (Jenkins Client)
  ---------------------------------
    Gears:   Located with ruby-1.9, haproxy-1.4, cron-1.4
    Job URL: https://jenkins-cdm.int.rhcloud.com/job/cruby19ssmin2gearsjkns-build/

$ rhc app start cruby19ssmin2gearsjkns
RESULT:
cruby19ssmin2gearsjkns started

$ rhc app restart cruby19ssmin2gearsjkns
RESULT:
cruby19ssmin2gearsjkns restarted