Bug 1045337

Summary:	INT node(ex-std-node1/2.int.rhcloud.com) fails
Product:	OpenShift Online	Reporter:	Qiushui Zhang <qiuzhang>
Component:	Image	Assignee:	Michal Fojtik <mfojtik>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.x	CC:	chunchen, libra-bugs, lxia, xtian
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-01-30 00:53:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Qiushui Zhang 2013-12-20 08:06:22 UTC

Description of problem:
Keep trying to create apps. When the application is assigned to node2 (ex-std-node2.int.rhcloud.com), it will fail with the following error:

rhc app create zd1 zend-6.1 mysql-5.5
Application 'zd1' not found.
Application Options
-------------------
Domain:     qiuzhang
Cartridges: zend-6.1, mysql-5.5
Gear Size:  default
Scaling:    no

Creating application 'zd1' ... 
Unable to complete the requested operation due to: An invalid exit code (157) was returned from the server
ex-std-node2.int.rhcloud.com.  This indicates an unexpected problem during the execution of your request..
Reference ID: 7f19093674011383808831ab65db71af


Version-Release number of selected component (if applicable):
INT(devenv_4158)

How reproducible:
always

Steps to Reproduce:
1. Keep trying to create apps
2.
3.

Actual results:
When the app is assigned to node2, it will fail to be created.

rhc app create zd1 zend-6.1 mysql-5.5
Application 'zd1' not found.
Application Options
-------------------
Domain:     qiuzhang
Cartridges: zend-6.1, mysql-5.5
Gear Size:  default
Scaling:    no

Creating application 'zd1' ... 
Unable to complete the requested operation due to: An invalid exit code (157) was returned from the server
ex-std-node2.int.rhcloud.com.  This indicates an unexpected problem during the execution of your request..
Reference ID: 7f19093674011383808831ab65db71af

Expected results:
App is created successfully.

Additional info:

Comment 1 Liang Xia 2013-12-20 08:24:09 UTC

Also met this on ex-std-node1.int.rhcloud.com.

# rhc app create app2 jbossas-7 mysql-5.5 mongodb-2.2 postgresql-8.4 --no-git

Application Options
-------------------
Domain:     lxiamigrate
Cartridges: jbossas-7, mysql-5.5, mongodb-2.2, postgresql-8.4
Gear Size:  default
Scaling:    no

Creating application 'app2' ... 
Starting MySQL cartridge
Details from MySQL error log:
131220 02:59:07 mysqld_safe Starting mysqld daemon with databases from
/var/lib/openshift/52b3f86203ef64fd0700000f/mysql//data/
131220  2:59:07 [Note] Plugin 'FEDERATED' is disabled.
131220  2:59:07 InnoDB: The InnoDB memory heap is disabled
131220  2:59:07 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131220  2:59:07 InnoDB: Compressed tables use zlib 1.2.3
131220  2:59:07 InnoDB: Using Linux native AIO
131220  2:59:07 InnoDB: Initializing buffer pool, size = 32.0M
131220  2:59:07 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
131220  2:59:07  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
131220  2:59:10  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 16 MB
InnoDB: Database physically writes the file full: wait...
131220  2:59:13  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 16 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
131220  2:59:35  InnoDB: Waiting for the background threads to start
Unable to complete the requested operation due to: An invalid exit code (157) was returned from the server
ex-std-node1.int.rhcloud.com.  This indicates an unexpected problem during the execution of your request..
Reference ID: 4d0a81e207ed2bd2a120ad3fe590ebe4

Comment 2 Michal Fojtik 2014-01-03 13:51:40 UTC

I'm not able to reproduce this on devenv_4178. The '157' error is if I do remember correctly something about the database files created with different version of mysql that is trying to use them....

Comment 3 Michal Fojtik 2014-01-03 14:01:46 UTC

+ I just tried this on current INT:

~ → rhc app create app2 jbossas-7 mysql-5.5 mongodb-2.2 postgresql-8.4 --no-git
Application Options
-------------------
  Domain:     mfojtik
  Cartridges: jbossas-7, mysql-5.5, mongodb-2.2, postgresql-8.4
  Gear Size:  default
  Scaling:    no

.....

Waiting for your DNS name to be available ... done
Your application 'app2' is now available.

Can you please confirm, that this is not longer a bug?

Comment 4 chunchen 2014-01-06 03:08:07 UTC

It's fixed, Check on latest INT, please refer to the following results:

1. on ex-std-node1
$ rhc app create app5 jbossas-7 mysql-5.5 mongodb-2.2 postgresql-8.4 --no-git
Application Options
-------------------
Domain:     cdm
Cartridges: jbossas-7, mysql-5.5, mongodb-2.2, postgresql-8.4
Gear Size:  default
Scaling:    no

Creating application 'app5' ... Artifact: ./ROOT.war is still deploying
done

  PostgreSQL 8.4 database added.  Please make note of these credentials:

   Root User: adminbyt5t9z
   Root Password: gZ3l5D-F4xlV
   Database Name: app5

Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT


MongoDB 2.2 database added.  Please make note of these credentials:

   Root User:     admin
   Root Password: EYlA9ZvMXj7r
   Database Name: app5

Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/

MySQL 5.5 database added.  Please make note of these credentials:

       Root User: admintl3QkFh
   Root Password: 61vyyqtYGFZ4
   Database Name: app5

Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/

You can manage your new MySQL database by also embedding phpmyadmin.
The phpmyadmin username and password will be the same as the MySQL credentials above.

Waiting for your DNS name to be available ... done

Your application 'app5' is now available.

  URL:        http://app5-cdm.int.rhcloud.com/
  SSH to:     52ca1b0003ef648d340000d7.rhcloud.com
  Git remote: ssh://52ca1b0003ef648d340000d7.rhcloud.com/~/git/app5.git/

Run 'rhc show-app app5' for more details about your app.

$ rhc ssh app5 'hostname'
Warning: Permanently added 'app5-cdm.int.rhcloud.com' (RSA) to the list of known hosts.
ex-std-node1.int.rhcloud.com

2. on ex-std-node2
$ rhc app create app2 jbossas-7 mysql-5.5 mongodb-2.2 postgresql-8.4 --no-git
Application Options
-------------------
Domain:     cdm
Cartridges: jbossas-7, mysql-5.5, mongodb-2.2, postgresql-8.4
Gear Size:  default
Scaling:    no

Creating application 'app2' ... 
done

  PostgreSQL 8.4 database added.  Please make note of these credentials:

   Root User: adminexpxr8z
   Root Password: rJ-l2RuKZgcx
   Database Name: app2

Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT


MongoDB 2.2 database added.  Please make note of these credentials:

   Root User:     admin
   Root Password: gICHA22C6xgl
   Database Name: app2

Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/

MySQL 5.5 database added.  Please make note of these credentials:

       Root User: adminSBdysiU
   Root Password: zYzg9DHiLueD
   Database Name: app2

Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/

You can manage your new MySQL database by also embedding phpmyadmin.
The phpmyadmin username and password will be the same as the MySQL credentials above.

Waiting for your DNS name to be available ... done

Your application 'app2' is now available.

  URL:        http://app2-cdm.int.rhcloud.com/
  SSH to:     52ca0e8c03ef646442000179.rhcloud.com
  Git remote: ssh://52ca0e8c03ef646442000179.rhcloud.com/~/git/app2.git/

Run 'rhc show-app app2' for more details about your app.

$ rhc ssh app2 'hostname'
Warning: Permanently added 'app2-cdm.int.rhcloud.com' (RSA) to the list of known hosts.
ex-std-node2.int.rhcloud.com

Comment 5 Liang Xia 2014-01-06 03:09:49 UTC

Tried on current INT to create 10 apps.

# for i in `seq 1 10` ; do rhc app create app$i jbossas-7 mysql-5.5 mongodb-2.2 postgresql-8.4 --no-git ; done

1 of them still met the error:
Creating application 'app5' ... 
Starting MySQL cartridge
Details from MySQL error log:
140105 21:46:08 mysqld_safe Starting mysqld daemon with databases from
/var/lib/openshift/52ca188203ef648d34000072/mysql//data/
140105 21:46:09 [Note] Plugin 'FEDERATED' is disabled.
140105 21:46:09 InnoDB: The InnoDB memory heap is disabled
140105 21:46:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140105 21:46:09 InnoDB: Compressed tables use zlib 1.2.3
140105 21:46:09 InnoDB: Using Linux native AIO
140105 21:46:09 InnoDB: Initializing buffer pool, size = 32.0M
140105 21:46:09 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
140105 21:46:09  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
140105 21:46:10  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 16 MB
InnoDB: Database physically writes the file full: wait...
140105 21:46:10  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 16 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
140105 21:46:36  InnoDB: Waiting for the background threads to start
140105 21:46:37 InnoDB: 5.5.32 started; log sequence number 0
140105 21:46:37 [Note] Server hostname (bind-address): '127.7.43.132'; port: 3306
140105 21:46:37 [Note]   - '127.7.43.132' resolves to '127.7.43.132';
140105 21:46:37 [Note] Server socket created on IP: '127.7.43.132'.
140105 21:46:37 [Note] Event Scheduler: Loaded 0 events
140105 21:46:37 [Note] /opt/rh/mysql55/root/usr/libexec/mysqld: ready for connections.
Version: '5.5.32'  socket: '/var/lib/openshift/52ca188203ef648d34000072/mysql//socket/mysql.sock'  port: 3306  MySQL
Community Server (GPL)
Unable to complete the requested operation due to: An invalid exit code (157) was returned from the server
ex-std-node2.int.rhcloud.com.  This indicates an unexpected problem during the execution of your request..
Reference ID: 14047b370aaf50ee71a818bb63984ddc

Comment 6 Liang Xia 2014-01-06 05:54:50 UTC

Move to verified based on comment #4 and following checkings.

1. Create 20 applications.
2. Verify apps are created successfully on node1/node2/node3.
# for i in `seq 1 20` ; do rhc ssh app$i hostname ; done | sort | uniq
ex-std-node1.int.rhcloud.com
ex-std-node2.int.rhcloud.com
ex-std-node3.int.rhcloud.com


Notes:
Issue in comment #5 will be tracked in a separate bug.