Bug 1051349

Summary: sql db cannot be connected via java datasource after db cartridge restart on the first time
Product: OpenShift Online Reporter: Meng Bo <bmeng>
Component: ImageAssignee: Ben Parees <bparees>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bparees, rsmeral
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-30 00:55:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
cdi-jpa-reproducer-kitchensink.war
none
script_to_connect_db none

Description Meng Bo 2014-01-10 05:56:43 UTC 
Description of problem:
Create jboss app with mysql added. Add a simple jsp file to use the data source to connect to the db. 
After the db restart, the first time of the connection will fail.

Version-Release number of selected component (if applicable):
devenv_4204

How reproducible:
always

Steps to Reproduce:
1.Create jboss app with mysql added
rhc app create jbeap1 jbosseap-6 mysql-5.1
2.Add a simple jsp script to use the datasource (see attachment)
3.Visit the jsp page
4.Restart the mysql cartridge
5.Visit the jsp page after the restart finished
6.Visit the jsp page again after the restart finished

Actual results:
The db connection will fail on step5. And success on step6.

Expected results:
The db connection should not fail after db restart.

Additional info:
Use a stop/start instead of restart will avoid this issue.
The script attached.

server.log at step5:
2014/01/09 22:29:47,705 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/].[jsp]] (http-127.1.244.1/127.1.244.1:8080-1) JBWEB000236: Servlet.service() for servlet jsp threw exception: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2540)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2990)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2979)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
	at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1709)
	at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1628)
	at org.jboss.jca.adapters.jdbc.WrappedStatement.executeUpdate(WrappedStatement.java:375)
	at org.apache.jsp.mysql_jsp._jspService(mysql_jsp.java:74)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69) [jbossweb.jar:7.2.2.Final-redhat-1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1]
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:365) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:309) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:242) [jbossweb.jar:7.2.2.Final-redhat-1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web.jar:7.3.0.Final-redhat-14]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb.jar:7.2.2.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb.jar:7.2.2.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
Comment 1 Ron Šmeral 2014-01-10 15:41:36 UTC 
When the application uses CDI and JPA, it doesn't recover after the first refresh. It keeps throwing exceptions and behaves as described here: https://www.openshift.com/forums/openshift/eap-cant-connect-to-db-after-db-restart

The attachment cdi-jpa-reproducer-kitchensink.war reproduces this behaviour:
1. Deploy to an EAP+MySQL OS app as ROOT.war
2. Open the app URL
3. Restart just the DB gear
4. Open the app URL again, exception appears

Also, stopping+starting doesn't make any difference compared to restarting in this case.
Comment 2 Ron Šmeral 2014-01-10 15:42:40 UTC 
Created attachment 848263 [details]
cdi-jpa-reproducer-kitchensink.war
Comment 3 Ben Parees 2014-01-10 21:18:40 UTC 
I'm going to decrease the connection validation period from 10 minutes to 1 minute in the default jboss configuration file.

If that is still not frequent enough, there are two additional options you can use by modifying your applications datasource definition in the standalone.xml:

1) validate-on-match (commented out in the new config) which will validate the connection every time it is checked out of the pool
2) lower the background check interval, which is currently 60,000 milliseconds.
Comment 4 Meng Bo 2014-01-13 01:54:11 UTC 
Created attachment 849099 [details]
script_to_connect_db

Ah, forgot to add the attachment..
Comment 5 Ron Šmeral 2014-01-13 13:20:51 UTC 
The advice given in comment 3 both seem to help the issue.

Ben, can you please apply the same fix in datasources other than MySQL as well? I can confirm that the same problem happens at least with PostgreSQL.
Comment 6 Ben Parees 2014-01-13 13:31:49 UTC 
Yes I am going to update both datasources that come out of the box, for both jbossas and jbosseap.
Comment 7 Ben Parees 2014-01-13 19:48:16 UTC 
PR for fix: https://github.com/openshift/origin-server/pull/4455
Comment 8 openshift-github-bot 2014-01-14 00:02:31 UTC 
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/5d979d0b269a062fdb2a097178b9faf9eceb4502
Bug 1051349 - sql db cannot be connected via java datasource after db cartridge restart on the first time
Comment 9 Meng Bo 2014-01-14 06:06:25 UTC 
Checked on devenv_4219, for both my case and Ron's case are worked fine now.

After only db restart the mysql db still can be connected.

Move bug to verified.