Description of problem: Create jboss app with mysql added. Add a simple jsp file to use the data source to connect to the db. After the db restart, the first time of the connection will fail. Version-Release number of selected component (if applicable): devenv_4204 How reproducible: always Steps to Reproduce: 1.Create jboss app with mysql added rhc app create jbeap1 jbosseap-6 mysql-5.1 2.Add a simple jsp script to use the datasource (see attachment) 3.Visit the jsp page 4.Restart the mysql cartridge 5.Visit the jsp page after the restart finished 6.Visit the jsp page again after the restart finished Actual results: The db connection will fail on step5. And success on step6. Expected results: The db connection should not fail after db restart. Additional info: Use a stop/start instead of restart will avoid this issue. The script attached. server.log at step5: 2014/01/09 22:29:47,705 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/].[jsp]] (http-127.1.244.1/127.1.244.1:8080-1) JBWEB000236: Servlet.service() for servlet jsp threw exception: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2540) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2990) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2979) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619) at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1709) at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1628) at org.jboss.jca.adapters.jdbc.WrappedStatement.executeUpdate(WrappedStatement.java:375) at org.apache.jsp.mysql_jsp._jspService(mysql_jsp.java:74) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69) [jbossweb.jar:7.2.2.Final-redhat-1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:365) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:309) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:242) [jbossweb.jar:7.2.2.Final-redhat-1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb.jar:7.2.2.Final-redhat-1] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web.jar:7.3.0.Final-redhat-14] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb.jar:7.2.2.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb.jar:7.2.2.Final-redhat-1] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
When the application uses CDI and JPA, it doesn't recover after the first refresh. It keeps throwing exceptions and behaves as described here: https://www.openshift.com/forums/openshift/eap-cant-connect-to-db-after-db-restart The attachment cdi-jpa-reproducer-kitchensink.war reproduces this behaviour: 1. Deploy to an EAP+MySQL OS app as ROOT.war 2. Open the app URL 3. Restart just the DB gear 4. Open the app URL again, exception appears Also, stopping+starting doesn't make any difference compared to restarting in this case.
Created attachment 848263 [details] cdi-jpa-reproducer-kitchensink.war
I'm going to decrease the connection validation period from 10 minutes to 1 minute in the default jboss configuration file. If that is still not frequent enough, there are two additional options you can use by modifying your applications datasource definition in the standalone.xml: 1) validate-on-match (commented out in the new config) which will validate the connection every time it is checked out of the pool 2) lower the background check interval, which is currently 60,000 milliseconds.
Created attachment 849099 [details] script_to_connect_db Ah, forgot to add the attachment..
The advice given in comment 3 both seem to help the issue. Ben, can you please apply the same fix in datasources other than MySQL as well? I can confirm that the same problem happens at least with PostgreSQL.
Yes I am going to update both datasources that come out of the box, for both jbossas and jbosseap.
PR for fix: https://github.com/openshift/origin-server/pull/4455
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/5d979d0b269a062fdb2a097178b9faf9eceb4502 Bug 1051349 - sql db cannot be connected via java datasource after db cartridge restart on the first time
Checked on devenv_4219, for both my case and Ron's case are worked fine now. After only db restart the mysql db still can be connected. Move bug to verified.