Red Hat Bugzilla – Bug 857132
Scale-Up operation fails with JBoss and Postgres
Last modified: 2012-09-13 19:48:22 EDT
Description of problem:
On a scaled JBoss application using Postgres attempts to scale up a gear appear to fail consistently. According to the logs, JBoss is unable to connect to the Postgres data store and show L7STS/404 under the LstChk column in the haproxy_status page. My local/home gear continues to function normally, but the new gear is never started fully.
On recommendation from mmcgrath in IRC, I SSH'd into the failed slave gear and executed 'ctl_all restart'. On completion, the gear was responding normally in haproxy_status and began to serve requests (and was connected successfully to Postgres).
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Run scaled application with Postgres database
2. Execute 'haproxy_ctld -u'
3. Monitor APP_URL/haproxy_status. Before the haproxy_ctld command completes, the new gear will show as DOWN
Could you please attach the scale_events.log file from your haproxy gear?
Created attachment 612529 [details]
Attached is the scale_events log from my shared haproxy/app gear.
Created attachment 612590 [details]
Redeploying New Code Now Fails
Now when pushing to the repository, the app is unable to deploy. It hangs where the maven build should be starting.
rhc app restart fails.
rhc app force-stop succeeds.
rhc app start fails with cartridge error 121.
More in the attached log.
Correction to my description, the initial "rhc app restart" says it has succeeded but does not actually work :)
Created attachment 612610 [details]
Local Gear's server.log
Here is the server.log from the last hour or so on the local gear.
Created attachment 612611 [details]
Log from gear 2
The error and related stacktraces below are a known issue with database connections being dropped/invalidated between the application and database gears. JBoss should recover and create a new connection. The warning errors and stacktraces are just (annoying) noise
2012/09/13 11:43:26,714 WARN [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (http-127.8.4.129-127.8.4.129-8080-2) Destroying connection that is not valid, due to the following exception: org.postgresql.jdbc4.Jdbc4Connection@23bd2a5d: org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
I'd like to close this bug and open another one for your second issue (rhc app restart failing) if that's alright?
(In reply to comment #8)
> I'd like to close this bug and open another one for your second issue (rhc
> app restart failing) if that's alright?
> Thanks -Bill
By all means, please do. If you could just CC me on the new bug or something that would be great.