Red Hat Bugzilla – Bug 961785
Local gear of a scaling application is always down after inserting a large amount of data to a DB cartridge times out
Last modified: 2015-05-14 19:18:21 EDT
Created attachment 746130 [details]
Sample application to insert large amout of data to db
Description of problem:
Given a scaling application with a DB cartridge embedded, when inserting a large amount of data (say, 860M) once, it always shows "Time Out" error, and hereafter, the local gear is down when visiting haproxy-status page.
This test would not be a common use for general users, but I was wondering if we could make nodes more robust.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. create an scaling application with a DB cartridge embedded
rhc app create spy27 python-2.7 mysql-5.1 -s
2. modify its setup.py file to enable enable "MySQL-python" module
3. modify the <app_repo>/wsgi/application file with the code in the attachment.
5. push the change
6. to verify the code works
7. insert a large amount of data into this app once by visiting
8. check the haproxy-status page of this app
(it will insert 500000 records into the MySQL cartridge, around 860M)
In Step 5, the insertion succeeds and it also shows the expected results when accessing <app_repo>/show
However, in step 7, it fails with a 504 time-out error, and the status of the app's local gear is down. Moreover, when visiting its homepage, it will always redirects to the haproxy-proxy page.
The app should be more robust to deal with heavy workload.
Commit pushed to master at https://github.com/openshift/origin-server
Bug 961785 - Cartridge URL install failed
* Insufficient output to debug
I tested orders of magnitude of ten up to 100k, and didn't experience a timeout. When I tried to insert 1M records, the request timed out, but the app stayed up. I don't think this represents a problem with the platform - the app is inserting single rows in a tight loop with autocommit while processing the http request, which isn't very robust. It would improve performance to turn off autocommit before doing the inserts.