Description of problem:
Sometimes mongodb on server shutdown when facing heavy load.

Version-Release number of selected component (if applicable):
rubygem-stickshift-node-0.8.2-1.el6_2.noarch
rubygem-stickshift-controller-0.8.12-1.el6_2.noarch
rubygem-stickshift-common-0.8.1-1.el6_2.noarch
rhc-common-0.76.1-1.el6_2.noarch
rhc-site-0.90.11-1.el6_2.noarch
rhc-node-0.90.5-1.el6_2.x86_64
rhc-devenv-0.91.8-1.el6_2.noarch
rhc-broker-0.90.17-1.el6_2.noarch
mongodb-2.0.2-1.el6_2.x86_64


How reproducible:
Sometimes

Steps to Reproduce:
This happens during automation testing. I did a lot concurrent creating app, snapshot, scale up operations, and started to got error message: "There was a problem communicating with the server. Response message: 503 "Service Unavailable".
  
Actual results:

MongoDb status:
$ service mongod status
Result:
mongod dead but subsys locked

MongoDb log:
==> /var/log/mongodb/mongodb.log <==
Wed Apr 11 07:27:01 [initandlisten] MongoDB starting : pid=3090 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-10-62-7-153
Wed Apr 11 07:27:01 [initandlisten] db version v2.0.2, pdfile version 4.5
Wed Apr 11 07:27:01 [initandlisten] git version: nogitversion
Wed Apr 11 07:27:01 [initandlisten] build info: Linux x86-004.build.bos.redhat.com 2.6.18-274.12.1.el5 #1 SMP Tue Nov 8 21:37:35 EST 2011 x86_64 BOOST_LIB_VERSION=1_41
Wed Apr 11 07:27:01 [initandlisten] options: { auth: true, bind_ip: "127.0.0.1", command: [ "run" ], config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", fork: "true", logpath: "/var/log/mongodb/mongodb.log", nojournal: true, oplogSize: 16, pidfilepath: "/var/run/mongodb/mongodb.pid", quiet: true, replSet: "libra_rs", smallfiles: true }
**************
Unclean shutdown detected.
Please visit http://dochub.mongodb.org/core/repair for recovery instructions.
*************
Wed Apr 11 07:27:01 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
Wed Apr 11 07:27:01 dbexit:
Wed Apr 11 07:27:01 [initandlisten] shutdown: going to close listening sockets...
Wed Apr 11 07:27:01 [initandlisten] shutdown: going to flush diaglog...
Wed Apr 11 07:27:01 [initandlisten] shutdown: going to close sockets...
Wed Apr 11 07:27:01 [initandlisten] shutdown: waiting for fs preallocator...
Wed Apr 11 07:27:01 [initandlisten] shutdown: closing all files...
Wed Apr 11 07:27:01 [initandlisten] closeAllFiles() finished
Wed Apr 11 07:27:01 dbexit: really exiting now


==> development.log <==
Started POST "/broker/cartridge" for 66.187.233.202 at Wed Apr 11 07:25:58 -0400 2012
  Processing by LegacyBrokerController#cartridge_post as
  Parameters: {"json_data"=>"{\"app_name\":\"phpcibldr\",\"api\":\"1.1.3\",\"debug\":true,\"rhlogin\":\"jizhao+3\",\"action\":\"deconfigure\"}", "password"=>"[FILTERED]"}
User-Agent = ''
Adding user jizhao+3...inside legacy_controller
MongoDataStore.find(CloudUser, jizhao+3, jizhao+3)

MongoDataStore.find(CloudUser, jizhao+3, jizhao+3)

DEBUG: DYNECT Login with path: /REST/Session/
DEBUG: DYNECT Response: {"status": "success", "data": {"token": "5eo4lAfm3sGC5gTXHAEpvpsWE+xBjdOA4gd/NHRJdHsxt26pgGq/tOfICqqr7rNuPR2jnRLgYG+JxgB/J9pyqY1bXa0k4oc1BYTK7gHe+eEwVsLbS+2AHWIwi7VP1Navnn5xCpYtz7+aEmQPwKhHxen12Q8sru8c", "version": "3.3.2"}, "job_id": 85930552, "msgs": [{"INFO": "login: Login successful", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]}
DEBUG: DYNECT Response data: token5eo4lAfm3sGC5gTXHAEpvpsWE+xBjdOA4gd/NHRJdHsxt26pgGq/tOfICqqr7rNuPR2jnRLgYG+JxgB/J9pyqY1bXa0k4oc1BYTK7gHe+eEwVsLbS+2AHWIwi7VP1Navnn5xDEPRECATION WARNING: config.action_view.debug_rjs will be removed in 3.1, from 3.1 onwards you will need to install prototype-rails to continue to use RJS templates . (called from /var/www/stickshift/broker/config/environment.rb:5)


Started HEAD "/broker" for 127.0.0.1 at Wed Apr 11 07:27:40 -0400 2012

ActionController::RoutingError (No route matches "/"):


Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/templates/rescues/routing_error.erb within rescues/layout (1.3ms)

Started HEAD "/broker" for 127.0.0.1 at Wed Apr 11 07:27:40 -0400 2012

ActionController::RoutingError (No route matches "/"):


Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/templates/rescues/routing_error.erb within rescues/layout (7.1ms)
DEPRECATION WARNING: config.action_view.debug_rjs will be removed in 3.1, from 3.1 onwards you will need to install prototype-rails to continue to use RJS templates . (called from /var/www/stickshift/broker/config/environment.rb:5)



Expected results:
MongoDb shouldn't be dead.

Additional info: